A kind of lightweight data processing framework based on chain of messages
Technical field
The present invention relates to distributed computing technology field, refer to especially a kind of lightweight data processing framework based on chain of messages.
Background technology
A distributed system need to be processed dissimilar message, and the processing logic of message was both had something in common, and also can need special processing logic according to the difference of type of message.We generally adopt message-oriented middleware to carry out the common problem of processing messages, and its treatment step is such:
1, in message-oriented middleware, generally there are three roles: message-oriented middleware, consumer, ISP;
2, consumer assembles request of data message and sends to message;
3, message-oriented middleware is transmitted to each ISP by consumer's request message and monitors queue, and ISP receives requested service message;
4, the asynchronous response message that returns to request message of ISP;
5, message-oriented middleware turns back to corresponding consumer by the response message of consumer's request message and replys queue, and consumer receives the response message of request, and does respective handling;
6, ISP generates the required data of consumer, and sends message to message-oriented middleware;
7, message-oriented middleware will notify message repeating to monitor queue to each consumer, and consumer receives informing business message;
8, the asynchronous response message that returns to notice message of consumer.
, the problem of message-oriented middleware below also existing in the little distributed environment of size of message:
1, message-oriented middleware is an independently message engine, can take part of server resource, for the little Distributed Application of size of message, there is no need to increase such expense, and the introducing third party's role, message (message) temporary through too much one deck queue in transmitting procedure, can lower the real-time of message.
2, message-oriented middleware does not have the treatment mechanism of flexible design to message processing logic, once message processing logic changes, needs recompile to realize.
3, message-oriented middleware does not have the special processing logic mechanism of design to message, unfavorable to the exploitation of particular service logic.
Summary of the invention
The technical matters that the present invention solves is to provide a kind of lightweight data processing framework based on chain of messages, solves the problem that Message Processing is dumb, development efficiency is low, message real-time is low that current universal information middleware exists;
The technical scheme that the present invention solves the problems of the technologies described above is:
Comprise step:
After upstream message is received from message channel, enter source processing chain module;
Message after the processing of the up processing logic of each processor on chain, is cached in upstream message queue successively;
Upstream message queue management thread takes out message and puts into message threads pond from message queue;
Message threads pond becomes object or structure by message parse;
Whether message observer accepts message according to the type decided of message object or structure, and it is carried out to particular service logical process;
Processing corresponding or user asks to change into message buffering in downstream message queue through message resolution module;
Message queue is kept at downstream message in queue to be sent;
In queue to be sent, message is fallen out in order and is entered source processing chain, successively through being sent to message channel after each processor downlink processing logical process on chain.
Described message is divided into upstream message and downstream message; Upstream message refers to that system receives and need to carry out the request of the corresponding command operation from message channel, and downstream message refers to the request that response after command operation is handled or user need to send.
Described source processing chain is the chain type set of Message Processing instrument, comprises upstream message processing chain and downstream message processing chain; The Message Processing instrument that upstream message processing chain comprises has that message unpacks, message decompress(ion), decrypt messages and message authentication etc.; The Message Processing instrument that downstream message processing chain comprises has message authentication, message encryption, message compression and message packing etc. accordingly;
Described message threads pond refers to the set of each Message Processing thread, has the function that message parse is become to each object or structure;
Described message observer refers to and can for the message object parsing, carry out the interface object of various different disposal, comprises daily record observer, audit observation person, virtual machine observer, logical volume observer etc.; Message observer adopts Observer Pattern design to realize, and same message support is successively by a plurality of observer's reception & disposals; , Observer Pattern is a kind of programming implementation method in Programming.
Described up-downgoing source processing chain tool set support dynamically can be expanded.
Described up-downgoing message queue is supported internal memory form and database form;
Described internal memory form message queue refers to that the space of applying for a certain size in internal memory, as message format, has the characteristic that efficiency is high;
Described database form refers to message is first saved in to database, and the scheduling of the processing threads that waits for the arrival of news has persistence characteristic, supports the feature that service fault message recovery is processed again.
Described message observer's particular service logical process refers to that being similar to daily record observer carries out log recording, and audit observation person audit, and virtual machine observer creates virtual machine, closes the business logic processing such as virtual machine.
The present invention program's beneficial effect is as follows:
1, framework of the present invention is lightweight, and less to the extra expense increasing of system, the real-time of message is high.
2, framework of the present invention is high flexible, and the processing logic in source processing chain can increase and decrease by configuration as required, without recompile, can move; Increase the special processing logic of message is also easier to.
3, method of the present invention can be developed distributed treatment application at short notice, and the processor in source processing chain is all according to the general purpose module of general character processing logic exploitation, only need to be configured or do a small amount of coding in actual use.
4, can improve the quality of distributed application development, increase customer satisfaction.Because this framework of application can be accomplished not write code or only write minute quantity code, therefore reduced the mistake that in performance history, coding produces, improved software quality.
Accompanying drawing explanation
Below in conjunction with accompanying drawing, the present invention is further described:
Fig. 1 is process flow diagram of the present invention;
Fig. 2 is that upstream data of the present invention is processed framework exploded view;
Embodiment
As shown in Figure 1, the lightweight data processing method that the present invention is based on chain of messages comprises the steps:
After upstream message is received from message channel, enter source processing chain module;
Message after the processing of the up processing logic of each processor on chain, is cached in upstream message queue successively;
Upstream message queue management thread takes out message and puts into message threads pond from message queue;
Message threads pond becomes object or structure by message parse;
Whether message observer accepts message according to the type decided of message object or structure, and it is carried out to particular service logical process;
Processing corresponding or user asks to change into message buffering in downstream message queue through message resolution module;
Message queue is kept at downstream message in queue to be sent;
In queue to be sent, message is fallen out in order and is entered source processing chain, successively through being sent to message channel after each processor downlink processing logical process on chain.
Described message is divided into upstream message and downstream message; Upstream message refers to that system receives and need to carry out the request of the corresponding command operation from message channel, and downstream message refers to the request that response after command operation is handled or user need to send.
Source processing chain is the chain type set of Message Processing instrument, comprises upstream message processing chain and downstream message processing chain; The Message Processing instrument that upstream message processing chain comprises has that message unpacks, message decompress(ion), decrypt messages and message authentication etc.; The Message Processing instrument that downstream message processing chain comprises has message authentication, message encryption, message compression and message packing etc. accordingly;
Message threads pond refers to the set of each Message Processing thread, has the function that message parse is become to each object or structure;
Message observer refers to and can for the message object parsing, carry out the interface object of various different disposal, comprises daily record observer, audit observation person, virtual machine observer, logical volume observer etc.; Message observer adopts Observer Pattern design to realize, and same message support is successively by a plurality of observer's reception & disposals; , Observer Pattern is a kind of programming implementation method in Programming.
The support of up-downgoing source processing chain tool set dynamically can be expanded.
Up-downgoing message queue is supported internal memory form and database form; Internal memory form message queue refers to that the space of applying for a certain size in internal memory, as message format, has the characteristic that efficiency is high; Database form refers to message is first saved in to database, and the scheduling of the processing threads that waits for the arrival of news has persistence characteristic, supports the feature that service fault message recovery is processed again.
Message observer's particular service logical process refers to that being similar to daily record observer carries out log recording, and audit observation person audit, and virtual machine observer creates virtual machine, closes the business logic processing such as virtual machine.
Again in conjunction with shown in Fig. 2, the stage of source processing chain in text message turnover in data processing framework, comprise unpack, the processing procedure such as decompress(ion), deciphering, authentication, each processing procedure is a Handler, and each Handler need inherit MsgHandler base class, the design of MsgHandler base class is as follows:
Each message processing procedure is dynamic extendible upstream message processing chain IncomeMsgChain and the downstream message processing chain OutcomeMsgChain of being assembled into as required.Assembling process is processed Handler for each exactly and is called unified interface setSuccessor, and making it in succession to link together becomes a source processing chain, and link-group process of assembling is as follows:
Upstream message processing chain construction process is as follows:
Downstream message processing chain construction process is as follows:
Downstream message processing chain has been carried out encryption, pressurization, compressed logic processing to message successively.
The same with source processing chain, message queue is also designed to upstream message queue and downstream message queue, and message queue administrative class is inherited IMqMng interface, and Interface design is as follows:
There are respectively two thread management in the team that goes out of queue.InComeMqPollThread and OutComeMqPollThread.Their work of two is exactly constantly message to be taken out from queue, then puts thread pool into and carries out.Thread pool is achieved as follows:
Message observer receives concrete message object and structure, carries out concrete business logic processing.In implementation, message observer inherits CommandObserver abstract class, and its code is as follows:
All message observers, by the unified management of message observer administrative class, have adopted Observer Pattern in implementation method, code is as follows: