CN103677844A - Lightweight class data processing frame based on message chain - Google Patents
Lightweight class data processing frame based on message chain Download PDFInfo
- Publication number
- CN103677844A CN103677844A CN201310714072.8A CN201310714072A CN103677844A CN 103677844 A CN103677844 A CN 103677844A CN 201310714072 A CN201310714072 A CN 201310714072A CN 103677844 A CN103677844 A CN 103677844A
- Authority
- CN
- China
- Prior art keywords
- message
- processing
- observer
- chain
- messages
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention relates to the technical field of distributed computation, in particular to a lightweight class data processing frame based on a message chain. Messages are cached into an uplink message queue after being received, entering a message processing chain module and then being processed by uplink processing logic on all processors on the chain; a manager thread of the uplink message queue fetches the messages and puts the messages into a massage thread pool; the message thread pool analyzes the messages into objects or structural bodies; a message observer determines whether to receive the messages or not according to types and conducts special service logic processing on the messages; user requests or responses are converted into messages through a message analysis module, and then the messages are cached into a downlink message queue; the message queue stores the downlink messages into a queue to be sent; the messages dequeue in sequence, enter the message processing chain and are sent to a message channel after being processed by downlink processing logic of all the processors on the chain. The lightweight class data processing frame solves the problem of message-oriented middleware, and can be used for rapid integrated development of third party software and hardware based on the data processing frame.
Description
Technical field
The present invention relates to distributed computing technology field, refer to especially a kind of lightweight data processing framework based on chain of messages.
Background technology
A distributed system need to be processed dissimilar message, and the processing logic of message was both had something in common, and also can need special processing logic according to the difference of type of message.We generally adopt message-oriented middleware to carry out the common problem of processing messages, and its treatment step is such:
1, in message-oriented middleware, generally there are three roles: message-oriented middleware, consumer, ISP;
2, consumer assembles request of data message and sends to message;
3, message-oriented middleware is transmitted to each ISP by consumer's request message and monitors queue, and ISP receives requested service message;
4, the asynchronous response message that returns to request message of ISP;
5, message-oriented middleware turns back to corresponding consumer by the response message of consumer's request message and replys queue, and consumer receives the response message of request, and does respective handling;
6, ISP generates the required data of consumer, and sends message to message-oriented middleware;
7, message-oriented middleware will notify message repeating to monitor queue to each consumer, and consumer receives informing business message;
8, the asynchronous response message that returns to notice message of consumer.
, the problem of message-oriented middleware below also existing in the little distributed environment of size of message:
1, message-oriented middleware is an independently message engine, can take part of server resource, for the little Distributed Application of size of message, there is no need to increase such expense, and the introducing third party's role, message (message) temporary through too much one deck queue in transmitting procedure, can lower the real-time of message.
2, message-oriented middleware does not have the treatment mechanism of flexible design to message processing logic, once message processing logic changes, needs recompile to realize.
3, message-oriented middleware does not have the special processing logic mechanism of design to message, unfavorable to the exploitation of particular service logic.
Summary of the invention
The technical matters that the present invention solves is to provide a kind of lightweight data processing framework based on chain of messages, solves the problem that Message Processing is dumb, development efficiency is low, message real-time is low that current universal information middleware exists;
The technical scheme that the present invention solves the problems of the technologies described above is:
Comprise step:
After upstream message is received from message channel, enter source processing chain module;
Message after the processing of the up processing logic of each processor on chain, is cached in upstream message queue successively;
Upstream message queue management thread takes out message and puts into message threads pond from message queue;
Message threads pond becomes object or structure by message parse;
Whether message observer accepts message according to the type decided of message object or structure, and it is carried out to particular service logical process;
Processing corresponding or user asks to change into message buffering in downstream message queue through message resolution module;
Message queue is kept at downstream message in queue to be sent;
In queue to be sent, message is fallen out in order and is entered source processing chain, successively through being sent to message channel after each processor downlink processing logical process on chain.
Described message is divided into upstream message and downstream message; Upstream message refers to that system receives and need to carry out the request of the corresponding command operation from message channel, and downstream message refers to the request that response after command operation is handled or user need to send.
Described source processing chain is the chain type set of Message Processing instrument, comprises upstream message processing chain and downstream message processing chain; The Message Processing instrument that upstream message processing chain comprises has that message unpacks, message decompress(ion), decrypt messages and message authentication etc.; The Message Processing instrument that downstream message processing chain comprises has message authentication, message encryption, message compression and message packing etc. accordingly;
Described message threads pond refers to the set of each Message Processing thread, has the function that message parse is become to each object or structure;
Described message observer refers to and can for the message object parsing, carry out the interface object of various different disposal, comprises daily record observer, audit observation person, virtual machine observer, logical volume observer etc.; Message observer adopts Observer Pattern design to realize, and same message support is successively by a plurality of observer's reception & disposals; , Observer Pattern is a kind of programming implementation method in Programming.
Described up-downgoing source processing chain tool set support dynamically can be expanded.
Described up-downgoing message queue is supported internal memory form and database form;
Described internal memory form message queue refers to that the space of applying for a certain size in internal memory, as message format, has the characteristic that efficiency is high;
Described database form refers to message is first saved in to database, and the scheduling of the processing threads that waits for the arrival of news has persistence characteristic, supports the feature that service fault message recovery is processed again.
Described message observer's particular service logical process refers to that being similar to daily record observer carries out log recording, and audit observation person audit, and virtual machine observer creates virtual machine, closes the business logic processing such as virtual machine.
The present invention program's beneficial effect is as follows:
1, framework of the present invention is lightweight, and less to the extra expense increasing of system, the real-time of message is high.
2, framework of the present invention is high flexible, and the processing logic in source processing chain can increase and decrease by configuration as required, without recompile, can move; Increase the special processing logic of message is also easier to.
3, method of the present invention can be developed distributed treatment application at short notice, and the processor in source processing chain is all according to the general purpose module of general character processing logic exploitation, only need to be configured or do a small amount of coding in actual use.
4, can improve the quality of distributed application development, increase customer satisfaction.Because this framework of application can be accomplished not write code or only write minute quantity code, therefore reduced the mistake that in performance history, coding produces, improved software quality.
Accompanying drawing explanation
Below in conjunction with accompanying drawing, the present invention is further described:
Fig. 1 is process flow diagram of the present invention;
Fig. 2 is that upstream data of the present invention is processed framework exploded view;
Embodiment
As shown in Figure 1, the lightweight data processing method that the present invention is based on chain of messages comprises the steps:
After upstream message is received from message channel, enter source processing chain module;
Message after the processing of the up processing logic of each processor on chain, is cached in upstream message queue successively;
Upstream message queue management thread takes out message and puts into message threads pond from message queue;
Message threads pond becomes object or structure by message parse;
Whether message observer accepts message according to the type decided of message object or structure, and it is carried out to particular service logical process;
Processing corresponding or user asks to change into message buffering in downstream message queue through message resolution module;
Message queue is kept at downstream message in queue to be sent;
In queue to be sent, message is fallen out in order and is entered source processing chain, successively through being sent to message channel after each processor downlink processing logical process on chain.
Described message is divided into upstream message and downstream message; Upstream message refers to that system receives and need to carry out the request of the corresponding command operation from message channel, and downstream message refers to the request that response after command operation is handled or user need to send.
Source processing chain is the chain type set of Message Processing instrument, comprises upstream message processing chain and downstream message processing chain; The Message Processing instrument that upstream message processing chain comprises has that message unpacks, message decompress(ion), decrypt messages and message authentication etc.; The Message Processing instrument that downstream message processing chain comprises has message authentication, message encryption, message compression and message packing etc. accordingly;
Message threads pond refers to the set of each Message Processing thread, has the function that message parse is become to each object or structure;
Message observer refers to and can for the message object parsing, carry out the interface object of various different disposal, comprises daily record observer, audit observation person, virtual machine observer, logical volume observer etc.; Message observer adopts Observer Pattern design to realize, and same message support is successively by a plurality of observer's reception & disposals; , Observer Pattern is a kind of programming implementation method in Programming.
The support of up-downgoing source processing chain tool set dynamically can be expanded.
Up-downgoing message queue is supported internal memory form and database form; Internal memory form message queue refers to that the space of applying for a certain size in internal memory, as message format, has the characteristic that efficiency is high; Database form refers to message is first saved in to database, and the scheduling of the processing threads that waits for the arrival of news has persistence characteristic, supports the feature that service fault message recovery is processed again.
Message observer's particular service logical process refers to that being similar to daily record observer carries out log recording, and audit observation person audit, and virtual machine observer creates virtual machine, closes the business logic processing such as virtual machine.
Again in conjunction with shown in Fig. 2, the stage of source processing chain in text message turnover in data processing framework, comprise unpack, the processing procedure such as decompress(ion), deciphering, authentication, each processing procedure is a Handler, and each Handler need inherit MsgHandler base class, the design of MsgHandler base class is as follows:
Each message processing procedure is dynamic extendible upstream message processing chain IncomeMsgChain and the downstream message processing chain OutcomeMsgChain of being assembled into as required.Assembling process is processed Handler for each exactly and is called unified interface setSuccessor, and making it in succession to link together becomes a source processing chain, and link-group process of assembling is as follows:
Upstream message processing chain construction process is as follows:
Downstream message processing chain construction process is as follows:
Downstream message processing chain has been carried out encryption, pressurization, compressed logic processing to message successively.
The same with source processing chain, message queue is also designed to upstream message queue and downstream message queue, and message queue administrative class is inherited IMqMng interface, and Interface design is as follows:
There are respectively two thread management in the team that goes out of queue.InComeMqPollThread and OutComeMqPollThread.Their work of two is exactly constantly message to be taken out from queue, then puts thread pool into and carries out.Thread pool is achieved as follows:
Message observer receives concrete message object and structure, carries out concrete business logic processing.In implementation, message observer inherits CommandObserver abstract class, and its code is as follows:
All message observers, by the unified management of message observer administrative class, have adopted Observer Pattern in implementation method, code is as follows:
Claims (7)
1. the lightweight data processing framework based on chain of messages, is characterized in that: comprise step:
After upstream message is received from message channel, enter source processing chain module;
Message after the processing of the up processing logic of each processor on chain, is cached in upstream message queue successively;
Upstream message queue management thread takes out message and puts into message threads pond from message queue;
Message threads pond becomes object or structure by message parse;
Whether message observer accepts message according to the type decided of message object or structure, and it is carried out to particular service logical process;
Processing corresponding or user asks to change into message buffering in downstream message queue through message resolution module;
Message queue is kept at downstream message in queue to be sent;
In queue to be sent, message is fallen out in order and is entered source processing chain, successively through being sent to message channel after each processor downlink processing logical process on chain.
2. lightweight data processing framework according to claim 1, is characterized in that: described message is divided into upstream message and downstream message; Upstream message refers to that system receives and need to carry out the request of the corresponding command operation from message channel, and downstream message refers to the request that response after command operation is handled or user need to send.
3. lightweight data processing framework according to claim 1, is characterized in that: described source processing chain is the chain type set of Message Processing instrument, comprises upstream message processing chain and downstream message processing chain; The Message Processing instrument that upstream message processing chain comprises has that message unpacks, message decompress(ion), decrypt messages and message authentication etc.; The Message Processing instrument that downstream message processing chain comprises has message authentication, message encryption, message compression and message packing etc. accordingly;
Described message threads pond refers to the set of each Message Processing thread, has the function that message parse is become to each object or structure;
Described message observer refers to and can for the message object parsing, carry out the interface object of various different disposal, comprises daily record observer, audit observation person, virtual machine observer, logical volume observer etc.; Message observer adopts Observer Pattern design to realize, and same message support is successively by a plurality of observer's reception & disposals; , Observer Pattern is a kind of programming implementation method in Programming.
4. lightweight data processing framework according to claim 3, is characterized in that: described up-downgoing source processing chain tool set support dynamically can be expanded.
5. according to the lightweight data processing framework described in claim 1 to 4 any one, it is characterized in that: described up-downgoing message queue is supported internal memory form and database form;
Described internal memory form message queue refers to that the space of applying for a certain size in internal memory, as message format, has the characteristic that efficiency is high;
Described database form refers to message is first saved in to database, and the scheduling of the processing threads that waits for the arrival of news has persistence characteristic, supports the feature that service fault message recovery is processed again.
6. according to the lightweight data processing framework described in claim 1 to 4 any one, it is characterized in that: described message observer's particular service logical process refers to that being similar to daily record observer carries out log recording, audit observation person audit, and virtual machine observer creates virtual machine, closes the business logic processing such as virtual machine.
7. lightweight data processing framework according to claim 5, it is characterized in that: described message observer's particular service logical process refers to that being similar to daily record observer carries out log recording, audit observation person audit, and virtual machine observer creates virtual machine, closes the business logic processing such as virtual machine.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310714072.8A CN103677844B (en) | 2013-12-20 | 2013-12-20 | Lightweight class data processing method based on message chain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310714072.8A CN103677844B (en) | 2013-12-20 | 2013-12-20 | Lightweight class data processing method based on message chain |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103677844A true CN103677844A (en) | 2014-03-26 |
| CN103677844B CN103677844B (en) | 2017-02-01 |
Family
ID=50315511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310714072.8A Active CN103677844B (en) | 2013-12-20 | 2013-12-20 | Lightweight class data processing method based on message chain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103677844B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648816A (en) * | 2016-12-09 | 2017-05-10 | 武汉斗鱼网络科技有限公司 | Multithread processing system and multithread processing method |
| CN106648444A (en) * | 2015-10-30 | 2017-05-10 | 大唐移动通信设备有限公司 | Message processing method and device |
| CN107545489A (en) * | 2017-08-18 | 2018-01-05 | 秒啊（北京）科技有限公司 | A kind of time method of commerce |
| CN111555955A (en) * | 2020-03-26 | 2020-08-18 | 福州慧校通教育信息技术有限公司 | Method, device, equipment and medium for implementing message center |
| CN114615213A (en) * | 2022-01-26 | 2022-06-10 | 福达新创通讯科技(厦门)有限公司 | Multi-network link interface implementation method and system based on message queue |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8359574B1 (en) * | 2009-01-16 | 2013-01-22 | Adobe Systems Incorporated | Offline mode in IDE for seamless service-oriented-architecture-based application development |
| CN101742728A (en) * | 2010-01-12 | 2010-06-16 | 深圳市五巨科技有限公司 | Method for realizing dynamic task of mobile phone platform |
| CN102523177A (en) * | 2011-12-19 | 2012-06-27 | 北京新媒传信科技有限公司 | Method and system for realization of message push service |
| WO2014078989A1 (en) * | 2012-11-20 | 2014-05-30 | 华为技术有限公司 | Message processing method and server |
| CN103024014B (en) * | 2012-12-03 | 2015-08-19 | 北京京东世纪贸易有限公司 | By the method and system of the mass data distribution processor of message queue |
-
2013
- 2013-12-20 CN CN201310714072.8A patent/CN103677844B/en active Active
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106648444A (en) * | 2015-10-30 | 2017-05-10 | 大唐移动通信设备有限公司 | Message processing method and device |
| CN106648444B (en) * | 2015-10-30 | 2019-06-28 | 大唐移动通信设备有限公司 | A kind of message treatment method and equipment |
| CN106648816A (en) * | 2016-12-09 | 2017-05-10 | 武汉斗鱼网络科技有限公司 | Multithread processing system and multithread processing method |
| CN106648816B (en) * | 2016-12-09 | 2020-03-17 | 武汉斗鱼网络科技有限公司 | Multithreading system and method |
| CN107545489A (en) * | 2017-08-18 | 2018-01-05 | 秒啊（北京）科技有限公司 | A kind of time method of commerce |
| CN111555955A (en) * | 2020-03-26 | 2020-08-18 | 福州慧校通教育信息技术有限公司 | Method, device, equipment and medium for implementing message center |
| CN111555955B (en) * | 2020-03-26 | 2022-08-05 | 福州慧校通教育信息技术有限公司 | Method, device, equipment and medium for implementing message center |
| CN114615213A (en) * | 2022-01-26 | 2022-06-10 | 福达新创通讯科技(厦门)有限公司 | Multi-network link interface implementation method and system based on message queue |
| CN114615213B (en) * | 2022-01-26 | 2024-04-12 | 福达新创通讯科技(厦门)有限公司 | Method and system for implementing multi-network link interface based on message queue |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103677844B (en) | 2017-02-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10055224B2 (en) | Reconfigurable hardware structures for functional pipelining of on-chip special purpose functions | |
| US9800691B2 (en) | Stream processing using a client-server architecture | |
| US10824537B2 (en) | Method, device, and computer readable medium for tracing computing system | |
| US20160301739A1 (en) | Endpoint management system providing an application programming interface proxy service | |
| CN103677844A (en) | Lightweight class data processing frame based on message chain | |
| US9596127B2 (en) | Scalable data feed system | |
| CN109726004B (en) | Data processing method and device | |
| US8977637B2 (en) | Facilitating field programmable gate array accelerations of database functions | |
| CN108038207A (en) | A kind of daily record data processing system, method and server | |
| US20190140982A1 (en) | Communication Methods and Systems, Electronic Devices, and Computer Clusters | |
| CN102833080A (en) | Process-to-process communication method of application software and system | |
| US20180337915A1 (en) | Role-based resource access control | |
| WO2017076149A1 (en) | Image processing system and image processing method | |
| CN110545324B (en) | Data processing method, device, system, network equipment and storage medium | |
| CN109379245A (en) | A kind of wifi report form generation method and system | |
| CN103324479A (en) | Distributed big-data computation middleware system framework in loose environment | |
| CN115244511A (en) | Distributed computing pipeline processing | |
| CN103338156B (en) | A kind of name pipeline server concurrent communication method based on thread pool | |
| US9614900B1 (en) | Multi-process architecture for a split browser | |
| CN113159970A (en) | Data format conversion method, device, system, electronic equipment and storage medium | |
| CN110288309B (en) | Data interaction method, device, system, computer equipment and storage medium | |
| EP2804103A1 (en) | System and method for forwarding a command stream | |
| CN114780216B (en) | Task scheduling method, device, computer equipment and storage medium | |
| US10884937B2 (en) | Reference cache maintenance optimizer | |
| Zhou | Eager Handlers-communication optimization in Java-based distributed applications with reconfigurable fine-grained code migration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: 523808 19th Floor, Cloud Computing Center, Chinese Academy of Sciences, No. 1 Kehui Road, Songshan Lake Hi-tech Industrial Development Zone, Dongguan City, Guangdong Province Patentee after: G-Cloud Technology Co., Ltd. Address before: 523808 No. 14 Building, Songke Garden, Songshan Lake Science and Technology Industrial Park, Dongguan City, Guangdong Province Patentee before: G-Cloud Technology Co., Ltd. |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20201209 Address after: Room 203, 2 / F, building 6, modern service industry headquarters park, 1769 Yunfeng Avenue, Yunlong demonstration zone, Zhuzhou City, Hunan Province Patentee after: Hunan Zhongke Guoyun big data industry development Co.,Ltd. Address before: 523808 19th floor, Cloud Computing Center, Chinese Academy of Sciences, No.1 Kehui Road, Songshanhu high tech Industrial Development Zone, Dongguan City, Guangdong Province Patentee before: G-CLOUD TECHNOLOGY Co.,Ltd. |