CN103607476B - System and method for distributed multi-stage persistence of huge number of messages - Google Patents
System and method for distributed multi-stage persistence of huge number of messages Download PDFInfo
- Publication number
- CN103607476B CN103607476B CN201310654061.5A CN201310654061A CN103607476B CN 103607476 B CN103607476 B CN 103607476B CN 201310654061 A CN201310654061 A CN 201310654061A CN 103607476 B CN103607476 B CN 103607476B
- Authority
- CN
- China
- Prior art keywords
- message
- engine
- information
- daily record
- asynchronous
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a method and system for distributed multi-stage persistence of a huge number of messages. The system comprises a service bus proxy, a service call monitoring subsystem, a log analysis engine and a log storage engine, wherein the service bus proxy is used for storing asynchronous request messages received by a message processing platform in an asynchronous message queue; the service call monitoring subsystem comprises a log receiving engine for receiving log files sent by an asynchronous message sending process; the log analysis engine is used for combining and analyzing the log files; the log storage engine is used for storing statistical information and message information processed by the log analysis engine. By adopting the method and the system with the structure for distributed multi-stage persistence of a huge number of messages, persistent asynchronous storage and persistent distributed storage can be realized by borrowing ideas from asynchronous and distributed file systems, and the performance of the message processing platform is greatly improved through multi-stage persistence, so that the method and the system have wider application ranges.
Description
Technical field
The present invention relates to message processing platform field, more particularly to message processing platform Message Processing field, specifically refer to
A kind of system and method for the distributed multistage persistence of magnanimity message.
Background technology
Message processing platform, there is provided connection maincenter most basic in network, is necessity unit for constructing nervous system of enterprise
Element, eliminates the technological disparity between different application, allows different application server coordinate operations, realizes between different services
Communication and integration.
Any on message processing platform once calls, and 1,2,3,4 four records will be produced on platform, sees that Fig. 1 is indicated
4 points.The maincenter bus that message processing platform connects as enterprises service, this 4 points statistical information and message information it is necessary
It is stored in data base.Facilitate query statistic, accounting checking arbitration.
It is common to be achieved in that integrated platform receives business request information, carry out persistent message, judge route, association
View conversion etc. is processed；Then business request information road is transmitted to into ISP；ISP receives service request and disappears
After breath, process is carried out to business request information immediately and returns to integrated platform；Integrated platform does persistence to result again and disappears
Breath, message route, protocol conversion etc. are processed；Most result is transmitted to service consumer at last.As shown in Figure 2.
This mode for directly carrying out persistence is affected very big by database external condition.If database performance is not
Height, can have a strong impact on the performance of message processing platform.
The content of the invention
The purpose of the present invention is the shortcoming for overcoming above-mentioned prior art, there is provided a kind of to realize the asynchronous of persistence
Storage and persistence ground distributed memory, the performance of message processing platform is substantially improved by multistage persistence, with wider
The system and method for the distributed multistage persistence of the magnanimity message of general range of application.
To achieve these goals, the system and method for the distributed multistage persistence of magnanimity message of the invention has
It is following to constitute：
The system of the distributed multistage persistence of the magnanimity message, which is mainly characterized by, and described system includes：
Service bus is acted on behalf of, and is preserved to asynchronous letter to the Asynchronous Request message for receiving described message processing platform
In breath queue；
Service call Monitor And Control Subsystem, described service call Monitor And Control Subsystem include：
Daily record receives engine, sends the journal file that process sends to receive asynchronous information；
Log analysis engine, described journal file is merged and be analyzed and processed；
Engine is stored in daily record, to store statistical information and the message information that described log analysis engine is processed.
It is preferred that described daily record storage engine includes：
Database storage unit, to store the statistical information that described log analysis engine is processed；
Distributed file system memory element, to store the message information that described log analysis engine is processed.
The invention further relates to a kind of method of the distributed multistage persistence based on described system magnanimity message, which is special
Levy and be, described method is comprised the following steps：
（1）The Asynchronous Request message that described message processing platform is received by described service bus agency is preserved to different
In step message queue；
（2）Described message processing platform is sent journal file to described daily record by asynchronous information transmission process and is connect
Spasm is held up；
（3）The journal file that described log analysis engine is stored in the memorizer to described service call monitoring system
Merge and analyze and process and generate statistical information and message information；
（4）Described log analysis engine is processed described daily record storage engine the statistical information and message information for generating
Preserved.
It is preferred that each service call protocol of described message processing platform is entered by described service bus agency
Row is preserved, specially：
Described service bus agency using read and write abruption technology by described message processing platform receive it is asynchronous please
Message is asked to preserve into asynchronous information queue.
It is preferred that described message processing platform sends process by asynchronous information journal file was sent to described day
Will receives engine, comprises the steps：
（21）The daily record of service call monitoring system receives the engine described message processing platform assigned catalogue of search automatically
Under journal file；
（22）Described daily record receives engine and sends the day that process reads described message processing platform by asynchronous information
Will file is simultaneously stored into the memorizer of described service call monitoring system；
（23）Described daily record receives engine and the journal file for having stored is deleted from described message processing platform；
（24）The daily record of described service call monitoring system receives engine and determines whether follow-up journal file needs
Read, if it is, continuing step（22）, otherwise continue step（3）.
It is preferred that described statistical information includes service consumer TF, ISP's TF and service bus
Elapsed time TF.
It is preferred that described daily record storage engine includes database storage unit and distributed system memory element, it is described
Daily record storage engine described log analysis engine is processed the statistical information that generates and message information is preserved, including with
Lower step：
（41）The statistical information that described log analysis engine is processed is stored by described database storage unit；
（42）The message information that described log analysis engine is processed is entered by described distributed file system memory element
Row storage.
The system and method for the distributed multistage persistence of magnanimity message in the invention is employed, with following beneficial
Effect：
（1）The thought of asynchronous and distributed file system is used for reference, the asynchronous storage of persistence is realized, persistence ground is distributed
Storage.
（2）By two benches persistence, realize that the performance of message processing platform is substantially improved, it is adaptable to which large-scale promotion should
With.
Description of the drawings
Fig. 1 is message processing platform and service consumer and the relation schematic diagram of ISP.
Fig. 2 is directly to carry out the flow process matters figure of message duration in prior art.
Structural representations of the Fig. 3 for the system for realizing message duration of synchronization call.
Fig. 4 is the schematic diagram of the system of the distributed multistage persistence of the magnanimity message of the present invention.
Fig. 5 is the comparison diagram of the flow process that synchronization call and asynchronous call realize message duration.
Fig. 6 is the test system knot tested by the system of the distributed multistage persistence of the magnanimity message to the present invention
Composition.
Fig. 7 is the data result figure after the system test of the distributed multistage persistence of the magnanimity message of the present invention.
Fig. 8 is the data result figure after the system test of the direct persistence of message.
Specific embodiment
In order to more clearly describe the technology contents of the present invention, carry out with reference to specific embodiment further
Description.
The principle for realizing message duration of synchronization call is as shown in Figure 3.
Service bus is acted on behalf of, and is responsible for preserving each service call protocol of message processing platform, is adjusted every time
With including 1,2,3,4 four records.Agent adopts read and write abruption technology, it is ensured that the read-write of journal file is not clashed.
Service call monitoring system（SMS, Service call monitoring system）Daily record receive engine.From
Journal file under dynamic search service bus assigned catalogue.After file reads, daily record is stored in SMS this earth magnetism one by one
In disk.After daily record storage, daily record receives engine and deletes the journal file on service bus, and reads next daily record text
Part.
The log analysis engine of SMS, is responsible for carrying out merging and the analyzing and processing of daily record.As service bus will generate 1,2,
3,4 four points of records, the time is not synchronous.4 points of messages are likely to be present in multiple journal files, need log analysis engine to enter
Row waits merging treatment.In addition log analysis engine is responsible for being analyzed the message after merging, generates statistical information and in detail
Message information.In 4 records that service bus is produced, gather and analyze through SMS, become two records, be recorded as time difference,
First is T1=" 4-1 " i.e. service consumer TFs；Second is T2=" 3-2 " i.e. ISP's TFs, if needed
Count ESB elapsed time TF T3=T1-T2.
Service consumer：T1
ISP：T2
Service bus：T3
【Original log form】
<:Thread Id>MSG Start:{ESB_SERVER_ID}|${REQ_DRIECT}|${REQUEST_ID}|$2011/
07/1914:37:10.012|${{URI}.{OPERATIONNAME}|{PROVIDER_URL}}|${STATE}|${AUD IT_
STATE}|${IP}|${FROM_APP}|${SOAPACTION}|${HEADER}|${BODY}|${SERVICE_ID}|$MSG
End
【Journal format after analysis】
request_id|from_app_id|from_ip|service_code|operation|esb_id|
timestamp|duration|status
request_id|esb_id|service_url|timestamp|duration|status
The storage engine of SMS supports that DB storages and distributed file system storage, storage engine are responsible for statistical information to deposit
Store in DB, detailed message information is responsible for being stored into distributed file system, and provides flexible and efficient query function, NoSQL
Technology can process the file record of high concurrent and mass data, support dynamic queries.
, mainly for synchronization call, the two-stage processing mode of asynchronous call is different for above ultimate principle.Asynchronous tune
With otherwise varied with synchronization call.As shown in Fig. 5~6, the landing of the not responsible original logs of agent of service bus is only responsible for
Asynchronous Request is stored in asynchronous information queue.1,2,3,4 four point of original log is complete by asynchronous information router
Into.
The system of the present invention includes：
Service bus is acted on behalf of, and is preserved to asynchronous information queue to the Asynchronous Request message for receiving message processing platform
In；
Service call Monitor And Control Subsystem, service call Monitor And Control Subsystem include：
Daily record receives engine, sends the journal file that process sends to receive asynchronous information；
Log analysis engine, to merge to journal file and analyze and process；
Engine is stored in daily record, to the statistical information and message information that store the process of log analysis engine.
Daily record storage engine includes：
Database storage unit, to the statistical information for storing the process of log analysis engine；
Distributed file system memory element, to the message information for storing the process of log analysis engine.
The method of the present invention is comprised the following steps：
（1）Service bus agency is preserved using the Asynchronous Request message that message processing platform is received by read and write abruption technology
Into asynchronous information queue；
The Asynchronous Request message that described message processing platform is received by described service bus agency is preserved to asynchronous
In message queue；Message processing platform receives service request information (A), does not directly invoke the service of ISP's offer,
But first service request is sent to asynchronous information queue (ActiveMQ), while to client feedback response message (D).Here
During, message processing platform is by the information record of A and D points to journal file.Message processing platform simultaneously can be from asynchronous information
Pending request is obtained in queue (ActiveMQ), and request (B) is sent to ISP's forwarding, and obtain service provide
The response (C) of person. in the process, message processing platform is by the information record of B and C points to journal file.Ultimately generate once
4 points of journal files of the ABCD for completely calling, for SMS process analysises.
（2）Message processing platform sends process by asynchronous information and journal file is sent to daily record reception engine；
Including following four sub-step：
（21）The daily record of service call monitoring system receives the day under the automatic search message processing platform assigned catalogue of engine
Will file；
（22）Daily record receives engine and sends the journal file of process reading message processing platform by asynchronous information and store
Into the memorizer of service call monitoring system；
（23）Daily record receives engine and the journal file for having stored is deleted from message processing platform；
（24）The daily record of service call monitoring system receives engine and determines whether that follow-up journal file needs to read, such as
Fruit is then to continue step（22）, otherwise continue step（3）.
（3）The journal file that log analysis engine is stored in the memorizer to service call monitoring system is merged and is divided
Analysis processes and generates statistical information and message information；
（4）Daily record is stored engine and preserves the statistical information and message information of the process generation of log analysis engine.
Including following two sub-steps：
（41）The statistical information that log analysis engine is processed is stored by database storage unit；
（42）The message information that log analysis engine is processed is stored by distributed file system memory element.
Two benches persistence and the contrast of direct persistence test.
Under above-mentioned same test environment, to Message Processing under direct persistence and two benches persistence both of which
Platform has carried out performance test contrast.Two benches persistence scene is as shown in fig. 7, directly persistence scene is as shown in Figure 8.Two ranks
Message processing platform tps under section persistence can reach 3200, be more than the twice of direct persistence.
The system and method for the distributed multistage persistence of magnanimity message in the invention is employed, with following beneficial
Effect：
（1）The thought of asynchronous and distributed file system is used for reference, the asynchronous storage of persistence is realized, persistence ground is distributed
Storage.
（2）By two benches persistence, realize that the performance of message processing platform is substantially improved, it is adaptable to which large-scale promotion should
With.
In this description, the present invention is described with reference to its specific embodiment.But it is clear that can still make
Various modifications and alterations are without departing from the spirit and scope of the present invention.Therefore, specification and drawings are considered as illustrative
And it is nonrestrictive.
Claims (6)
1. the system of the distributed multistage persistence of a kind of magnanimity message, it is characterised in that described system includes：
Service bus is acted on behalf of, and is preserved into asynchronous information queue to the Asynchronous Request message for receiving message processing platform；
Service call Monitor And Control Subsystem, described service call Monitor And Control Subsystem include：
Daily record receives engine, sends the journal file that process sends to receive asynchronous information；
Log analysis engine, described journal file is merged and be analyzed and processed；
Engine is stored in daily record, to store statistical information and the message information that described log analysis engine is processed；
Described system carries out following process：
(1) Asynchronous Request that described message processing platform is received by the service bus agency described in using read and write abruption technology
Message is preserved into asynchronous information queue；
(2) message processing platform described in sends process and sends to described daily record to receive by journal file by asynchronous information and draws
Hold up；
(3) journal file that the log analysis engine described in is stored in the memorizer to described service call Monitor And Control Subsystem enters
Row merges and analyzes and processes and generate statistical information and message information；
(4) statistical information and message information that described log analysis engine processes generation are carried out by the daily record storage engine described in
Preserve.
2. the system of the distributed multistage persistence of magnanimity message according to claim 1, it is characterised in that described
Daily record storage engine includes：
Database storage unit, to store the statistical information that described log analysis engine is processed；
Distributed file system memory element, to store the message information that described log analysis engine is processed.
3. a kind of method of the distributed multistage persistence of the system magnanimity message based on described in claim 1, its feature exist
In described method is comprised the following steps：
(1) Asynchronous Request that described message processing platform is received by the service bus agency described in using read and write abruption technology
Message is preserved into asynchronous information queue；
(2) message processing platform described in sends process and sends to described daily record to receive by journal file by asynchronous information and draws
Hold up；
(3) journal file that the log analysis engine described in is stored in the memorizer to described service call Monitor And Control Subsystem enters
Row merges and analyzes and processes and generate statistical information and message information；
(4) statistical information and message information that described log analysis engine processes generation are carried out by the daily record storage engine described in
Preserve.
4. the method for the distributed multistage persistence of magnanimity message according to claim 3, it is characterised in that described
Message processing platform sends process by asynchronous information and journal file is sent to described daily record reception engine, including following step
Suddenly：
(21) daily record of service call monitoring system is received under message processing platform assigned catalogue of the engine automatically described in search
Journal file；
(22) daily record described in receives engine and sends the daily record text that process reads described message processing platform by asynchronous information
Part is simultaneously stored into the memorizer of described service call monitoring system；
(23) daily record described in receives engine and the journal file for having stored is deleted from described message processing platform；
(24) daily record of the service call monitoring system described in receives engine and determines whether that follow-up journal file needs to read,
If it is, continuing step (22), otherwise continue step (3).
5. the method for the distributed multistage persistence of magnanimity message according to claim 3, it is characterised in that described
Statistical information includes service consumer TF, ISP's TF and service bus elapsed time TF.
6. the method for the distributed multistage persistence of magnanimity message according to claim 3, it is characterised in that described
Daily record storage engine includes database storage unit and distributed system memory element, and described daily record storage engine will be described
The statistical information and message information that the process of log analysis engine is generated is preserved, and is comprised the following steps：
(41) statistical information that described log analysis engine is processed is stored by the database storage unit described in；
(42) message information that described log analysis engine is processed is stored by the distributed system memory element described in.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310654061.5A CN103607476B (en) | 2013-12-05 | 2013-12-05 | System and method for distributed multi-stage persistence of huge number of messages |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310654061.5A CN103607476B (en) | 2013-12-05 | 2013-12-05 | System and method for distributed multi-stage persistence of huge number of messages |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103607476A CN103607476A (en) | 2014-02-26 |
| CN103607476B true CN103607476B (en) | 2017-05-17 |
Family
ID=50125676
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310654061.5A Active CN103607476B (en) | 2013-12-05 | 2013-12-05 | System and method for distributed multi-stage persistence of huge number of messages |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103607476B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104408132B (en) * | 2014-11-28 | 2018-03-02 | 北京京东尚科信息技术有限公司 | Data push method and system |
| CN106354765B (en) * | 2016-08-19 | 2020-06-26 | 广东亿迅科技有限公司 | Log analysis system and method based on distributed acquisition |
| CN107678922A (en) * | 2017-09-29 | 2018-02-09 | 郑州云海信息技术有限公司 | Time-consuming management method and relevant apparatus applied to distributed file system |
| CN110213317B (en) * | 2018-07-18 | 2021-10-29 | 腾讯科技（深圳）有限公司 | Message storage method, device and storage medium |
| CN111208946A (en) * | 2020-01-06 | 2020-05-29 | 北京同有飞骥科技股份有限公司 | Data persistence method and system supporting KB-level small file concurrent IO |
| CN112115112A (en) * | 2020-08-10 | 2020-12-22 | 上海金仕达软件科技有限公司 | Log information processing method and device and electronic equipment |
| CN112347073A (en) * | 2020-10-27 | 2021-02-09 | 山东开创云计算有限公司 | Multiple data system |
| CN112667747B (en) * | 2020-12-31 | 2021-09-21 | 北京赛思信安技术股份有限公司 | Dynamic configuration multi-database distributed persistence method supporting user-defined plug-in |
| CN113542406A (en) * | 2021-07-14 | 2021-10-22 | 上海浦东发展银行股份有限公司 | Asynchronous processing system of intersystem messages |
| CN115017128B (en) * | 2022-05-26 | 2023-02-10 | 上海介方信息技术有限公司 | Lightweight log service design method and system based on CORBA middleware |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101262498A (en) * | 2008-04-22 | 2008-09-10 | 华为技术有限公司 | A distributed message call method and device |
| CN101707633A (en) * | 2009-11-27 | 2010-05-12 | 山东中创软件商用中间件股份有限公司 | Message-oriented middleware persistent message storing method based on file system |
| CN102024032A (en) * | 2010-11-29 | 2011-04-20 | 广州明朝网络科技有限公司 | Distributed data caching and persisting method and system based on Erlang |
| CN102663543A (en) * | 2012-03-22 | 2012-09-12 | 北京英孚斯迈特信息技术有限公司 | Scheduling system used for enterprise data unification platform |
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
| CN103425462A (en) * | 2012-05-14 | 2013-12-04 | 阿里巴巴集团控股有限公司 | Method and device for workflow data persistence |
-
2013
- 2013-12-05 CN CN201310654061.5A patent/CN103607476B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101262498A (en) * | 2008-04-22 | 2008-09-10 | 华为技术有限公司 | A distributed message call method and device |
| CN101707633A (en) * | 2009-11-27 | 2010-05-12 | 山东中创软件商用中间件股份有限公司 | Message-oriented middleware persistent message storing method based on file system |
| CN102024032A (en) * | 2010-11-29 | 2011-04-20 | 广州明朝网络科技有限公司 | Distributed data caching and persisting method and system based on Erlang |
| CN102663543A (en) * | 2012-03-22 | 2012-09-12 | 北京英孚斯迈特信息技术有限公司 | Scheduling system used for enterprise data unification platform |
| CN103425462A (en) * | 2012-05-14 | 2013-12-04 | 阿里巴巴集团控股有限公司 | Method and device for workflow data persistence |
| CN103024014A (en) * | 2012-12-03 | 2013-04-03 | 北京京东世纪贸易有限公司 | Method and system for distributing and processing mass data passing through message queues |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103607476A (en) | 2014-02-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103607476B (en) | System and method for distributed multi-stage persistence of huge number of messages | |
| KR102029277B1 (en) | Device Identifier-Dependent Operation Processing for Packet-Based Data Communications | |
| CN100449543C (en) | Method and apparatus for holding journal | |
| CN110300056B (en) | Real-time response type API gateway and request processing method thereof | |
| CN102655481B (en) | A kind of instant messaging chat content inspection method based on webpage and system | |
| CN103905497B (en) | Realize the method, apparatus and application platform of third-party application business website log | |
| CN102737065B (en) | Method and device for acquiring data | |
| US20080144655A1 (en) | Systems, methods, and computer program products for passively transforming internet protocol (IP) network traffic | |
| US11265279B2 (en) | Enhancing messages with dynamic content | |
| EP2779583B1 (en) | Telecommunication method and system | |
| CN107133161B (en) | Method and device for generating client performance test script | |
| CN103166827A (en) | Method and system for user behavior data reporting | |
| US20120036208A1 (en) | Polling-based secure network message notification system and method with performance enhancing features | |
| CN104410546A (en) | Testing method and device of real-time processing system | |
| CN102833125A (en) | Test server, test system adopting test server, and test method | |
| US20200366967A1 (en) | Method and system for monitoring quality of streaming media | |
| CN103685354A (en) | Method and device for testing based on RMI protocol | |
| US20190005534A1 (en) | Providing media assets to subscribers of a messaging system | |
| CN111708611B (en) | Lightweight Kubernetes monitoring system and method | |
| CN101068237B (en) | Data access system and data access method | |
| CN104394041A (en) | Access log generation method and device | |
| CN112764795A (en) | Service-oriented cross-platform and cross-terminal information publishing system and method | |
| US9652309B2 (en) | Mediator with interleaved static and dynamic routing | |
| CN101296201B (en) | Network information sharing method, system and instant communication device | |
| US9590934B2 (en) | Method and system of creating a graylist for message transmission |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |