CN103607476A - System and method for distributed multi-stage persistence of huge number of messages - Google Patents

System and method for distributed multi-stage persistence of huge number of messages Download PDF

Info

Publication number
CN103607476A
CN103607476A CN201310654061.5A CN201310654061A CN103607476A CN 103607476 A CN103607476 A CN 103607476A CN 201310654061 A CN201310654061 A CN 201310654061A CN 103607476 A CN103607476 A CN 103607476A
Authority
CN
China
Prior art keywords
message
engine
information
asynchronous
daily record
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201310654061.5A
Other languages
Chinese (zh)
Other versions
CN103607476B (en
Inventor
刘正云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SHANGHAI PUKUN INFORMATION TECHNOLOGY Co Ltd
Original Assignee
SHANGHAI PUKUN INFORMATION TECHNOLOGY Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SHANGHAI PUKUN INFORMATION TECHNOLOGY Co Ltd filed Critical SHANGHAI PUKUN INFORMATION TECHNOLOGY Co Ltd
Priority to CN201310654061.5A priority Critical patent/CN103607476B/en
Publication of CN103607476A publication Critical patent/CN103607476A/en
Application granted granted Critical
Publication of CN103607476B publication Critical patent/CN103607476B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Abstract

The invention relates to a method and system for distributed multi-stage persistence of a huge number of messages. The system comprises a service bus proxy, a service call monitoring subsystem, a log analysis engine and a log storage engine, wherein the service bus proxy is used for storing asynchronous request messages received by a message processing platform in an asynchronous message queue; the service call monitoring subsystem comprises a log receiving engine for receiving log files sent by an asynchronous message sending process; the log analysis engine is used for combining and analyzing the log files; the log storage engine is used for storing statistical information and message information processed by the log analysis engine. By adopting the method and the system with the structure for distributed multi-stage persistence of a huge number of messages, persistent asynchronous storage and persistent distributed storage can be realized by borrowing ideas from asynchronous and distributed file systems, and the performance of the message processing platform is greatly improved through multi-stage persistence, so that the method and the system have wider application ranges.

Description

The system and method for the distributed multistage persistence of magnanimity message
Technical field
The present invention relates to Message Processing platform field, relate in particular to Message Processing platform Message Processing field, specifically refer to a kind of system and method for distributed multistage persistence of magnanimity message.
Background technology
Message Processing platform provides connection maincenter the most basic in network, is the essential elements of constructing nervous system of enterprise, eliminates the technological disparity between different application, allows different application server coordinate operation, has realized communication and integration between different services.
Any on Message Processing platform once calls, and will on platform, produce 1,2,3,4 four record, sees 4 points that Fig. 1 indicates.The maincenter bus that Message Processing platform connects as enterprises service, this statistical information of 4 and message information must be deposited in database.Facilitate query statistic, accounting checking arbitration.
Common implementation is that integrated platform receives business request information, carries out the processing such as persistence message, judgement route, protocol conversion; Then business request information road is transmitted to ISP; ISP receives after business request information, immediately business request information is processed and is returned to integrated platform; Integrated platform is done the processing such as persistence message, message route, protocol conversion again to result; Result is transmitted to service consumer the most at last.As shown in Figure 2.
It is very large that this mode of directly carrying out persistence is affected by database external condition.If database performance is not high, can have a strong impact on the performance of Message Processing platform.
Summary of the invention
The object of the invention is to overcome the shortcoming of above-mentioned prior art, provide a kind of asynchronous storage that can realize persistence and persistence ground distributed memory, by multistage persistence significantly promote Message Processing platform performance, there is the system and method for distributed multistage persistence of the magnanimity message of broader applications scope.
To achieve these goals, the system and method for the distributed multistage persistence of magnanimity message of the present invention has following formation:
The system of the distributed multistage persistence of this magnanimity message, its main feature is that described system comprises:
Service bus agency, is saved in asynchronous information queue in order to the Asynchronous Request message that described Message Processing platform is received;
Service call Monitor And Control Subsystem, described service call Monitor And Control Subsystem comprises:
Daily record receives engine, the journal file sending in order to receive asynchronous information transmission process;
Log analysis engine, merges and analyzing and processing in order to the journal file to described;
Daily record storage engine, statistical information and the message information in order to store described log analysis engine, processed.
Preferably, described daily record storage engine comprises:
Database storage unit, the statistical information of processing in order to store described log analysis engine;
Distributed file system memory cell, the message information of processing in order to store described log analysis engine.
A kind of method that the invention still further relates to distributed multistage persistence of the system magnanimity message based on described, is characterized in that, described method comprises the following steps:
(1) the Asynchronous Request message that described service bus agency receives described Message Processing platform is saved in asynchronous information queue;
(2) described Message Processing platform sends process by asynchronous information journal file is sent to described daily record reception engine;
(3) described log analysis engine carries out merging and analyzing and processing and generates statistical information and message information the journal file of storing in the memory of described service call supervisory control system;
(4) described daily record storage engine is processed generation by described log analysis engine statistical information and message information are preserved.
Preferably, described service bus agency preserves each service call original record of described Message Processing platform, is specially:
The Asynchronous Request message that described service bus agency adopts read-write isolation technics that described Message Processing platform is received is saved in asynchronous information queue.
Preferably, described Message Processing platform sends process by asynchronous information journal file is sent to described daily record reception engine, comprises the steps:
(21) journal file under the Message Processing platform assigned catalogue described in the daily record of service call supervisory control system reception engine automatic search;
(22) described daily record reception engine reads the journal file of described Message Processing platform by asynchronous information transmission process and is stored in the memory of described service call supervisory control system;
(23) described daily record reception engine is deleted the journal file of having stored from described Message Processing platform;
(24) daily record of described service call supervisory control system receives engine and judges whether that follow-up journal file need to read, and if so, continues step (22), otherwise continues step (3).
Preferably, described statistical information comprises service consumer TF, ISP's TF and service bus elapsed time TF.
Preferably, described daily record storage engine comprises database storage unit and distributed system memory cell, and described daily record storage engine is processed by described log analysis engine the statistical information and the message information that generate and preserved, and comprises the following steps:
(41) statistical information that described database storage unit is processed described log analysis engine is stored;
(42) message information that described distributed file system memory cell is processed described log analysis engine is stored.
The system and method that has adopted the distributed multistage persistence of the magnanimity message in this invention, has following beneficial effect:
(1) use for reference asynchronous and thought distributed file system, realize the asynchronous storage of persistence, persistence ground distributed memory.
(2) by two stage persistences, the performance that realizes Message Processing platform significantly promotes, and is applicable to large-scale promotion application.
Accompanying drawing explanation
Fig. 1 is Message Processing platform and service consumer and ISP's the schematic diagram that is related to.
Fig. 2 is the flow process matters figure that directly carries out message persistence in prior art.
Fig. 3 is the structural representation of the system that realizes message persistence of synchronization call.
Fig. 4 is the schematic diagram of system of the distributed multistage persistence of magnanimity message of the present invention.
Fig. 5 is the comparison diagram that synchronization call and asynchronous call are realized the flow process of message persistence.
The test system structure figure of Fig. 6 for the system of the distributed multistage persistence of magnanimity message of the present invention is tested.
Fig. 7 is the data result figure after the system testing of distributed multistage persistence of magnanimity message of the present invention.
Fig. 8 is the data result figure after the system testing of the direct persistence of message.
Embodiment
In order more clearly to describe technology contents of the present invention, below in conjunction with specific embodiment, conduct further description.
The principle that realizes message persistence of synchronization call as shown in Figure 3.
Service bus agency, is responsible for each service call original record of Message Processing platform to preserve, and calls and comprises 1,2 at every turn, 3,4 four records.Agent adopts read-write isolation technics, guarantees that the read-write of journal file does not clash.
The daily record of service call supervisory control system (SMS, Service call monitoring system) receives engine.Journal file under automatic search service bus assigned catalogue.After file reads, daily record is stored in SMS local disk one by one.After daily record is deposited, daily record receives engine this journal file on service bus is deleted, and reads next journal file.
The log analysis engine of SMS, merging and the analyzing and processing of being responsible for carrying out daily record.Because service bus will generate 1,2,3,4 four records, the time is not synchronous.4 messages may be present in a plurality of journal files, need the processing to be combined such as log analysis engine carries out.The message of log analysis engine after being responsible for being combined analyzed in addition, generates statistical information and message information in detail.4 records that produce at service bus, gather and analyze through SMS, become two records, are recorded as the time difference, and first is that T1=" 4-1 " is service consumer TF; Second is that T2=" 3-2 " is ISP's TF, if need to add up ESB elapsed time TF T3=T1-T2.
Service consumer: T1
ISP: T2
Service bus: T3
[original log form]
<: Thread Id >MSG Start:{ESB_SERVER_ID}| $ { REQ_DRIECT}| $ { REQUEST_ID}| $ 2011/07/1914:37:10.012| $ { { URI}.{OPERATIONNAME}|{PROVIDER_URL}}| $ { STATE}| $ { AUD IT_STATE}| $ { IP}| $ { FROM_APP}| $ { SOAPACTION}| $ { HEADER}| $ { BODY}| $ { SERVICE_ID}| $ MSG End
[journal format after analyzing]
request_id|from_app_id|from_ip|service_code|operation|esb_id|timestamp|duration|status
request_id|esb_id|service_url|timestamp|duration|status
The storage engine of SMS is supported DB storage and distributed file system storage, storage engine is responsible for statistical information to be stored in DB, message information is responsible for being stored into distributed file system in detail, and provide flexible and efficient query function, NoSQL technology can be processed the file record of high concurrent and mass data, supports dynamic queries.
Above basic principle is mainly for synchronization call, and two phase process modes of asynchronous call are different.Asynchronous call and synchronization call are distinguished to some extent.As shown in Fig. 5~6, landing of the not responsible original log of agent of service bus, is only responsible for Asynchronous Request to be stored in asynchronous information queue.The original log of 1,2,3,4 four completes by asynchronous information transmission program.
System of the present invention comprises:
Service bus agency, is saved in asynchronous information queue in order to the Asynchronous Request message that Message Processing platform is received;
Service call Monitor And Control Subsystem, service call Monitor And Control Subsystem comprises:
Daily record receives engine, the journal file sending in order to receive asynchronous information transmission process;
Log analysis engine, in order to merge and analyzing and processing journal file;
Daily record storage engine, statistical information and the message information in order to storing daily record analysis engine, processed.
Daily record storage engine comprises:
Database storage unit, the statistical information of processing in order to storing daily record analysis engine;
Distributed file system memory cell, the message information of processing in order to storing daily record analysis engine.
Method of the present invention comprises the following steps:
(1) the Asynchronous Request message that service bus agency adopts read-write isolation technics that Message Processing platform is received is saved in asynchronous information queue;
The Asynchronous Request message that described service bus agency receives described Message Processing platform is saved in asynchronous information queue; Message Processing platform is accepted service request information (A), does not directly call the service that ISP provides, but first service request is sent to asynchronous information queue (ActiveMQ), simultaneously to client feedback response message (D).In this process, the information recording that Message Processing platform is ordered A and D is to journal file.Message Processing platform can obtain pending request simultaneously from asynchronous information queue (ActiveMQ), and to ISP, turn and send request (B), and obtain ISP's response (C). in this process, the information recording that Message Processing platform is ordered B and C is to journal file.Final once 4 journal files of the complete ABCD calling that generate, for SMS process analysis.
(2) Message Processing platform sends process by asynchronous information journal file is sent to daily record reception engine;
Comprise following four sub-steps:
(21) daily record of service call supervisory control system receives the journal file under engine automatic search Message Processing platform assigned catalogue;
(22) daily record reception engine reads the journal file of Message Processing platform by asynchronous information transmission process and is stored in the memory of service call supervisory control system;
(23) daily record reception engine is deleted the journal file of having stored from Message Processing platform;
(24) daily record of service call supervisory control system receives engine and judges whether that follow-up journal file need to read, and if so, continues step (22), otherwise continues step (3).
(3) log analysis engine carries out merging and analyzing and processing and generates statistical information and message information the journal file of storing in the memory of service call supervisory control system;
(4) daily record storage engine is processed generation by log analysis engine statistical information and message information are preserved.
Comprise following two sub-steps:
(41) statistical information that database storage unit is processed log analysis engine is stored;
(42) message information that distributed file system memory cell is processed log analysis engine is stored.
Two stage persistences and the directly contrast of persistence test.
Under above-mentioned same test environment, directly under persistence and two kinds of patterns of two stage persistences, message processing platform is being carried out to performance test contrast.As shown in Figure 7, directly persistence scene as shown in Figure 8 for two stage persistence scenes.Message Processing platform tps under two stage persistences can reach 3200, is more than the twice of direct persistence.
The system and method that has adopted the distributed multistage persistence of the magnanimity message in this invention, has following beneficial effect:
(1) use for reference asynchronous and thought distributed file system, realize the asynchronous storage of persistence, persistence ground distributed memory.
(2) by two stage persistences, the performance that realizes Message Processing platform significantly promotes, and is applicable to large-scale promotion application.
In this specification, the present invention is described with reference to its specific embodiment.But, still can make various modifications and conversion obviously and not deviate from the spirit and scope of the present invention.Therefore, specification and accompanying drawing are regarded in an illustrative, rather than a restrictive.

Claims (7)

1. a system for the distributed multistage persistence of magnanimity message, is characterized in that, described system comprises:
Service bus agency, is saved in asynchronous information queue in order to the Asynchronous Request message that described Message Processing platform is received;
Service call Monitor And Control Subsystem, described service call Monitor And Control Subsystem comprises:
Daily record receives engine, the journal file sending in order to receive asynchronous information transmission process;
Log analysis engine, merges and analyzing and processing in order to the journal file to described;
Daily record storage engine, statistical information and the message information in order to store described log analysis engine, processed.
2. the system of the distributed multistage persistence of magnanimity message according to claim 1, is characterized in that, described daily record storage engine comprises:
Database storage unit, the statistical information of processing in order to store described log analysis engine;
Distributed file system memory cell, the message information of processing in order to store described log analysis engine.
3. a method for the distributed multistage persistence based on system magnanimity message claimed in claim 1, is characterized in that, described method comprises the following steps:
(1) the Asynchronous Request message that described service bus agency adopts read-write isolation technics that described Message Processing platform is received is saved in asynchronous information queue;
(2) described Message Processing platform sends process by asynchronous information journal file is sent to described daily record reception engine;
(3) described log analysis engine carries out merging and analyzing and processing and generates statistical information and message information the journal file of storing in the memory of described service call supervisory control system;
(4) described daily record storage engine is processed generation by described log analysis engine statistical information and message information are preserved.
4. the method for the distributed multistage persistence of magnanimity message according to claim 3, is characterized in that, described service bus agency preserves each service call original record of described Message Processing platform, is specially:
The Asynchronous Request message that described service bus agency adopts read-write isolation technics that described Message Processing platform is received is saved in asynchronous information queue.
5. the method for the distributed multistage persistence of magnanimity message according to claim 3, is characterized in that, described Message Processing platform sends process by asynchronous information journal file is sent to described daily record reception engine, comprises the steps:
(21) journal file under the Message Processing platform assigned catalogue described in the daily record of service call supervisory control system reception engine automatic search;
(22) described daily record reception engine reads the journal file of described Message Processing platform by asynchronous information transmission process and is stored in the memory of described service call supervisory control system;
(23) described daily record reception engine is deleted the journal file of having stored from described Message Processing platform;
(24) daily record of described service call supervisory control system receives engine and judges whether that follow-up journal file need to read, and if so, continues step (22), otherwise continues step (3).
6. the method for the distributed multistage persistence of magnanimity message according to claim 3, is characterized in that, described statistical information comprises service consumer TF, ISP's TF and service bus elapsed time TF.
7. the method for the distributed multistage persistence of magnanimity message according to claim 3, it is characterized in that, described daily record storage engine comprises database storage unit and distributed system memory cell, described daily record storage engine is processed by described log analysis engine the statistical information and the message information that generate and is preserved, and comprises the following steps:
(41) statistical information that described database storage unit is processed described log analysis engine is stored;
(42) message information that described distributed file system memory cell is processed described log analysis engine is stored.
CN201310654061.5A 2013-12-05 2013-12-05 System and method for distributed multi-stage persistence of huge number of messages Active CN103607476B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310654061.5A CN103607476B (en) 2013-12-05 2013-12-05 System and method for distributed multi-stage persistence of huge number of messages

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310654061.5A CN103607476B (en) 2013-12-05 2013-12-05 System and method for distributed multi-stage persistence of huge number of messages

Publications (2)

Publication Number Publication Date
CN103607476A true CN103607476A (en) 2014-02-26
CN103607476B CN103607476B (en) 2017-05-17

Family

ID=50125676

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310654061.5A Active CN103607476B (en) 2013-12-05 2013-12-05 System and method for distributed multi-stage persistence of huge number of messages

Country Status (1)

Country Link
CN (1) CN103607476B (en)

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104408132A (en) * 2014-11-28 2015-03-11 北京京东尚科信息技术有限公司 Data push method and system
CN106354765A (en) * 2016-08-19 2017-01-25 广东亿迅科技有限公司 Log analysis system and method based on distributed collection
CN107678922A (en) * 2017-09-29 2018-02-09 郑州云海信息技术有限公司 Time-consuming management method and relevant apparatus applied to distributed file system
CN110213317A (en) * 2018-07-18 2019-09-06 腾讯科技(深圳)有限公司 The method, apparatus and storage medium of message storage
CN111208946A (en) * 2020-01-06 2020-05-29 北京同有飞骥科技股份有限公司 Data persistence method and system supporting KB-level small file concurrent IO
CN112115112A (en) * 2020-08-10 2020-12-22 上海金仕达软件科技有限公司 Log information processing method and device and electronic equipment
CN112347073A (en) * 2020-10-27 2021-02-09 山东开创云计算有限公司 Multiple data system
CN112667747A (en) * 2020-12-31 2021-04-16 北京赛思信安技术股份有限公司 Dynamic configuration multi-database distributed persistence method supporting user-defined plug-in
CN113542406A (en) * 2021-07-14 2021-10-22 上海浦东发展银行股份有限公司 Asynchronous processing system of intersystem messages
CN115017128A (en) * 2022-05-26 2022-09-06 上海介方信息技术有限公司 Lightweight log service design method and system based on CORBA middleware

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101262498A (en) * 2008-04-22 2008-09-10 华为技术有限公司 A distributed message call method and device
CN101707633A (en) * 2009-11-27 2010-05-12 山东中创软件商用中间件股份有限公司 Message-oriented middleware persistent message storing method based on file system
CN102024032A (en) * 2010-11-29 2011-04-20 广州明朝网络科技有限公司 Distributed data caching and persisting method and system based on Erlang
CN102663543A (en) * 2012-03-22 2012-09-12 北京英孚斯迈特信息技术有限公司 Scheduling system used for enterprise data unification platform
CN103024014A (en) * 2012-12-03 2013-04-03 北京京东世纪贸易有限公司 Method and system for distributing and processing mass data passing through message queues
CN103425462A (en) * 2012-05-14 2013-12-04 阿里巴巴集团控股有限公司 Method and device for workflow data persistence

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101262498A (en) * 2008-04-22 2008-09-10 华为技术有限公司 A distributed message call method and device
CN101707633A (en) * 2009-11-27 2010-05-12 山东中创软件商用中间件股份有限公司 Message-oriented middleware persistent message storing method based on file system
CN102024032A (en) * 2010-11-29 2011-04-20 广州明朝网络科技有限公司 Distributed data caching and persisting method and system based on Erlang
CN102663543A (en) * 2012-03-22 2012-09-12 北京英孚斯迈特信息技术有限公司 Scheduling system used for enterprise data unification platform
CN103425462A (en) * 2012-05-14 2013-12-04 阿里巴巴集团控股有限公司 Method and device for workflow data persistence
CN103024014A (en) * 2012-12-03 2013-04-03 北京京东世纪贸易有限公司 Method and system for distributing and processing mass data passing through message queues

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104408132A (en) * 2014-11-28 2015-03-11 北京京东尚科信息技术有限公司 Data push method and system
CN104408132B (en) * 2014-11-28 2018-03-02 北京京东尚科信息技术有限公司 Data push method and system
CN106354765A (en) * 2016-08-19 2017-01-25 广东亿迅科技有限公司 Log analysis system and method based on distributed collection
CN106354765B (en) * 2016-08-19 2020-06-26 广东亿迅科技有限公司 Log analysis system and method based on distributed acquisition
CN107678922A (en) * 2017-09-29 2018-02-09 郑州云海信息技术有限公司 Time-consuming management method and relevant apparatus applied to distributed file system
CN110213317A (en) * 2018-07-18 2019-09-06 腾讯科技(深圳)有限公司 The method, apparatus and storage medium of message storage
CN111208946A (en) * 2020-01-06 2020-05-29 北京同有飞骥科技股份有限公司 Data persistence method and system supporting KB-level small file concurrent IO
CN112115112A (en) * 2020-08-10 2020-12-22 上海金仕达软件科技有限公司 Log information processing method and device and electronic equipment
CN112347073A (en) * 2020-10-27 2021-02-09 山东开创云计算有限公司 Multiple data system
CN112667747A (en) * 2020-12-31 2021-04-16 北京赛思信安技术股份有限公司 Dynamic configuration multi-database distributed persistence method supporting user-defined plug-in
CN113542406A (en) * 2021-07-14 2021-10-22 上海浦东发展银行股份有限公司 Asynchronous processing system of intersystem messages
CN115017128A (en) * 2022-05-26 2022-09-06 上海介方信息技术有限公司 Lightweight log service design method and system based on CORBA middleware

Also Published As

Publication number Publication date
CN103607476B (en) 2017-05-17

Similar Documents

Publication Publication Date Title
CN103607476A (en) System and method for distributed multi-stage persistence of huge number of messages
CN106375458B (en) Service calling system, method and device
CN111752799A (en) Service link tracking method, device, equipment and storage medium
CN103095819A (en) Data information pushing method and data information pushing system
US11188443B2 (en) Method, apparatus and system for processing log data
CN107888666A (en) A kind of cross-region data-storage system and method for data synchronization and device
CN103795762A (en) Reverse agent test system and method
CN103685590A (en) Method and system for obtaining IP (internet protocol) address
CN112711635B (en) Method for cross-regional data consistency of distributed Internet of things equipment and service cluster
CN109547524A (en) User behavior storage method, device, equipment and storage medium based on Physical Network
CN111831748A (en) Data synchronization method, device and storage medium
CN111209467A (en) Data real-time query system under multi-concurrency multi-channel environment
CN109151056B (en) Method and system for pushing messages based on Canal
CN107040576A (en) Information-pushing method and device, communication system
US20200366967A1 (en) Method and system for monitoring quality of streaming media
CN116204438A (en) Test case generation method, automatic test method and related device
CN1971599A (en) Error monitoring method and system of software application
CN113127232A (en) Message processing method, device, equipment and storage medium
CN109389271B (en) Application performance management method and system
CN111935316B (en) Method and device for acquiring front-end equipment catalog
CN104462235A (en) Internet of Things general event service mechanism based on Restful Web Service
CN103067375A (en) Method and device of connection session among network nodes
CN110740151B (en) Micro-service adjusting method, device, server and computer storage medium
KR101874484B1 (en) A system and method for extracting, transforming, loading spatial data based on stream network
CN107547601B (en) Method for sending message, routing equipment, server and system

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant