CN112416633A - Method and system for pushing data to external system and realizing data synchronization - Google Patents

Method and system for pushing data to external system and realizing data synchronization Download PDF

Info

Publication number
CN112416633A
CN112416633A CN202011506437.4A CN202011506437A CN112416633A CN 112416633 A CN112416633 A CN 112416633A CN 202011506437 A CN202011506437 A CN 202011506437A CN 112416633 A CN112416633 A CN 112416633A
Authority
CN
China
Prior art keywords
push
data
task
pushing
pushed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202011506437.4A
Other languages
Chinese (zh)
Inventor
杨兴荣
潘昌贵
杨兴海
李建州
吴秀峰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shijihengtong Technology Co ltd
Original Assignee
Shijihengtong Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shijihengtong Technology Co ltd filed Critical Shijihengtong Technology Co ltd
Priority to CN202011506437.4A priority Critical patent/CN112416633A/en
Publication of CN112416633A publication Critical patent/CN112416633A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/54Interprogram communication
    • G06F9/546Message passing systems or structures, e.g. queues
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23Updating
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2457Query processing with adaptation to user needs
    • G06F16/24578Query processing with adaptation to user needs using ranking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • G06F16/9535Search customisation based on user profiles and personalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/50Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • G06F9/5027Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
    • G06F9/5038Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2209/00Indexing scheme relating to G06F9/00
    • G06F2209/54Indexing scheme relating to G06F9/54
    • G06F2209/548Queue

Abstract

The invention discloses a method for pushing data to an external system and realizing data synchronization, which comprises the following steps: in the push data receiving and storing module, the push data is stored in a local database to form a push task, and a push task ID is sent to a push task queuing MQ 2; in a pushed data analysis and execution module, inquiring a database according to a pushed task ID to obtain pushed data and relevant configuration, obtaining a target client system to be pushed after analysis and processing, converting the pushed data into a corresponding format and initiating an HTTP request, and pushing the pushed data to the target client system; monitoring by a pushing delay retry module, and if HTTP request return fails or request response is overtime, carrying out long-time multiple-interval delay processing on the pushing task; monitoring whether a pushing task which is delayed for multiple times at intervals for a long time and fails to be pushed is sent to a processor by a pushing early warning module; the invention also discloses a system for pushing data to an external system and realizing data synchronization.

Description

Method and system for pushing data to external system and realizing data synchronization
Technical Field
The present invention relates to the field of data processing and data communication, and in particular, to a method and system for pushing data to an external system and synchronizing the data.
Background
With the development of internet + technology, more and more industries have become internet, more and more business processes are switched from online to online, and this naturally results in many online business systems. Basically, no matter enterprises, government agencies, social organizations and the like have application systems related to own industries, along with market demands or self development needs, a large amount of business interaction exists between the enterprises, between the enterprises and the government agencies and the like, so that a large amount of data interaction also exists between the application systems of the two parties.
The data interaction of the systems of the two parties not only relates to the mutual data query and call between the systems, but also relates to the data synchronization and callback notification between the systems; in the process of business cooperation, some business data of the system of the party may need to be synchronized to the system of the other party, and some business data of the system of the other party may need to be synchronized to the system of the party so as to realize the consistency of the business data of the systems of the two parties.
Generally speaking, the system data synchronization between the two parties is realized by the way of HTTP interface call; in a conventional processing method, when data push is performed between a system of our party and an external system, a mode of directly calling an external system interface inside a service system is often adopted, that is: when a service system needs to push data to an external system, an HTTP request is directly initiated in the system of the service system, an interface of the external system is called, and the data is directly pushed to an opposite system; the traditional method is direct, but has a great risk, because of the uncontrollable network request and the stability of the external system, the HTTP request initiated by the service system to the external system cannot guarantee one hundred percent success, which relates to the problem of handling the abnormal condition of request failure, if the problem is not handled well, in the subsequent service cooperation, a great number of inconsistent system data of both parties will be generated, and a great amount of later maintenance cost of both parties will be increased.
However, this problem is still troublesome for the service system, if the service system directly initiates an HTTP request to call an interface of an external system, many processing methods are that retry is not performed if the HTTP request fails, and even if retry processing is performed, generally retry is performed only a few times in a short time (for example, retry is performed after 2 seconds, retry is performed after 5 seconds if the HTTP request fails again, retry is performed after 10 seconds if the HTTP request fails again, and only a failure log record can be printed if the HTTP request fails again). This results in inconsistency between the system data of the two parties because the push does not arrive.
In order to solve the technical problem, the present invention provides a method and a system for pushing data to an external system based on data processing to achieve data synchronization between two systems based on the idea.
Disclosure of Invention
In view of the above, an object of the present invention is to provide a method for pushing data to an external system and implementing data synchronization, which is independent from a business system and a client system and can implement synchronization of the pushed data between the business system and the client system after multiple pushing processes.
One of the purposes of the invention is realized by the following technical scheme:
a method for pushing data to an external system and realizing data synchronization comprises the following steps:
(1) in the push data receiving and storing module, when push data in a business service system is written into a data receiving queue MQ1, the push data is taken out from the queue and stored into a local database to form a push task, and the unique ID of the push task is sent into a push task queuing queue MQ2 to be queued;
(2) in a pushed data analysis and execution module, when a pushed task unique identifier ID is written into a pushed task queuing MQ2, a database is inquired according to the pushed task unique identifier ID to obtain pushed data and related configuration, after analysis and processing, a target client system to which the pushed data needs to be pushed is matched, the pushed data is converted into a corresponding format according to a docking protocol with the target client system and an HTTP request is initiated, finally the pushed data is pushed to the target client system, and a short-time multi-instant retry mechanism is started when the HTTP request is initiated;
(3) monitoring a pushing execution result of the pushing data analysis and execution module by the pushing delay retry module, updating data of a local database and setting the state of a pushing task to be a successful state when an initiated HTTP request returns successfully, updating data of the local database and setting the state of the pushing task to be a failed state if the HTTP request returns unsuccessfully or the request response is overtime, and simultaneously carrying out long-time delay processing on the pushing task for multiple times at intervals;
(4) the push early warning module monitors whether the push delayed retry module has a push task of long-time multiple interval delayed push failure or not, and if yes, the event is sent to relevant processing personnel in a communication mode according to relevant configuration.
Further, the multiple interval time in the long-time multiple-interval delay processing is sequentially lengthened.
One of the purposes of the invention is realized by the following technical scheme:
a system for pushing data to external system and realizing data synchronization includes
The push data receiving and storing module: when the push data in the business service system is written into a data receiving queue MQ1 in the module, the push data is taken out from the queue and stored in a local database to form a push task, and the unique identification ID of the push task is sent into a push task queuing queue MQ2 in the module to be queued;
the push data analysis and execution module: when a push task unique identifier ID is written into the push task queuing MQ2, the push data analysis and execution module queries a database according to the push task unique identifier ID to obtain push data and related configuration, and after analysis and processing, matches a target client system to which the push data needs to be pushed, converts the push data into a corresponding format according to a docking protocol with the target client system and initiates an HTTP request, and finally pushes the push data to the target client system, and a short-time multi-instant retry mechanism is started when the HTTP request is initiated.
Push delay retry module: the system comprises a push data analysis and execution module, a push task processing module and a push execution module, wherein the push execution module is used for monitoring a push execution result of the push data analysis and execution module, updating data of a local database and setting the state of the push task as a success state when an initiated HTTP request returns successfully, and updating the data of the local database and setting the state of the push task as a failure state when the HTTP request returns unsuccessfully or a request response is overtime, and meanwhile, the push task is subjected to long-time delay processing for multiple times at intervals;
a pushing early warning module: the system is used for monitoring whether the pushing delayed retry module has a pushing task of long-time multiple interval delayed pushing failure or not, and if so, the event is sent to a relevant processing person in a communication mode according to relevant configuration.
The invention has the beneficial effects that:
the method for pushing data to an external system and realizing data synchronization is independent of a business service system and a client system, the pushed data of all the business service systems are processed by the method in a queue mode, the complexity of the business system for data pushing related processing is reduced to a certain extent, the reliability of the business service system for pushing data to the external client system is ensured by adding a long-time multiple-interval delay retry and early warning notification function, and the synchronization of the pushed data between the business system and the client system is finally realized; the system for pushing data to the external system and realizing data synchronization has the advantages of reasonable structure, high compatibility and strong practicability.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof.
Drawings
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings, in which:
fig. 1 is a schematic structural diagram of a system for pushing data to an external system and implementing data synchronization according to the present invention.
Detailed Description
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are illustrative of the invention only and are not limiting upon the scope of the invention.
A method for pushing data to an external system and realizing data synchronization comprises the following steps:
(1) in the push data receiving and storing module, when push data in a business service system is written into a data receiving queue MQ1, the push data is taken out from the queue and stored into a local database to form a push task, and the unique ID of the push task is sent into a push task queuing queue MQ2 to be queued; the service system generates push data which needs to be synchronized to a corresponding client system;
(2) in the pushed data analysis and execution module, when a pushed task unique identifier ID is written into a pushed task queuing MQ2, inquiring a database according to the pushed task unique identifier ID to obtain pushed data and related configuration (the configuration is a target client system corresponding to the pushed data), matching a target client system to which the pushed data needs to be pushed after analysis and processing, converting the pushed data into a corresponding format according to a docking protocol with the target client system and initiating an HTTP request, and finally pushing the pushed data to the target client system, wherein a short-time multi-instant retry mechanism is started when the HTTP request is initiated;
(3) monitoring a pushing execution result of the pushing data analysis and execution module by the pushing delay retry module, updating data of a local database and setting the state of a pushing task to be a successful state when an initiated HTTP request returns successfully, updating data of the local database and setting the state of the pushing task to be a failed state if the HTTP request returns unsuccessfully or the request response is overtime, and simultaneously carrying out long-time delay processing on the pushing task for multiple times at intervals; the multiple interval time in the long-time multiple-interval delay processing is sequentially lengthened;
(4) and the push early warning module monitors whether the push delayed retry module has a push task of long-time multiple interval delayed push failure or not, and if so, the event is sent to related processing personnel in a mail mode according to related configuration.
As shown in fig. 1, a system for pushing data to an external system and implementing data synchronization is characterized in that: comprises that
The push data receiving and storing module S103: when the push data in the business service system S101 is written into the data receiving queue MQ1 in the module, the push data is taken out from the queue and stored in a local database to form a push task, and the unique identification ID of the push task is sent into the push task queuing queue MQ2 in the module to be queued; the reliability of data transmission between the service system S101 and the push system S102 is ensured by using the abnormal retry mechanism of MQ (e.g., retry after 2 seconds, retry after 5 seconds if failure, retry after 10 seconds if failure).
The pushed data analyzing and executing module S104: when a unique push task identifier ID is written into the push task queuing queue MQ2, the push data analysis and execution module queries the database according to the unique push task identifier ID to obtain push data and related configuration, and after analysis and processing, matches a target client system S105 to which the push data needs to be pushed, converts the push data into a corresponding format according to a docking protocol with the target client system S105 and initiates an HTTP request, and finally pushes the push data to the target client system S105, and when initiating the HTTP request, a short-time multi-instant retry mechanism is enabled, that is, a request fails for 3 retries in a short time, for example, retries once after 2 seconds, retries once after 5 seconds if again fails, and retries once again after 10 seconds if still fails;
push delay retry module S106: the system is used for monitoring the pushing execution result of the pushing data analysis and execution module S104, when the initiated HTTP request returns a success, the data of the local database is updated and the state of the pushing task is set to a success state, if the HTTP request returns a failure or the request response is overtime, the data of the local database is updated and the state of the pushing task is set to a failure state, and meanwhile, the pushing task is subjected to long-time multi-interval delay processing, specifically, 1 hour of delay is performed after the first failure, 3 hours of delay is performed after the second failure, 12 hours of delay is performed after the third failure, and a long-time multi-interval delay retry mechanism increases the fault tolerance of the system, and can effectively reduce the probability of inconsistency of system data between two parties due to the fact that pushing does not arrive. If the three times of push delay retries still fail, sending the push failure event into a push early warning module for processing;
the pushing early warning module S107: the system is used for monitoring whether a pushing delayed retry module has a pushing task of long-time multiple interval delayed pushing failure, if yes, the event is sent to relevant processing personnel in a mail mode according to relevant configuration, the relevant processing personnel can timely troubleshoot the cause of the problem after receiving the early warning mail, and then the pushing can be manually restarted by clicking a relevant function button through a management background page so as to achieve the purpose of manually intervening and repairing the problem.
Finally, the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all of them should be covered by the claims of the present invention.

Claims (3)

1. A method for pushing data to an external system and realizing data synchronization is characterized in that: the method comprises the following steps:
(1) in the push data receiving and storing module, when push data in a business service system is written into a data receiving queue MQ1, the push data is taken out from the queue and stored into a local database to form a push task, and the unique ID of the push task is sent into a push task queuing queue MQ2 to be queued;
(2) in a pushed data analysis and execution module, when a pushed task unique identifier ID is written into a pushed task queuing MQ2, a database is inquired according to the pushed task unique identifier ID to obtain pushed data and related configuration, after analysis and processing, a target client system to which the pushed data needs to be pushed is matched, the pushed data is converted into a corresponding format according to a docking protocol with the target client system and an HTTP request is initiated, finally the pushed data is pushed to the target client system, and a short-time multi-instant retry mechanism is started when the HTTP request is initiated;
(3) monitoring a pushing execution result of the pushing data analysis and execution module by the pushing delay retry module, updating data of a local database and setting the state of a pushing task to be a successful state when an initiated HTTP request returns successfully, updating data of the local database and setting the state of the pushing task to be a failed state if the HTTP request returns unsuccessfully or the request response is overtime, and simultaneously carrying out long-time delay processing on the pushing task for multiple times at intervals;
(4) the push early warning module monitors whether the push delayed retry module has a push task of long-time multiple interval delayed push failure or not, and if yes, the event is sent to relevant processing personnel in a communication mode according to relevant configuration.
2. The method of claim 1, wherein the method comprises the steps of: the multiple interval time in the delay processing of the long-time multiple interval is sequentially lengthened.
3. A system for pushing data to an external system and realizing data synchronization is characterized in that: comprises that
The push data receiving and storing module: when the push data in the business service system is written into a data receiving queue MQ1 in the module, the push data is taken out from the queue and stored in a local database to form a push task, and the unique identification ID of the push task is sent into a push task queuing queue MQ2 in the module to be queued;
the push data analysis and execution module: when a push task unique identifier ID is written into a push task queuing MQ2, a push data analysis and execution module queries a database according to the push task unique identifier ID to acquire push data and related configuration, and matches a target client system to which the push data needs to be pushed after analysis and processing, converts the push data into a corresponding format according to a docking protocol with the target client system and initiates an HTTP request, and finally pushes the push data to the target client system, and a short-time multi-instant retry mechanism is started when the HTTP request is initiated;
push delay retry module: the system comprises a push data analysis and execution module, a push task processing module and a push execution module, wherein the push execution module is used for monitoring a push execution result of the push data analysis and execution module, updating data of a local database and setting the state of the push task as a success state when an initiated HTTP request returns successfully, and updating the data of the local database and setting the state of the push task as a failure state when the HTTP request returns unsuccessfully or a request response is overtime, and meanwhile, the push task is subjected to long-time delay processing for multiple times at intervals;
a pushing early warning module: the system is used for monitoring whether the pushing delayed retry module has a pushing task of long-time multiple interval delayed pushing failure or not, and if so, the event is sent to a relevant processing person in a communication mode according to relevant configuration.
CN202011506437.4A 2020-12-18 2020-12-18 Method and system for pushing data to external system and realizing data synchronization Pending CN112416633A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011506437.4A CN112416633A (en) 2020-12-18 2020-12-18 Method and system for pushing data to external system and realizing data synchronization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011506437.4A CN112416633A (en) 2020-12-18 2020-12-18 Method and system for pushing data to external system and realizing data synchronization

Publications (1)

Publication Number Publication Date
CN112416633A true CN112416633A (en) 2021-02-26

Family

ID=74782720

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011506437.4A Pending CN112416633A (en) 2020-12-18 2020-12-18 Method and system for pushing data to external system and realizing data synchronization

Country Status (1)

Country Link
CN (1) CN112416633A (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030145103A1 (en) * 2002-01-30 2003-07-31 Jim Pruyne Method and system for providing exactly-once semantics for web-based transaction processing
CN1822533A (en) * 2006-03-27 2006-08-23 阿里巴巴公司 Message reliable informing method and system between systems
US20060218237A1 (en) * 2005-03-24 2006-09-28 Bank Of America Corporation Wireless data device with confirmation and retry capabilities for pushed data
CN102523177A (en) * 2011-12-19 2012-06-27 北京新媒传信科技有限公司 Method and system for realization of message push service
CN108337296A (en) * 2018-01-12 2018-07-27 深圳壹账通智能科技有限公司 Message push processing method, device, computer equipment and storage medium
CN109788053A (en) * 2019-01-04 2019-05-21 深圳壹账通智能科技有限公司 Method of data synchronization and system
CN111311142A (en) * 2019-12-31 2020-06-19 江苏苏宁物流有限公司 Customized real-time data efficient pushing method and system

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030145103A1 (en) * 2002-01-30 2003-07-31 Jim Pruyne Method and system for providing exactly-once semantics for web-based transaction processing
US20060218237A1 (en) * 2005-03-24 2006-09-28 Bank Of America Corporation Wireless data device with confirmation and retry capabilities for pushed data
CN1822533A (en) * 2006-03-27 2006-08-23 阿里巴巴公司 Message reliable informing method and system between systems
CN102523177A (en) * 2011-12-19 2012-06-27 北京新媒传信科技有限公司 Method and system for realization of message push service
CN108337296A (en) * 2018-01-12 2018-07-27 深圳壹账通智能科技有限公司 Message push processing method, device, computer equipment and storage medium
CN109788053A (en) * 2019-01-04 2019-05-21 深圳壹账通智能科技有限公司 Method of data synchronization and system
CN111311142A (en) * 2019-12-31 2020-06-19 江苏苏宁物流有限公司 Customized real-time data efficient pushing method and system

Similar Documents

Publication Publication Date Title
US7599476B2 (en) System and method for voice-mail and e-mail synchronization
CN111030784A (en) Information synchronization method and device
JP2018195279A (en) System for transferring facility relevant information at real time and method therefor
CN111464621A (en) Method for detecting message sending and receiving quantity in asynchronous communication of distributed system
US8767707B2 (en) Monitoring a mobile data service associated with a mailbox
CN112416633A (en) Method and system for pushing data to external system and realizing data synchronization
CN101714105A (en) Error reporting and error solving replying system of application program and method thereof
EP1662704B1 (en) Monitoring system, apparatus to be monitored, monitoring apparatus and monitoring method
US20130282846A1 (en) System and method for processing similar emails
JP5427497B2 (en) Mail gateway
CN113794620A (en) Message sending method, device, equipment, system and storage medium
CN112053150A (en) Data processing method, device and storage medium
US20220191256A1 (en) Agnostic data collection platform
KR101043167B1 (en) Interworking System among legacy network management systems and method thereof
CN111666178A (en) Safety monitoring method and system
US20220382618A1 (en) Data Feed Meta Detail Categorization for Confidence
US20220382737A1 (en) Data Feed Meta Detail Categorization for Confidence
CN113535469B (en) Switching method and switching system for disaster recovery database
CN109688198B (en) Distributed system and fault detection method
CN116760741B (en) Data state monitoring method, device, equipment and medium
JP4299572B2 (en) Failure notification method and failure notification method
CN115277366A (en) SLA alarm system based on interface
CN117097692A (en) Session storage system
CN113468024A (en) Visual on-duty emergency disposal interaction method and device
CN113472469A (en) Data synchronization method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination