CN113507497A - Multi-node asynchronous issuing and delay integration method for multi-type data - Google Patents

Multi-node asynchronous issuing and delay integration method for multi-type data Download PDF

Info

Publication number
CN113507497A
CN113507497A CN202110608360.XA CN202110608360A CN113507497A CN 113507497 A CN113507497 A CN 113507497A CN 202110608360 A CN202110608360 A CN 202110608360A CN 113507497 A CN113507497 A CN 113507497A
Authority
CN
China
Prior art keywords
data
network
file
server
sending
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110608360.XA
Other languages
Chinese (zh)
Inventor
刘峥
魏巍
韦入铭
王旦
叶虎
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Changzhou Haoming Information Technology Co ltd
Original Assignee
Changzhou Haoming Information Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Changzhou Haoming Information Technology Co ltd filed Critical Changzhou Haoming Information Technology Co ltd
Priority to CN202110608360.XA priority Critical patent/CN113507497A/en
Publication of CN113507497A publication Critical patent/CN113507497A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/12Avoiding congestion; Recovering from congestion
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/32Flow control; Congestion control by discarding or delaying data units, e.g. packets or frames
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1095Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses a method for multi-node asynchronous issuing and delayed integration of multi-type data, which comprises the following steps: s1: incremental deployment data are generated at a data source server, and each data node server acquires the content of an incremental deployment data file and issues a requirement; s2: the data node server generates a corresponding configuration file; s3: sending the configuration file to each service terminal server to carry out synchronous deployment of real-time increment and delay increment; s4: for delayed data transmission, judging whether the current time interval is a network busy time interval or a network idle time interval; s5: and for the real-time data delivery, judging whether the data delivery is successful, if the data delivery is failed, periodically and circularly detecting the failed data delivery and delaying the data delivery by the data node server in the idle period of the network until all the data delivery is successful. The invention improves the utilization rate of the network in the idle period of the network, and effectively prevents the data file from being missed due to the failure of sending; and the synchronous sharing of real-time data and the delayed integration of non-real-time data are realized.

Description

Multi-node asynchronous issuing and delay integration method for multi-type data
Technical Field
The invention relates to a method for multi-node asynchronous issuing and delayed integration of multi-type data, belonging to the field of data transmission.
Background
With the rapid development of the internet, the appearance of mass data and the change of a data structure bring huge challenges to data processing problems of various industries. The synchronous processing of data is a very common and important data processing task, which involves the process of synchronizing data from one system to another. The data migration process of the data synchronization task is performed manually at the beginning, which not only consumes manpower, but also has limited application places. And then the mode that adopts the automation utensil to carry out data total import appeared, nevertheless along with the increase of data volume, this can bring very big pressure for the network bandwidth, and stability is relatively poor, still has higher to the requirement of service construction.
In order to solve the above problems, the concept of real-time incremental transmission arises, that is, whether incremental data exists at the data source end is monitored in real time, and only incremental data is transmitted each time. However, this data synchronization method has a problem that when transmission fails, data difference occurs between the data source end and the data destination end. Therefore, the solution provided by the prior art is real-time incremental transmission and asynchronous transmission, and delay integration is combined to ensure the data consistency of the data source end and the data destination end.
However, in the prior art, the accuracy of real-time incremental transmission cannot be guaranteed on the basis of the problem of data synchronization, and the accuracy of data synchronization needs to be guaranteed by introducing full-volume transmission occupying a large bandwidth, so that a large pressure is still generated on the network bandwidth along with the increase of the data volume, and the waste of the network bandwidth is caused.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides a multi-node asynchronous issuing and delay integration method for multi-type data, which classifies and uploads the dispersed multi-type heterogeneous data according to network performance and real-time requirements, avoids the backlog phenomenon of uploading data files when the network is busy, reduces the network pressure, improves the synchronous uploading efficiency of the files, further improves the utilization rate of the network during idle time of the network, and effectively prevents the missing transmission of the data files caused by uploading failure.
The invention mainly adopts the technical scheme that:
a method for multi-node asynchronous distribution and delayed integration of multi-type data comprises the following steps:
s1: generating incremental deployment data in a data source server in advance, and acquiring the content of an incremental deployment data file and the issuing requirement of the incremental deployment data file from the data source server by each data node server;
s2: the data node server generates a configuration file of incremental synchronous deployment configuration based on the incremental deployment data information generated in the step S1 according to the load information of the data node server;
s3: sending the configuration file generated in the step S2 to each service terminal server to perform real-time incremental and delay incremental synchronous deployment, so as to implement real-time incremental synchronization and delay incremental synchronization between each data node server;
s4: for delayed issuing data, judging whether the current time interval is a network busy time interval or a network idle time interval, and if the current time interval is the network busy time interval, marking a delayed issuing identifier for the delayed issuing data by the data node server; if the current time period is the network idle time period, executing S5;
s5: and for real-time data sending, the data node server sends a webservice file-based sending request to the service terminal server, and judges whether the sending is successful, if the sending is failed, the data node server periodically and circularly detects the sending failure data and delays the sending data in the idle period of the network, and sends the webservice file-based sending request to the service terminal server for sending until all the sending is successful, if the sending is successful, the data sending is finished.
Preferably, the specific steps of S2 are as follows:
s2-1, acquiring load information of each data node server;
s2-2, acquiring the incremental deployment data information generated in S1;
and S2-3, presetting synchronous deployment configuration of each data node server for real-time incremental data transmission according to the information obtained in S2-1 and S2-2, and generating configuration files according to the synchronous deployment configuration and preset configuration templates, wherein the configuration files comprise types and sizes of the data files and configuration parameters set in the service terminal server and each data node server.
Preferably, the specific steps of S3 are as follows:
s3-1: the service terminal server defines the idle time of the network, the busy time of the network and the request type of the data file of the server;
s3-2: analyzing and judging the data file request type generated by the service terminal server by adopting a data analysis and judgment module, and dividing the data file into real-time issued data and delayed issued data;
s3-3: and cutting the specified file and marking the integrated flag bit.
Preferably, in S3-2, the specific process of analyzing and judging the data file includes the following steps:
s3-21, obtaining the type and size of data file and the configuration parameter set in the service terminal server and each data node server;
s3-22, judging whether the data file of the type in the configuration parameter needs to be cut, if so, cutting by combining the service data and marking an integrated flag bit; if the cutting is not needed, judging whether the current time period is a network idle time period or a network busy time period;
s3-23, if the current time section is the network idle time section, executing S5 to directly issue; if the current time period is the network busy time period, judging whether the type of data file in the configuration parameters needs to be issued in the network busy time period;
s3-24, if the data file of the type in the configuration parameter needs to be issued in busy time period of the network, executing S5 to issue, if the data file does not need to be issued in busy time period of the network, marking delayed issuing, and waiting for the issuing in idle time period of the network.
Preferably, the types of the data files in S3-21 include short video data, file data, structured service information, and a running log.
Preferably, in S3-22, the data file is cut by the following method:
and setting a cutting point threshold according to the size of the data file, and dividing the data file according to the cutting point threshold by combining with the service data requirement.
Preferably, each terminal server of the system combines the cut files to form a complete data file according to the integrated flag bit.
Has the advantages that: the invention provides a multi-node asynchronous issuing and delayed integration method for multi-type data, which evaluates the load of a multi-node server according to the network performance and real-time requirements of the multi-type data, classifies the multi-node server load into real-time issued data and delayed issued data, reduces the busy time of a network, and issues backlog of data files, improves the synchronous issuing efficiency of the files while reducing the network pressure, improves the utilization rate of the network in idle time, and effectively prevents the missed transmission of the data files caused by issuing failure; meanwhile, the main data management system only issues applications once, the system provides real-time and non-real-time analysis, real-time data are synchronously shared, non-real-time data are repeatedly captured through a defined idle period until data sharing is completed, and delay integration is achieved.
Reference numerals
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solutions in the present application, the technical solutions in the embodiments of the present application are clearly and completely described below, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
A method for multi-node asynchronous distribution and delayed integration of multi-type data comprises the following steps:
s1: generating incremental deployment data in a data source server in advance, and acquiring the content of an incremental deployment data file and the issuing requirement of the incremental deployment data file from the data source server by each data node server;
s2: the data node server generates a configuration file of incremental synchronous deployment configuration based on the incremental deployment data information generated in the step S1 according to the load information of the data node server;
s3: sending the configuration file generated in the step S2 to each service terminal server to perform real-time incremental and delay incremental synchronous deployment, so as to implement real-time incremental synchronization and delay incremental synchronization between each data node server;
s4: for delayed issuing data, judging whether the current time interval is a network busy time interval or a network idle time interval, and if the current time interval is the network busy time interval, marking a delayed issuing identifier for the delayed issuing data by the data node server; if the current time period is the network idle time period, executing S5;
s5: and for real-time data sending, the data node server sends a webservice file-based sending request to the service terminal server, and judges whether the sending is successful, if the sending is failed, the data node server periodically and circularly detects the sending failure data and delays the sending data in the idle period of the network, and sends the webservice file-based sending request to the service terminal server for sending until all the sending is successful, if the sending is successful, the data sending is finished.
Preferably, the specific steps of S2 are as follows:
s2-1, acquiring load information of each data node server;
s2-2, acquiring the incremental deployment data information generated in S1;
and S2-3, presetting synchronous deployment configuration of each data node server for real-time incremental data transmission according to the information obtained in S2-1 and S2-2, and generating configuration files according to the synchronous deployment configuration and preset configuration templates, wherein the configuration files comprise types and sizes of the data files and configuration parameters set in the service terminal server and each data node server.
Preferably, the specific steps of S3 are as follows:
s3-1: the service terminal server defines the idle time of the network, the busy time of the network and the request type of the data file of the server;
s3-2: analyzing and judging the data file request type generated by the service terminal server by adopting a data analysis and judgment module, and dividing the data file into real-time issued data and delayed issued data;
s3-3: and cutting the specified file and marking the integrated flag bit.
Preferably, in S3-2, the specific process of analyzing and judging the data file includes the following steps:
s3-21, obtaining the type and size of data file and the configuration parameter set in the service terminal server and each data node server;
s3-22, judging whether the data file of the type in the configuration parameter needs to be cut, if so, cutting by combining the service data and marking an integrated flag bit; if the cutting is not needed, judging whether the current time period is a network idle time period or a network busy time period;
s3-23, if the current time section is the network idle time section, executing S5 to directly issue; if the current time period is the network busy time period, judging whether the type of data file in the configuration parameters needs to be issued in the network busy time period;
s3-24, if the data file of the type in the configuration parameter needs to be issued in busy time period of the network, executing S5 to issue, if the data file does not need to be issued in busy time period of the network, marking delayed issuing, and waiting for the issuing in idle time period of the network.
Preferably, the types of the data files in S3-21 include short video data, file data, structured service information, and a running log.
Preferably, in S3-22, the data file is cut by the following method:
and setting a cutting point threshold according to the size of the data file, and dividing the data file according to the cutting point threshold by combining with the service data requirement.
Preferably, each terminal server of the system combines the cut files to form a complete data file according to the integrated flag bit.
In the invention, the data file analyzing and judging module comprises a data file dividing module which is used for setting a cutting point threshold value according to the size of the data file and dividing the data file according to the cutting point threshold value by combining with the service data requirement.
In the invention, the system monitors the transmission process of incremental data between data node servers; and when the transmission of the incremental data fails, recording the information of the transmission failure of the incremental data, and transmitting the failed data for multiple times according to the configuration file until all the data are transmitted.
The working principle of the invention is as follows:
(1) data synchronization among multiple data node servers is realized based on monitoring nodes, firstly, real-time incremental synchronization deployment is carried out on each data node server to realize real-time incremental synchronization among the data node servers, the minimization of bandwidth occupation in the data synchronization process is realized, meanwhile, the transmission process of incremental data among the nodes is monitored, and when the transmission of the incremental data fails, information of the transmission failure of the incremental data is recorded. On the basis, the incremental data which are failed to be transmitted are conveniently monitored and transmitted by manpower or machines, and the subsequent processing is carried out on the incremental data, so that the accuracy of data synchronization among the data node servers is ensured.
(2) And the service terminal server defines the idle time period of the network, the busy time period of the network and the issuing requirement of each data file.
(3) Analyzing and judging according to a data file generated by a service terminal server, and dividing the data file into real-time issued data and delayed issued data, wherein the delayed issued data service terminal server is marked with a delayed issued identifier; and sending a file issuing request to the service terminal server for the real-time issued data, judging whether the issuing is successful, if the issuing is failed, detecting the issuing failure data and delaying the issuing data by the service terminal server in the idle period of the network, and repeating the steps until the issuing is finished.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims (7)

1. A multi-node asynchronous issuing and delay integration method for multi-type data is characterized by comprising the following steps:
s1: generating incremental deployment data in a data source server in advance, and acquiring the content of an incremental deployment data file and the issuing requirement of the incremental deployment data file from the data source server by each data node server;
s2: the data node server generates a configuration file of incremental synchronous deployment configuration based on the incremental deployment data information generated in the step S1 according to the load information of the data node server;
s3: sending the configuration file generated in the step S2 to each service terminal server to perform real-time incremental and delay incremental synchronous deployment, so as to implement real-time incremental synchronization and delay incremental synchronization between each data node server;
s4: for delayed issuing data, judging whether the current time interval is a network busy time interval or a network idle time interval, and if the current time interval is the network busy time interval, marking a delayed issuing identifier for the delayed issuing data by the data node server; if the current time period is the network idle time period, executing S5;
s5: and for real-time data sending, the data node server sends a webservice file-based sending request to the service terminal server, and judges whether the sending is successful, if the sending is failed, the data node server periodically and circularly detects the sending failure data and delays the sending data in the idle period of the network, and sends the webservice file-based sending request to the service terminal server for sending until all the sending is successful, if the sending is successful, the data sending is finished.
2. The method for multi-node asynchronous distribution and delay integration of multi-type data according to claim 1, wherein the specific steps of S2 are as follows:
s2-1, acquiring load information of each data node server;
s2-2, acquiring the incremental deployment data information generated in S1;
and S2-3, presetting synchronous deployment configuration of each data node server for real-time incremental data transmission according to the information obtained in S2-1 and S2-2, and generating configuration files according to the synchronous deployment configuration and preset configuration templates, wherein the configuration files comprise types and sizes of the data files and configuration parameters set in the service terminal server and each data node server.
3. The method for multi-node asynchronous distribution and delay integration of multi-type data according to claim 1, wherein the specific steps of S3 are as follows:
s3-1: the service terminal server defines the idle time of the network, the busy time of the network and the request type of the data file of the server;
s3-2: analyzing and judging the data file request type generated by the service terminal server by adopting a data analysis and judgment module, and dividing the data file into real-time issued data and delayed issued data;
s3-3: and cutting the specified file and marking the integrated flag bit.
4. The method according to claim 3, wherein the specific process of analyzing and determining the data file in S3-2 includes the following steps:
s3-21, obtaining the type and size of data file and the configuration parameter set in the service terminal server and each data node server;
s3-22, judging whether the data file of the type in the configuration parameter needs to be cut, if so, cutting by combining the service data and marking an integrated flag bit; if the cutting is not needed, judging whether the current time period is a network idle time period or a network busy time period;
s3-23, if the current time section is the network idle time section, executing S5 to directly issue; if the current time period is the network busy time period, judging whether the type of data file in the configuration parameters needs to be issued in the network busy time period;
s3-24, if the data file of the type in the configuration parameter needs to be issued in busy time period of the network, executing S5 to issue, if the data file does not need to be issued in busy time period of the network, marking delayed issuing, and waiting for the issuing in idle time period of the network.
5. The method for multi-node asynchronous distribution and delay integration of multi-type data according to claim 4, wherein the types of the data files in the S3-21 comprise short video data, file data, structured service information and a running log.
6. The method for multi-node asynchronous distribution and delay integration of multi-type data according to claim 1, wherein in S3-22, the data file is cut by:
and setting a cutting point threshold according to the size of the data file, and dividing the data file according to the cutting point threshold by combining with the service data requirement.
7. The method of claim 3, wherein each terminal server of the system merges the cut files into a complete data file according to the integration flag bit.
CN202110608360.XA 2021-06-01 2021-06-01 Multi-node asynchronous issuing and delay integration method for multi-type data Pending CN113507497A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110608360.XA CN113507497A (en) 2021-06-01 2021-06-01 Multi-node asynchronous issuing and delay integration method for multi-type data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110608360.XA CN113507497A (en) 2021-06-01 2021-06-01 Multi-node asynchronous issuing and delay integration method for multi-type data

Publications (1)

Publication Number Publication Date
CN113507497A true CN113507497A (en) 2021-10-15

Family

ID=78008691

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110608360.XA Pending CN113507497A (en) 2021-06-01 2021-06-01 Multi-node asynchronous issuing and delay integration method for multi-type data

Country Status (1)

Country Link
CN (1) CN113507497A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113873045A (en) * 2021-11-11 2021-12-31 深圳市云语科技有限公司 File multi-node transmission method based on capability negotiation

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6202085B1 (en) * 1996-12-06 2001-03-13 Microsoft Corportion System and method for incremental change synchronization between multiple copies of data
WO2017084410A1 (en) * 2015-11-17 2017-05-26 中兴通讯股份有限公司 Network management data synchronization method and apparatus
CN110650185A (en) * 2019-09-04 2020-01-03 安徽省粮油信息中心(安徽粮食批发交易市场管委会) Method and system for asynchronous uploading and delayed integration of multi-type heterogeneous data
CN110941665A (en) * 2019-10-31 2020-03-31 北京浪潮数据技术有限公司 Data synchronization method, data synchronization device and data synchronization equipment between nodes
CN111984729A (en) * 2020-08-14 2020-11-24 北京人大金仓信息技术股份有限公司 Heterogeneous database data synchronization method, device, medium and electronic equipment
CN112507020A (en) * 2020-11-20 2021-03-16 平安普惠企业管理有限公司 Data synchronization method and device, computer equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6202085B1 (en) * 1996-12-06 2001-03-13 Microsoft Corportion System and method for incremental change synchronization between multiple copies of data
WO2017084410A1 (en) * 2015-11-17 2017-05-26 中兴通讯股份有限公司 Network management data synchronization method and apparatus
CN110650185A (en) * 2019-09-04 2020-01-03 安徽省粮油信息中心(安徽粮食批发交易市场管委会) Method and system for asynchronous uploading and delayed integration of multi-type heterogeneous data
CN110941665A (en) * 2019-10-31 2020-03-31 北京浪潮数据技术有限公司 Data synchronization method, data synchronization device and data synchronization equipment between nodes
CN111984729A (en) * 2020-08-14 2020-11-24 北京人大金仓信息技术股份有限公司 Heterogeneous database data synchronization method, device, medium and electronic equipment
CN112507020A (en) * 2020-11-20 2021-03-16 平安普惠企业管理有限公司 Data synchronization method and device, computer equipment and storage medium

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113873045A (en) * 2021-11-11 2021-12-31 深圳市云语科技有限公司 File multi-node transmission method based on capability negotiation
CN113873045B (en) * 2021-11-11 2023-09-12 深圳市云语科技有限公司 File multi-node transmission method based on capability negotiation

Similar Documents

Publication Publication Date Title
CN107729366B (en) Universal multi-source heterogeneous large-scale data synchronization system
US10776159B2 (en) Distributed storage-based filed delivery system and method using calculated dependencies between tasks to ensure consistancy of files
US20150237113A1 (en) Method and system for file transmission
JP2022547748A (en) Methods and apparatus for transmitting data in IoT systems, gateway devices and their storage media
US9818063B2 (en) Forecasting interest in an object over a future period of time using a three-stage time-series analysis process
TWI654563B (en) Computing resource expansion and release method and device thereof for real-time stream computing
CN112507029B (en) Data processing system and data real-time processing method
CN112019605B (en) Data distribution method and system for data stream
CN111970195B (en) Data transmission method and streaming data transmission system
CN113507497A (en) Multi-node asynchronous issuing and delay integration method for multi-type data
CN110650185A (en) Method and system for asynchronous uploading and delayed integration of multi-type heterogeneous data
CN105827448A (en) Resource distribution method and apparatus
CN110995566A (en) Message data pushing method, system and device
CN108093063A (en) Big file multithreading FTP method for uploading
CN115086379B (en) Numerical control machine tool data acquisition method based on edge calculation and virtualization technology
CN105721526A (en) Terminal and server file synchronization methods and device
CN109660753A (en) Resource synchronization method and device
CN109600423B (en) Data synchronization method and device, electronic equipment and storage medium
CN114553852A (en) Rapid uploading method based on asynchronous industrial file server
CN112000544A (en) Real-time monitoring method for large screen of Internet of things equipment
CN111224891A (en) Traffic application identification system and method based on dynamic learning triples
CN103414595B (en) Power dispatch data network link monitoring system topological drawing generating method
CN115391429A (en) Time sequence data processing method and device based on big data cloud computing
CN112256454B (en) Message delay processing method and system
CN113630442B (en) Data transmission method, device and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination