CN111817890A - Data synchronization processing method and device, computer equipment and storage medium - Google Patents

Data synchronization processing method and device, computer equipment and storage medium Download PDF

Info

Publication number
CN111817890A
CN111817890A CN202010644993.1A CN202010644993A CN111817890A CN 111817890 A CN111817890 A CN 111817890A CN 202010644993 A CN202010644993 A CN 202010644993A CN 111817890 A CN111817890 A CN 111817890A
Authority
CN
China
Prior art keywords
scheduling
real
receiving
bandwidth
time residual
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010644993.1A
Other languages
Chinese (zh)
Other versions
CN111817890B (en
Inventor
严莉
刘荫
韩圣亚
徐浩
张悦
殷齐林
黄振
刘子雁
张凯
朱韶松
倪金超
汤琳琳
刘学
马领
李琪
杨冠群
崔博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
State Grid Corp of China SGCC
Information and Telecommunication Branch of State Grid Shandong Electric Power Co Ltd
Original Assignee
State Grid Corp of China SGCC
Information and Telecommunication Branch of State Grid Shandong Electric Power Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by State Grid Corp of China SGCC, Information and Telecommunication Branch of State Grid Shandong Electric Power Co Ltd filed Critical State Grid Corp of China SGCC
Priority to CN202010644993.1A priority Critical patent/CN111817890B/en
Publication of CN111817890A publication Critical patent/CN111817890A/en
Application granted granted Critical
Publication of CN111817890B publication Critical patent/CN111817890B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0896—Bandwidth or capacity management, i.e. automatically increasing or decreasing capacities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G06F16/275—Synchronous replication
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the application provides a data synchronous processing method and device, computer equipment and a storage medium, relates to the technical field of data processing, and is used for solving the bandwidth competition problem when a multi-source server performs data backup on a single target server. The method mainly comprises the following steps: receiving a process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process; receiving a process RPiCalculating MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value; if real-time residual bandwidth BiIf the value is less than the preset threshold value, the process is receivedRPiSending request scheduling information to a scheduling process, wherein the scheduling information is used for requesting the scheduling process to send a receiving process RPiScheduling bandwidth resources; if real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiBandwidth resources within.

Description

Data synchronization processing method and device, computer equipment and storage medium
Technical Field
The embodiment of the application relates to the technical field of data processing, in particular to a data synchronous processing method and device, computer equipment and a storage medium.
Background
The golden gate software is a log-based structured data replication software. The golden gate can realize real-time capture, transformation and delivery of a large amount of transaction data, realize data synchronization of the source server and the target server and keep data delay of a sub-second level.
Currently, the golden gate software has the capability of backing up a multi-source server to a single target server, but the backup capability only relates to the condition that the amount of traffic of the source server in a single time interval corresponding to the target server is small. When the number of the source servers is far greater than that of the corresponding target servers, for example, hundreds or thousands of source servers are required to back up to the target servers in the power grid system, the bandwidth competition problem between the source servers still exists by using the own backup function of the golden gate software. Especially, when the local bandwidth is in shortage, how to achieve timely data backup becomes a problem which needs to be solved urgently at present.
Disclosure of Invention
The embodiment of the application provides a data synchronization processing method, equipment, a system, computer equipment and a storage medium, which are used for solving the bandwidth competition problem when a multi-source server performs data backup on a single target server.
The embodiment of the invention provides a data synchronous processing method, which is applied to a target server, wherein the target server comprises a scheduling process and K receiving processes, and the receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with the K receiving process buses, and the method comprises the following steps:
the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
if the real-time residual bandwidth BiIf the value is smaller than the preset threshold value, the receiving process RP isiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
if the real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
The embodiment of the invention provides a data synchronous processing device, which is applied to a target server, wherein the target server comprises a scheduling process and K receiving processes, and the receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with the K receiving process buses, and the device comprises:
a receiving module for the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
a calculation module for the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
a first scheduling module for determining if the real-time residual bandwidth BiLess than said presetThreshold value, then the receiving process RPiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
a second scheduling module for determining if the real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the above data synchronization processing method when executing the computer program.
A computer-readable storage medium storing a computer program which, when executed by a processor, implements the above-described data synchronization processing method.
The invention provides a data synchronous processing method, a device, computer equipment and a storage medium, which are applied to a target server, wherein the target server comprises a scheduling process and K receiving processes, and the receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with K receiving process buses, and the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process; receiving a process RPiCalculating MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value; if real-time residual bandwidth BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process, wherein the scheduling information is used for requesting the scheduling process to send a receiving process RPiScheduling bandwidth resources; if real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiBandwidth resources within. Namely the invention according toReal-time residual bandwidth BiTo determine the corresponding scheduling scheme, in particular, the real-time remaining bandwidth BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process; real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiThe invention solves the problem of bandwidth competition when the multi-source server performs data backup to a single target server.
Drawings
FIG. 1 is a block diagram of a data synchronization processing system according to an embodiment of the present application;
fig. 2 is a flowchart of a data synchronization processing method according to an embodiment of the present application;
FIG. 3 is a block diagram of a data synchronization process according to an embodiment of the present application;
fig. 4 is a schematic diagram of a computer device according to an embodiment of the present application.
Detailed Description
In order to better understand the technical solutions described above, the technical solutions of the embodiments of the present application are described in detail below with reference to the drawings and the specific embodiments, and it should be understood that the specific features of the embodiments and the embodiments of the present application are detailed descriptions of the technical solutions of the embodiments of the present application, and are not limitations of the technical solutions of the present application, and the technical features of the embodiments and the embodiments of the present application may be combined with each other without conflict.
The golden gate software is a log-based structured data replication software. The golden gate can realize real-time capture, transformation and delivery of a large amount of transaction data, realize data synchronization of a source database and a target database and keep data delay of a sub-second level. Oracle Golden Gate (OGG) can implement real-time transaction change data capture, conversion and transmission in diverse and complex IT architectures; the data processing and exchange are in transaction units, and support heterogeneous platforms, for example: DB2, MSSQL, etc.
However, in the OGG, the bandwidth competition problem is faced when the multi-source server performs data backup to a single target server. Especially, when the local bandwidth is in shortage, how to achieve the purpose of data backup in time.
In order to solve the above problems, the present invention provides a data synchronization processing method, apparatus, computer device and storage medium, wherein the present invention is based on the real-time residual bandwidth BiDetermining the corresponding scheduling scheme, specifically, if the real-time residual bandwidth is BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process; if real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiThe invention solves the problem of bandwidth competition when the multi-source server performs data backup to a single target server.
The following embodiments can be applied to the data synchronization processing system shown in fig. 1, where the system shown in fig. 1 includes a target server 10 and at least one source server 20 physically connected to the target server 10.
The target server 10 comprises a scheduling process and K OGG receiving processes, wherein the OGG receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with the K OGGs receiving process buses; each source end server 20 runs one OGG sending process, and there are M OGG sending processes in total. M OGG sending processes are divided into K groups, and the ith group has MiA process;
it is clear that,
Figure BDA0002572694880000041
the ith group of sending process and receiving process RPiCommunication connection for the RP to the receiving processiAnd sending the data. In particular, any of the sending processes SPijAcquiring a golden gate queue file (trail), and sending data in the queue file to a receiving process RP (remote protocol/Internet protocol) through a TCP/IP (transmission control protocol/Internet protocol)iSending process SPijIs the jth sending process of the ith group. Any receiving process RPiAll pass through MiA communication link for respectively transmitting M of the processes with the ith groupiThe sending process communicates.
The invention provides a data synchronous processing system, which is applied to a target server, wherein the target server comprises a scheduling process and K receiving processes, and the receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with K receiving process buses, and the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process; receiving a process RPiCalculating MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value; if real-time residual bandwidth BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process, wherein the scheduling information is used for requesting the scheduling process to send a receiving process RPiScheduling bandwidth resources; if real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiBandwidth resources within. I.e. the invention is based on the real-time residual bandwidth BiTo determine the corresponding scheduling scheme, in particular, the real-time remaining bandwidth BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process; real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiThe invention solves the problem of bandwidth competition when the multi-source server performs data backup to a single target server.
Referring to fig. 2, a data synchronization processing method according to a first embodiment of the present invention is shown, which can be applied to a target server in the data synchronization processing system. The data synchronization processing system can implement the method through hardware and/or software, and the method specifically comprises steps S01-S04.
S01, receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijThe data to be transmitted.
In the embodiment of the invention, the target server comprises a scheduling process and K receiving processes, wherein the receiving processes are respectively RP1，RP2，...，RPKAnd the scheduling process is connected with the K receiving process buses. Wherein, each source server correspondingly runs a sending process, M sending processes are divided into K groups, and the ith group has MiA process, the ith group having MiThe process comprises the following steps:
it is clear that,
Figure BDA0002572694880000051
the ith group of sending process and receiving process RPiCommunication connection for the RP to the receiving processiAnd sending the data. In particular, any of the sending processes SPijAcquiring a golden gate queue file (trail), and sending data in the queue file to a receiving process RP (remote protocol/Internet protocol) through a TCP/IP (transmission control protocol/Internet protocol)iSending process SPijIs the jth sending process of the ith group. Any receiving process RPiAll pass through MiA communication link for respectively transmitting M of the processes with the ith groupiThe sending process communicates.
S02, receiving process RPiCalculating MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether it is less than a preset threshold.
In one embodiment of the present invention, the real-time residual bandwidth
Figure BDA0002572694880000061
B isijFor the receiving process RPiAnd said sending process SPijReal-time residual bandwidth of the communication link therebetween.
Specifically, Bij＝RBij-Dij/Tij
Or, as is preferred, the first and second,
Figure BDA0002572694880000062
DeltaB is a minimum bandwidth scheduling unit. Preferably, RBijIs an integer multiple of DeltaB.
Wherein RBijIs RPiAnd SPijOf the communication link, DijIs SPijSize of compressed data volume, T, in the acquired golden Gate queue fileijIs SPijIs determined to be the preset maximum backup time. It should be noted that, in the ogg (golden gate) system, the real-time backup time can be theoretically compressed to the sub-second level, but in practical application, the real-time backup time T of the data is not necessarily requiredijIt is also tolerable to complete the data backup in sub-second, e.g., 20 seconds. Because DijIs dynamically changing, so SPijObtaining D at each atomic time tijAnd can be according to DijCalculation at atomic time to obtain BijThereby to convert BijTo the receiving process RPiThus receiving a process RPiIt is also possible to compute the acquisition B at atomic timei. Atomic time is the agreed minimum processing time unit, any one TijAre all integer multiples of the atomic time t. For example, t is 1 second or 0.1 second, and the embodiment of the present invention is not particularly limited.
S03, if the real-time residual bandwidth is BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process, wherein the scheduling information is used for requesting the scheduling process to send a receiving process RPiAnd scheduling bandwidth resources.
The preset threshold may be set according to actual requirements, and specifically may be 0, 1, 2, and the like, and the embodiment of the present invention is not particularly limited.
S04, if the real-time residual bandwidth is BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiBandwidth resources within.
In the embodiment of the present invention, it is,the preset threshold is 0, and if the real-time residual bandwidth B isiGreater than or equal to the preset threshold, and when receiving the process RPiEach B receivedijWhen the bandwidth is greater than 0, the link has the residual bandwidth, so that special processing is not needed.
If the real-time residual bandwidth Bi is greater than or equal to the preset threshold value and in the receiving process RPiReceiving the jth BijWhen the value is less than 0, the following treatment is carried out:
s100, to Bi1，Bi2，...Bij-1，Bij+1，...BiMiSorting in descending order, and sorting to be A1>＝A2>＝...>＝AMi
S200, finding the minimum z so that
Figure BDA0002572694880000071
Receiving a process RPiA is to be1To AzCorresponding communication link bandwidth scheduling to sending process SPijThe modulation amount is Al-Az+1And l has a value of 1 to z.
S300, receiving a process RPiUpdate AlAnd sending the updated rated bandwidth to alAnd (4) corresponding sending process.
For example, if the minimum z is found to be 1, such that A1-A2>＝BijThen process RP is receivediA is to be1Corresponding communication link bandwidth scheduling to sending process SPijThe modulation amount is (preferably) A1-A2Or the modulation amount is Bij。
As another example, if the minimum z is found to be 2, such that A1-A3+A2-A3>＝BijThen A will be1And A2Bandwidth scheduling SP of communication linkij，A1Has a modulation amount of1-A3，A2Has a modulation amount of2-A3。
The invention provides a data synchronous processing method, a receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process; receiving a process RPiCalculating MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value; if real-time residual bandwidth BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process, wherein the scheduling information is used for requesting the scheduling process to send a receiving process RPiScheduling bandwidth resources; if real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiBandwidth resources within. I.e. the invention is based on the real-time residual bandwidth BiTo determine the corresponding scheduling scheme, in particular, the real-time remaining bandwidth BiIf the value is less than the preset threshold value, receiving the process RPiSending request scheduling information to a scheduling process; real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the process RP is receivediSelf-scheduling real-time residual bandwidth BiThe invention solves the problem of bandwidth competition when the multi-source server performs data backup to a single target server.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
In an embodiment, a data synchronization processing apparatus is provided, and the data synchronization processing apparatus corresponds to the data synchronization processing method in the above embodiment one to one. As shown in fig. 3, the apparatus is applied to a target server, and the target server includes a scheduling process and K receiving processes, where the receiving processes are respectively RPs1，RP2，...，RPKThe scheduling process is connected with the K receiving process buses, and the device comprises: the system comprises a receiving module 10, a calculating module 20, a first scheduling module 30 and a second scheduling module 40. Each functionThe module is described in detail as follows:
a receiving module 10 for the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
a calculation module 20 for the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
a first scheduling module 30, configured to determine if the real-time remaining bandwidth is BiIf the value is smaller than the preset threshold value, the receiving process RP isiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
a second scheduling module 40, configured to determine if the real-time remaining bandwidth is BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
Further, the calculating module is specifically configured to calculate the real-time remaining bandwidth
Figure BDA0002572694880000081
B isijFor the receiving process RPiAnd said sending process SPijReal-time residual bandwidth of the communication link therebetween.
In particular, Bij＝RBij-Dij/TijOr is or
The above-mentioned
Figure BDA0002572694880000091
DeltaB is a minimum bandwidth scheduling unit;
wherein RBijIs RPiAnd SPijOf the communication link, DijIs SPijObtaining in-queue file warp pressesReduced data size, TijIs SPijIs determined to be the preset maximum backup time.
Further, the preset threshold is 0, and the second scheduling module 40 is configured to, when the receiving process RP is a request for receiving the request, the request is sent to the second scheduling moduleiEach B receivedijWhen the values are all larger than 0, the treatment is not needed;
when the receiving process RPiReceiving the jth BijWhen less than 0, according to each BijB of a size less than 0ijAnd scheduling bandwidth resources.
Specifically, said each of said BijB of a size less than 0ijScheduling bandwidth resources, comprising:
to Bi1，Bi2，...Bij-1，Bij+1，...BiMiSorting in descending order, and sorting to obtain A1>＝A2>＝...>＝AMi；
Find the minimum z so that
Figure BDA0002572694880000092
The receiving process RPiA is to be1To AzCorresponding communication link bandwidth scheduling to sending process SPijThe modulation amount is Al-Az+1The value of i is 1 to z;
the receiving process RPiUpdate AlAnd sending the updated rated bandwidth to alAnd (4) corresponding sending process.
For the specific limitations of the device, reference may be made to the above limitations of the data synchronization processing method, which is not described herein again. The various modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 4. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data synchronization processing method.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
if the real-time residual bandwidth BiIf the value is smaller than the preset threshold value, the receiving process RP isiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
if the real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
if the real-time residual bandwidth BiIf the value is smaller than the preset threshold value, the receiving process RP isiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
if the real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.

Claims (10)

1. A data synchronization processing method is applied to a target server and is characterized in that the target server comprises a scheduling process and K receiving processes, and the receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with the K receiving process buses, and the method comprises the following steps:
the receiving process RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
if the real-time residual bandwidth BiIf the value is smaller than the preset threshold value, the receiving process RP isiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
if the real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
2. The data synchronization processing method according to claim 1, wherein the receiving process RP is configured to perform synchronization processing on the received dataiCalculating the MiReal-time residual bandwidth B of a communication linkiThe method comprises the following steps:
the real-time residual bandwidth
Figure FDA0002572694870000011
B isijFor the receiving process RPiAnd said sending process SPijReal-time residual bandwidth of the communication link therebetween.
3. The data synchronization processing method according to claim 2, wherein B isij＝RBij-Dij/TijOr is or
The above-mentioned
Figure FDA0002572694870000012
DeltaB is a minimum bandwidth scheduling unit;
wherein RBijIs RPiAnd SPijOf the communication link, DijIs SPijSize, T, of compressed data in the acquired queue fileijIs SPijIs determined to be the preset maximum backup time.
4. The data synchronization processing method according to claim 2, wherein the predetermined threshold is 0, if the real-time remaining bandwidth B is greater than the predetermined threshold, the real-time remaining bandwidth B is greater than the predetermined thresholdiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within, including:
when the receiving process RPiEach received BijWhen the values are all larger than 0, the treatment is not needed;
when the receiving process RPiReceiving the jth BijWhen less than 0, according to each BijB of a size less than 0ijAnd scheduling bandwidth resources.
5. The data synchronization processing method according to claim 4, wherein said B is based on each of said BijB of a size less than 0ijScheduling bandwidth resources, comprising:
to Bi1，Bi2，...Bij-1，Bij+1，...BiMiSorting in descending order, and sorting to obtain A1>＝A2>＝...>＝AMi；
Find the minimum z so that
Figure FDA0002572694870000021
The receiving process RPiA is to be1To AzCorresponding communication link bandwidth scheduling to sending process SPijThe modulation amount is Al-Az+1The value of i is 1 to z;
the receiving process RPiUpdate AlAnd sending the updated rated bandwidth to alAnd (4) corresponding sending process.
6. A data synchronization processing device is applied to a target server and is characterized in that the target server comprises a scheduling process and K receiving processes, and the receiving processes are respectively RP1，RP2，...，RPKThe scheduling process is connected with the K receiving process buses, and the device comprises:
a receiving module for the receiving intoProgram RPiBy MiReceiving sending process SP in M source servers by one communication linkijFor the transmitted data, each source server correspondingly runs a transmitting process, M transmitting processes are divided into K groups, and the ith group has MiA process;
a calculation module for the receiving process RPiCalculating the MiReal-time residual bandwidth B of a communication linki(ii) a And determining the real-time residual bandwidth BiWhether the value is smaller than a preset threshold value;
a first scheduling module for determining if the real-time residual bandwidth BiIf the value is smaller than the preset threshold value, the receiving process RP isiSending request scheduling information to the scheduling process, wherein the scheduling information is used for requesting the scheduling process to send the request scheduling information to the receiving process RPiScheduling bandwidth resources;
a second scheduling module for determining if the real-time residual bandwidth BiIf the value is greater than or equal to the preset threshold value, the receiving process RP isiSelf-scheduling the real-time residual bandwidth BiBandwidth resources within.
7. The data synchronization processing apparatus according to claim 6, wherein the calculating module is specifically configured to calculate the real-time residual bandwidth
Figure FDA0002572694870000031
B isijFor the receiving process RPiAnd said sending process SPijReal-time residual bandwidth of the communication link therebetween.
8. The data synchronization processing apparatus according to claim 7, wherein B isij＝RBij-Dij/TijOr is or
The above-mentioned
Figure FDA0002572694870000032
DeltaB is a minimum bandwidth scheduling unit;
wherein RBijIs RPiAnd SPijOf the communication link, DijIs SPijSize, T, of compressed data in the acquired queue fileijIs SPijIs determined to be the preset maximum backup time.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the data synchronization processing method according to any one of claims 1 to 5 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, implements a data synchronization processing method according to any one of claims 1 to 5.
CN202010644993.1A 2020-07-07 2020-07-07 Data synchronization processing method and device, computer equipment and storage medium Active CN111817890B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010644993.1A CN111817890B (en) 2020-07-07 2020-07-07 Data synchronization processing method and device, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010644993.1A CN111817890B (en) 2020-07-07 2020-07-07 Data synchronization processing method and device, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN111817890A true CN111817890A (en) 2020-10-23
CN111817890B CN111817890B (en) 2023-04-18

Family

ID=72843503

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010644993.1A Active CN111817890B (en) 2020-07-07 2020-07-07 Data synchronization processing method and device, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN111817890B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113014435A (en) * 2021-03-15 2021-06-22 杭州米络星科技(集团)有限公司 Link detection method and device, electronic equipment and storage medium

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7324554B1 (en) * 2003-11-05 2008-01-29 Cisco Technology, Inc. Communication bandwidth distribution system and method
CN101911596A (en) * 2008-01-02 2010-12-08 汤姆森许可贸易公司 System and method for sharing an access line bandwidth
CN103731886A (en) * 2012-10-11 2014-04-16 中国移动通信集团公司 Bandwidth allocation system, method and device
US20140226571A1 (en) * 2013-02-13 2014-08-14 Qualcomm Incorporated Apparatus and method for enhanced application coexistence on an access terminal in a wireless communication system
CN106464608A (en) * 2014-04-03 2017-02-22 瑞典爱立信有限公司 Method and arrangement for scheduling in shared communication networks
CN108337189A (en) * 2018-01-29 2018-07-27 新华三技术有限公司 Bandwidth scheduling method and device
CN110300431A (en) * 2018-03-22 2019-10-01 华为技术有限公司 A kind of data traffic processing method and related network device

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7324554B1 (en) * 2003-11-05 2008-01-29 Cisco Technology, Inc. Communication bandwidth distribution system and method
CN101911596A (en) * 2008-01-02 2010-12-08 汤姆森许可贸易公司 System and method for sharing an access line bandwidth
CN103731886A (en) * 2012-10-11 2014-04-16 中国移动通信集团公司 Bandwidth allocation system, method and device
US20140226571A1 (en) * 2013-02-13 2014-08-14 Qualcomm Incorporated Apparatus and method for enhanced application coexistence on an access terminal in a wireless communication system
CN106464608A (en) * 2014-04-03 2017-02-22 瑞典爱立信有限公司 Method and arrangement for scheduling in shared communication networks
CN108337189A (en) * 2018-01-29 2018-07-27 新华三技术有限公司 Bandwidth scheduling method and device
CN110300431A (en) * 2018-03-22 2019-10-01 华为技术有限公司 A kind of data traffic processing method and related network device

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
崔勇等: "基于合作博弈的多虚拟机实时迁移带宽分配机制", 《通信学报》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113014435A (en) * 2021-03-15 2021-06-22 杭州米络星科技(集团)有限公司 Link detection method and device, electronic equipment and storage medium

Also Published As

Publication number Publication date
CN111817890B (en) 2023-04-18

Similar Documents

Publication Publication Date Title
CN112099958B (en) Distributed multi-task management method and device, computer equipment and storage medium
US8713163B2 (en) Monitoring cloud-runtime operations
CN111080449B (en) Cross-chain transaction method of blockchain, management node and blockchain network
CN102724103B (en) Proxy server, hierarchical network system and distributed workload management method
US20180157688A1 (en) Ledger-chained distributed information handling systems and methods
Yang et al. Resource analysis of blockchain consensus algorithms in hyperledger fabric
EP3306866B1 (en) Message processing method, device and system
US11297161B1 (en) Systems and methods for managing an automotive edge computing environment
CN111817890B (en) Data synchronization processing method and device, computer equipment and storage medium
CN111490963A (en) Data processing method, system, equipment and storage medium based on QUIC protocol stack
CN113703997A (en) Bidirectional asynchronous communication middleware system integrating multiple message agents and implementation method
CN111401991A (en) Data information distribution method and device, storage medium and computer equipment
CN111090532A (en) Application service calling method and device, electronic equipment and computer storage medium
CN111163140A (en) Method, apparatus and computer readable storage medium for resource acquisition and allocation
CN108334410A (en) A kind of distributed application program client light weight method and computer equipment
CN114610765B (en) Stream calculation method, device, equipment and storage medium
DE112016004290T5 (en) TECHNOLOGIES FOR AGGREGATION-BASED MESSAGE SYNCHRONIZATION
CN115643271A (en) Method, device, server and medium for synchronizing multi-application data on cloud
CN104699799A (en) Data transmission method based on cross system
CN111309467B (en) Task distribution method and device, electronic equipment and storage medium
CN111082964B (en) Distribution method and device of configuration information
Ward et al. EDS hardware architecture
CN116261839A (en) Operation of a distributed deterministic network
CN113641498A (en) Data checking method, system, equipment and storage medium
CN106487694A (en) A kind of data flow processing method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant