CN113055430A - Data synchronization method and related equipment - Google Patents
Data synchronization method and related equipment Download PDFInfo
- Publication number
- CN113055430A CN113055430A CN201911423080.0A CN201911423080A CN113055430A CN 113055430 A CN113055430 A CN 113055430A CN 201911423080 A CN201911423080 A CN 201911423080A CN 113055430 A CN113055430 A CN 113055430A
- Authority
- CN
- China
- Prior art keywords
- partition
- leader
- offset
- data
- follower
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000005192 partition Methods 0.000 claims abstract description 359
- 230000001360 synchronised effect Effects 0.000 claims abstract description 65
- 230000005540 biological transmission Effects 0.000 claims description 33
- 238000012545 processing Methods 0.000 claims description 23
- 230000003044 adaptive effect Effects 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 230000005012 migration Effects 0.000 description 12
- 238000013508 migration Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 238000009826 distribution Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a data synchronization method and related equipment. Wherein, the method comprises the following steps: a follower partition receives an offset selection strategy; the follower partition determines a target offset in the leader partition of the topic to be synchronized based on the offset selection strategy; the follower partition synchronizes data in the leader partition to the follower partition starting from the determined target offset of the leader partition. The method can improve the flexibility of data synchronization, improve the data synchronization efficiency and shorten the data synchronization time.
Description
Technical Field
The present invention relates to the field of information technologies, and in particular, to a data synchronization method and related devices.
Background
The distributed publish-subscribe messaging system (kafka) has good functional characteristics, provides persistence of messages through a disk structure, can maintain stable performance for a long time, has high throughput, can support millions of messages per second, and can support parallel data loading, so the kafka is generally used for processing mass stream data generated by an application service system or an Artificial Intelligence (AI) platform.
Kafka, the most commonly used distributed message queue system currently, generally caches message data on a local disk, objects published and subscribed in Kafka are categories (topics), each topic represents a category of data, one topic is divided into a plurality of partitions (partitions), and each partition is an ordered queue. Different partitionings of each topic are distributed on different disks or hosts, meanwhile, kafka has multiple copies (replica) for ensuring reliability, one master copy is selected among the multiple copies, the master copy may be called a leader partition (leader partition), and the other copies are slave copies and may be called follower partitions (follower partitions). The Follower partition will keep on synchronizing the latest message data from the leader partition, and the consumer (the client subscribing to the message from the topic) and the producer (the client publishing the message to the topic) will interact directly with the leader partition during consumption and production. When a node where a certain leader partition is located suddenly fails or loses power, a new leader partition is elected by the follower partition corresponding to the partition, so that the reliability of service and the safety of data are ensured.
When a node in the kafka cluster fails to recover or the storage capacity of the kafka cluster needs to be expanded (i.e., the number of nodes is increased), the copies existing in the kafka cluster need to be migrated to ensure the integrity of the copies, and currently, the migration of the copies in the kafka cluster is completed by a migration tool provided by the kafka itself, but this method is not flexible enough, the migration time is long, the efficiency is low, and Central Processing Unit (CPU) resources and network bandwidth resources are continuously occupied, which affects the normal operation of a service.
Disclosure of Invention
The embodiment of the invention discloses a data synchronization method and related equipment, wherein a follower partition can flexibly select data to be synchronized from a leader partition, so that the data synchronization efficiency is improved, the data synchronization time is shortened, and the influence on services is reduced.
In a first aspect, the present application provides a method for data synchronization, including: a follower partition receives an offset selection strategy; the follower partition determines a target offset in a leader partition of the topic to be synchronized based on the offset selection strategy; the follower partition synchronizes data in the leader partition to the follower partition starting from the determined target offset of the leader partition.
In this embodiment of the application, the follower partition can flexibly select the target offset that needs to be synchronized as needed, and after the target offset is determined, the target offset determined in the leader partition starts to synchronize the data in the leader partition to the follower partition, so that the data synchronization efficiency can be improved, the data synchronization time can be shortened, and the applicable scene can be expanded.
In a possible implementation manner, the follower partition sends a query request to the leader partition, where the query request is used to query an offset in the leader partition at the current time; and the follower partition calculates a difference value between the offset in the follower partition and the offset in the follower partition at the current moment, and determines to synchronize the data in the follower partition to the follower partition when the difference value is greater than a preset threshold value.
In this embodiment of the application, before performing data synchronization, the follower partition needs to query the offset in the leader partition at the current time, then calculate a difference between the offset in the leader partition and the offset in the follower partition, and perform data synchronization when the difference is greater than a preset threshold. Therefore, frequent data synchronization can be avoided, and the data synchronization can be performed only when the conditions are met, so that the system efficiency is improved.
In a possible implementation manner, the offset selection policy is an offset adaptive determination policy, and the follower partition obtains the average data writing speed of the leader partition; and the follower partition calculates to obtain the target offset according to the average data writing speed of the leader partition.
In this embodiment of the application, when the offset selection policy is the offset adaptive determination policy, the follower partition needs to obtain the average data writing speed of the leader partition, so as to calculate the target offset.
In a possible implementation manner, the offset selection policy is an offset adaptive determination policy, and the follower partition acquires a start offset of the leader partition and a data average writing speed of the leader partition; and the follower partition calculates the target offset according to the initial offset of the leader partition and the data average writing speed of the leader partition.
In the embodiment of the application, when the offset selection strategy is an offset adaptive determination strategy, the follower partition needs to acquire the initial offset and the average data writing speed of the leader partition, so that the target offset obtained by calculation is more accurate.
In a possible implementation manner, when the average data writing speed of the leader partition is greater than the data synchronous transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time; and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed, the target offset is the initial offset of the leader partition.
In the embodiment of the application, the target offset is determined by comparing the relation between the average data writing speed of the leader partition and the synchronous data transmission speed, so that the determined target offset can be ensured to be more reasonable.
In a possible implementation manner, when the average data writing speed of the leader partition is greater than the data synchronous transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time; and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed and the synchronization time required by the current data total amount of the leader partition is less than a preset time length, the target offset is the starting offset of the leader partition.
In the embodiment of the application, the relationship between the average data writing speed of the leader partition and the synchronous data transmission speed and the relationship between the synchronization time required by the total current data of the leader partition and the preset time length are considered at the same time, so that the target offset is determined, and the reasonability and the accuracy of the determined target offset can be further improved.
In a possible implementation manner, the offset selection policy is a selection policy based on a data value, the follower partition acquires consumption offsets of a plurality of consumer groups, and the follower partition determines that the minimum consumption offset is the target offset; wherein the consumer groups are configured to consume data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is configured to indicate a number of data in the leader partition consumed by the consumer group.
In the embodiment of the application, when the offset selection policy is a selection policy based on a data value, it is necessary to obtain consumption offsets of a plurality of consumer groups consuming data in topic to be synchronized, so as to determine a target offset, that is, a minimum consumption offset is determined as the target offset.
In a second aspect, the present application provides a data synchronization apparatus, including: the acquisition unit is used for acquiring an offset selection strategy; the processing unit is used for determining a target offset in the leader partition of the topic to be synchronized based on the offset selection strategy; a synchronization unit, configured to synchronize data in the leader partition from the determined target offset of the leader partition.
In a possible implementation manner, the data synchronization apparatus further includes a sending unit, where the sending unit is configured to send a query request to the leader partition, where the query request is used to query an offset in the leader partition at a current time; the processing unit is further configured to calculate a difference between an offset in the leader partition at the current time and an offset in the data synchronization apparatus, and determine to synchronize data in the leader partition when the difference is greater than a preset threshold.
In a possible implementation manner, the offset selecting policy is an offset adaptive determination policy, and the processing unit is specifically configured to: acquiring the average data writing speed of leader partition; and calculating to obtain the target offset according to the average data writing speed of the leader partition.
In a possible implementation manner, the offset selecting policy is an offset adaptive determination policy, and the processing unit is specifically configured to: acquiring the initial offset of the leader partition and the average data writing speed of the leader partition; and calculating to obtain the target offset according to the initial offset of the leader partition and the data average writing speed of the leader partition.
In a possible implementation manner, when the average data writing speed of the leader partition is greater than the data synchronous transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time; and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed, the target offset is the initial offset of the leader partition.
In a possible implementation manner, when the average data writing speed of the leader partition is greater than the data synchronous transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time; and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed and the synchronization time required by the current data total amount of the leader partition is less than a preset time length, the target offset is the starting offset of the leader partition.
In a possible implementation manner, the offset selection policy is a selection policy based on a data value, and the processing unit is specifically configured to: acquiring consumption offset of a plurality of consumer groups, and determining the minimum consumption offset as the target offset; wherein the consumer groups are configured to consume data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is configured to indicate a number of data in the leader partition consumed by the consumer group.
In a third aspect, the present application provides a computing device comprising a processor and a memory, wherein the processor executes computer instructions stored in the memory to cause the computing device to perform the first aspect and the method in combination with any one implementation manner of the first aspect.
In a fourth aspect, the present application provides a computer storage medium storing a computer program that, when executed by a computing device, implements the first aspect described above and a method incorporating any one of the implementations of the first aspect.
In a fifth aspect, the present application provides a computer program product comprising computer instructions that, when executed by a computing device, enable the computing device to perform the first aspect and the method in combination with any one of the implementations of the first aspect.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1A is a schematic diagram of an application scenario provided in an embodiment of the present application;
fig. 1B is a schematic diagram of another application scenario provided in the embodiment of the present application;
fig. 1C is a schematic diagram of another application scenario provided in the embodiment of the present application;
FIG. 2 is a schematic diagram of data synchronization provided by an embodiment of the present application;
fig. 3 is a schematic flowchart of a method for data synchronization according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application;
fig. 5 is a schematic structural diagram of a computing device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application are described below clearly and completely with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is explicitly and implicitly understood by one skilled in the art that the embodiments described herein can be combined with other embodiments.
First, a part of words and related technologies referred to in the present application will be explained with reference to the accompanying drawings so as to be easily understood by those skilled in the art.
Kafka is a distributed, partition-supporting, multi-copy, zookeeper coordination-based distributed messaging system that can process large amounts of data in real-time to meet various demand scenarios: such as hadoop-based batch processing systems, low latency real-time systems, storm/spark streaming engines, access logs, message services, and the like. Characteristics of Kafka include: the method has the advantages of high throughput, low time delay, strong expandability, good durability, high fault tolerance and high concurrency support, and is widely applied to log collection, user activity tracking and the like.
Zookeeper is a distributed, open-source distributed application coordination service, which is an important component of Hadoop and Hbase. It is a software that provides a consistent service for distributed applications, and the functions provided include: configuration maintenance, domain name service, distributed synchronization, group service, etc. The Zookeeper aims to package complex and error-prone key services and provide a simple and easy-to-use interface and a system with high performance and stable functions for users.
Fig. 1A shows a possible application scenario of the embodiment of the present application. In this application scenario, the node 110, the node 120, and the node 130 are node devices in the kafka cluster 100, for example: a server, etc., each node stores different partitions of different topics, for example, the node 110 stores a leader partition111 of a first partition, a follower partition112 of the first partition, and a follower partition113 of a second partition; the node 120 stores a follower partition121 of the first partition, a follower partition122 of the second partition, and a leader partition123 of the third partition; the node 130 stores two follower partitions 131 and 132 of the third partition and the leader partition of the second partition. The first partition, the second partition and the third partition may be different partitions under the same topic, or different partitions under different topics, and each partition stores data by using three copies. At this time, the kafka cluster 100 needs to be expanded, that is, one node 140 is newly added, at this time, no data is stored in the newly added node 140, the data distribution of the entire kafka cluster 100 is not balanced, and a migration tool needs to be used to migrate a part of the partitions stored in other nodes to the newly added node 140, so as to ensure balanced data distribution of the entire cluster. For example, the follower partition112 of node 110 is migrated into node 140 and the follower partition131 of node 130 is migrated into node 140 using the copy migration tool kafka-reassign-partitions provided by kafka.
Fig. 1B shows yet another possible application scenario. As shown in fig. 1B, the node 110 stores a leader partition111 of the first partition, a follower partition112 of the first partition, and a follower partition113 of the second partition; the node 120 stores a follower partition121 of the first partition, a follower partition122 of the second partition, and a leader partition123 of the third partition; the node 130 stores two follower partitions 131 and 132 of the third partition and the leader partition of the second partition. At this time, when the node 120 fails during the operation, the data stored in the node 120 needs to be migrated to the node 110 and the node 130 to achieve data distribution balance of the entire cluster 100, the migration tool is used to migrate the follower partition121 and the leader partition123 in the node 120 to the node 110, and migrate the follower partition122 in the node 120 to the node 130.
Fig. 1C shows yet another possible application scenario. As shown in fig. 1C, the node 110 stores the leader partition111 of the first partition, a follower partition112 of the first partition, and a follower partition113 of the second partition; the node 120 stores a follower partition121 of the first partition and a leader partition122 of the third partition; the leader partition131 of the second partition is stored in the node 130. At this time, the data distribution of the cluster 100 is not balanced, and the migration tool needs to migrate the follower partition113 stored in the node 110 to the node 130, so as to achieve more balanced data distribution of the cluster 100.
For the application scenarios shown in fig. 1A, fig. 1B, and fig. 1C, it is necessary to migrate the folder partition on a certain node to another node by using a migration tool, where the migration does not refer to directly transporting data from one node to another node, but refers to pulling data from the leader partition corresponding to the folder partition in a synchronous manner, and finally achieves the consistency with the data in the leader partition, thereby completing the purpose of implementing the migration. However, the copy migration tool kafka-replication-partitions provided by kafka is currently used to complete the migration of the follower partition (i.e. data synchronization), and when a new follower partition is created in a node, the copy synchronization thread will pull data from the leader partition corresponding to the follower partition until the data of the follower partition is synchronized with the data of the follower partition. However, in the above solution, the newly created follower partition starts to synchronize from the start offset of the follower partition when synchronizing data, when the amount of the cache data of the follower partition is large, a large amount of time is required to complete data synchronization, and CPU resources, network bandwidth resources, etc. of nodes are continuously occupied during data synchronization, which affects normal operation of a service, and in addition, in many scenarios, the value of the data cached in kafka gradually decreases with the passage of time, and an actual service or application is only interested in the latest data, for example, in an application scenario in which the number of times of occurrence of an object within one minute is counted, only the data within the current minute is valuable, and the previous data value is low, so that it does not make sense to synchronize a large amount of history cache data, and the follower partition can only synchronize from the start offset of the follower partition, and cannot flexibly specify an offset, that is, synchronization cannot be performed from any offset according to actual needs, and synchronization of worthless history cache data cannot be avoided.
In order to solve the above problems, the present application provides a data synchronization method and related device, which can flexibly select data to be synchronized from a leader partition when performing data synchronization, improve data synchronization efficiency, shorten data synchronization time, and reduce the influence on services.
Referring to fig. 2, fig. 2 is a schematic diagram of data synchronization provided in the present application. As shown in fig. 2, a node 210 deploys a leader partition of a certain partition, data written by a producer is cached in the leader partition, and includes history write data and latest write data, a node 220 deploys a follower partition corresponding to the same partition, the follower partition needs to synchronize data from the leader partition in the node 210, and the follower partition can flexibly select data that needs to be synchronized, for example, data synchronization can be performed from a start offset of the leader partition, that is, an offset 1, and all data are synchronized; or starting synchronization from the offset 7, and synchronizing only the data written by the leader partition newly; or an arbitrary offset (e.g., offset 5, etc.) may be selected to begin synchronization.
Referring to fig. 3 in combination with the application scenarios shown in fig. 1A, fig. 1B, and fig. 1C and the data synchronization diagram shown in fig. 2, fig. 3 is a flowchart of a data synchronization method according to an embodiment of the present disclosure. As shown in fig. 3, the method includes, but is not limited to, the following steps:
s301: and the follower partition acquires an offset selection strategy.
For example, a user may access the zookeeper through a world wide Web (Web) browser or other interfaces (e.g., restful interfaces, etc.) provided by the kafka cluster, and set an offset selecting policy, where the policy is set by taking topic as a unit, that is, an offset obtaining policy is set for all partitions under the same topic. After the setting is completed, the set offset obtaining policy may be saved as a JavaScript object notation (JSON) formatted file, and the file may be recorded in a specified directory of the zookeeper.
In addition, all nodes in the kafka cluster will elect a management node that monitors zookeeper using the watch mechanism. Therefore, when the offset obtaining policy is recorded in a directory specified by the zookeeper, the management node senses that the directory of the zookeeper changes, and then sends a synchronization request to the related node, that is, the node of the folder partition of topic corresponding to the offset obtaining policy is newly established, after the related node receives the synchronization request sent by the management node, the copy synchronization thread is started, and after the copy synchronization thread is started, the related node reads the offset obtaining policy of the related topic on the zookeeper.
In a possible implementation manner, the follower partition sends a query request to the leader partition, where the query request is used to query an offset in the leader partition at the current time; the follower partition calculates a difference value between the maximum offset of the leader partition at the current time and the maximum offset of the follower partition, and determines that the data in the synchronous leader partition reaches the follower partition when the difference value is greater than a preset threshold value.
Specifically, each log has an offset to uniquely mark a message, and the offset has a value of 8 bytes to indicate its position in the partition, so that each message (i.e., data) stored in the leader partition and the follower partition corresponds to a unique offset. After the copy synchronization thread is started, the Follower partition sends a query request to the leader partition to obtain the maximum offset of the leader partition at the current time, then a difference value between the offset and the Follower partition is calculated, if the difference value exceeds a preset threshold value, it is indicated that a large amount of data in the leader partition is not synchronized into the leader partition, and at the moment, the data in the leader partition needs to be synchronized into the local. It should be understood that the preset threshold may be set as required, and the present application is not limited thereto.
S302: the follower partition determines a target offset in the leader partition of the topic to be synchronized based on an offset selection policy.
Specifically, the offset selection policy needs to be determined before data synchronization is performed, and after the offset selection policy is determined, a target offset in the leader partition of topic to be synchronized needs to be further determined, that is, from which offset data synchronization is performed.
In a possible implementation manner, the offset selection strategy is an offset adaptive determination strategy, and the follower partition acquires the average data writing speed of the leader partition; the follower partition calculates the target offset according to the average data writing speed of the leader partition.
Specifically, the follower partition obtains the average data writing speed of the follower partition through an interface provided by Java management extensions (JMX) in the node where the follower partition is located, for example, the average data writing speed of the follower partition in the last hour is obtained through the interface provided by JMX. The target offset can be calculated after the average writing speed of the data of the leader partition is obtained.
Optionally, after the follower partition acquires the average data writing speed of the leader partition, a query request may be further sent to the leader partition to query the start offset of the leader partition, and then the target offset may be calculated according to the queried start offset of the leader partition and the average data writing speed of the leader partition.
Illustratively, after obtaining the average data writing speed of the leader partition, the follower partition first determines whether a speed threshold is set, where the speed threshold is used to limit the data synchronization transmission speed of the follower partition, so as to avoid occupying excessive resources such as CPU resources and network bandwidth during data synchronization, and avoid affecting a service, which may result in that the service cannot operate normally. If the speed threshold is set, taking the speed threshold as the data synchronous transmission speed; if the threshold is not set, the consumption speed of the follower partition is used as the data synchronous transmission speed.
For example, when performing data synchronization, the essence of the folder partition is to start another consuming thread in the folder partition, and the folder partition is used as a consumer to consume the data in the leader partition to achieve data synchronization. Therefore, in the case where the speed threshold is not set, the consumption speed of the follower partition is the data synchronous transmission speed.
Then, the follower partition judges the magnitude relation between the data synchronous transmission speed and the data average writing speed of the follower partition, and when the data average writing speed of the follower partition is greater than the data synchronous transmission speed, it indicates that even though the follower partition continuously performs data synchronization all the time, the follower partition cannot completely catch up with the follower partition, that is, the follower partition always synchronizes the historical data of the follower partition, so that in this case, only the latest data in the follower partition needs to be synchronized, the maximum offset of the follower partition at the current time is used as the target offset, and the follower partition performs data synchronization from the maximum offset of the follower partition at the current time. When the average writing speed of the data of the leader partition is less than or equal to the synchronous data transmission speed, it indicates that the follower partition can catch up with the leader partition after enough time, that is, the follower partition can synchronize all the data in the leader partition to the local, and in this scenario, further judgment processing is required to determine the final target offset.
In one possible implementation, the follower partition calculates the time required for the current data total of the synchronization leader partition. Optionally, the follower partition may obtain the current data total while obtaining the average writing speed of the leader partition through an interface provided by JMX, or may obtain the current data total of the leader partition through other manners, which is not limited in this application. After the current total data amount is acquired, the Follower partition divides the total data amount by the data synchronous transmission speed to obtain the synchronous time. When the synchronization time is less than or equal to the preset time, it is indicated that the total amount of the current data of the leader partition is not much, the synchronization process cannot cause excessive resource consumption, the follower partition takes the initial offset of the leader partition as the target offset, and the follower partition synchronizes all the data in the leader partition to the local. The preset time can be set according to needs, for example, one hour, which is not limited in this application.
When the synchronization time is greater than the preset time, the target offset amount may be calculated according to the following equation 1:
((y-y0)*n)/v1 ═ ((y1-y) n)/(v2-v1) formula 1
Wherein y0 represents the start offset of the leader partition, y represents the target offset, y1 represents the maximum offset of the leader partition, n represents the average data amount of one message, v1 represents the average data writing speed of the leader partition, and v2 represents the data synchronous transmission speed.
Further simplifying the above equation 1, the following equation 2 can be obtained:
y＝(y1*v1+(v2-v1) y0)/v2 formula 2
It can be seen that after the initial offset and the maximum offset of the leader partition, the average data writing speed of the leader partition and the synchronous data transmission speed are obtained, the target offset can be calculated, and the data in the leader partition is synchronized from the target offset, so that the influence on the service due to the long-term occupation of resources such as a CPU (central processing unit), a network bandwidth and the like in the data synchronization process can be avoided, and the synchronization of some outdated and meaningless historical data can be avoided, thereby improving the synchronization efficiency and shortening the synchronization time.
In yet another possible implementation manner, the offset selection policy is a selection policy based on a data value, the follower partition obtains consumption offsets of multiple consumer groups, and the follower partition determines that a minimum consumption offset is a target offset, where the consumer groups are used to consume data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is used to indicate a number of data in the leader partition consumed by the consumer group.
Specifically, each consumer belongs to a particular group, and a group name may be assigned to each consumer, or a default group may be assigned if not. When consuming data, the consumer groups are carried out by taking topic as a unit, and different consumer groups can consume the same topic. For consumers in a consumer group, consumers in the same group cannot consume the same partition of the same topic, i.e. different consumers cannot consume the same partition, but the same consumer can consume multiple partitions.
Therefore, for the topic to be synchronized, when a plurality of consumer groups consume data in the topic at the same time, a consumer consuming the leader partition to be synchronized is found from each consumer group, the consumption offset of the consumer, namely the number of data in the leader partition currently consumed by the consumer, is obtained, then the obtained consumption offset is compared, and the smallest consumption offset is selected as the target offset.
It is easy to understand that, the minimum consumption offset is used as the target offset, so that when a node where the leader partition is located suddenly fails and goes down, all consumer groups can still completely consume the data in the topic, and the influence on the service is avoided.
In yet another possible implementation manner, the offset selection policy is an offset specification policy, and the follower partition uses a specified offset as a target offset.
Specifically, after determining that the data in the leader partition needs to be synchronized to the leader partition, the follower partition directly uses the specified offset as the target offset, and performs data synchronization from the target offset. The specific value of the specified offset can be set according to actual needs, and the application is not limited to this.
S303: the follower partition synchronizes data in the leader partition to the follower partition starting from the determined target offset of the leader partition.
Specifically, after determining the target offset, the following thread starts a consuming thread to consume the data in the leader partition from the target offset, that is, to synchronize the data in the leader partition to the local.
It should be understood that steps S301 to S303 involved in the above method embodiments are merely schematic descriptions and should not constitute a specific limitation, and the steps involved may be added, reduced or combined as needed.
The method of the embodiments of the present application is described in detail above, and in order to better implement the above-mentioned aspects of the embodiments of the present application, correspondingly, the following also provides related equipment for implementing the above-mentioned aspects in a matching manner.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a data synchronization apparatus according to an embodiment of the present application. As shown in fig. 4, the data synchronization apparatus 400 includes an acquisition unit 410, a processing unit 420, and a synchronization unit 430. Wherein the content of the first and second substances,
the obtaining unit 410 is configured to obtain an offset selecting policy.
Specifically, the obtaining unit 410 is configured to execute the foregoing step S301, and optionally execute an optional method in the foregoing step.
A processing unit 420, configured to determine a target offset in the leader partition of the topic to be synchronized based on the offset selection policy.
Specifically, the processing unit 420 is configured to execute the foregoing step S302, and optionally execute an optional method in the foregoing step.
A synchronizing unit 430, configured to synchronize the data in the leader partition from the determined target offset of the leader partition.
Specifically, the synchronization unit 430 is configured to perform the foregoing step S303, and optionally perform an optional method in the foregoing step.
In a possible implementation manner, the data synchronization apparatus 400 further includes a sending unit 440, where the sending unit 440 is configured to send a query request to the leader partition, where the query request is used to query an offset in the leader partition at the current time; the processing unit 420 is further configured to calculate a difference between an offset in the leader partition at the current time and an offset in the data synchronization apparatus, and determine to synchronize data in the leader partition when the difference is greater than a preset threshold.
In a possible implementation manner, the offset selecting policy is an offset adaptive determination policy, and the processing unit 420 is specifically configured to: acquiring the average data writing speed of leader partition; and calculating to obtain the target offset according to the average data writing speed of the leader partition.
In a possible implementation manner, the offset selecting policy is an offset adaptive determination policy, and the processing unit 420 is specifically configured to: acquiring the initial offset of the leader partition and the average data writing speed of the leader partition; and calculating to obtain the target offset according to the initial offset of the leader partition and the data average writing speed of the leader partition.
In a possible implementation manner, when the average data writing speed of the leader partition is greater than the data synchronous transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time; and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed, the target offset is the initial offset of the leader partition.
In a possible implementation manner, when the average data writing speed of the leader partition is greater than the data synchronous transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time; and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed and the synchronization time required by the current data total amount of the leader partition is less than a preset time length, the target offset is the starting offset of the leader partition.
In a possible implementation manner, the offset selection policy is a selection policy based on a data value, and the processing unit 420 is specifically configured to: acquiring consumption offset of a plurality of consumer groups, and determining the minimum consumption offset as the target offset; wherein the consumer groups are configured to consume data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is configured to indicate a number of data in the leader partition consumed by the consumer group.
It should be noted that the structure of the data synchronization apparatus and the process of implementing data synchronization by using the data synchronization apparatus are only examples, and should not be construed as being particularly limited, and the units in the data synchronization apparatus may be added, reduced or combined as needed. In addition, for the sake of brevity, the operations and/or functions of the modules in the data synchronization apparatus are not described again in order to implement the corresponding process of the method described in fig. 3.
Referring to fig. 5, fig. 5 is a schematic structural diagram of a computing device according to an embodiment of the present application. As shown in fig. 5, the computing device 500 includes: a processor 510, a communication interface 520 and a memory 530, said processor 510, communication interface 520 and memory 530 being interconnected by an internal bus 540. It should be understood that the computing device may be a library server.
The computing device 500 may be the server in fig. 1A-1C, 2, and 3 that has the follower partition deployed. The functions performed by the server in fig. 1A-1C, 2 and 3 are actually performed by the processor 510 of the server.
The processor 510 may be formed by one or more general-purpose processors, such as a Central Processing Unit (CPU), or a combination of a CPU and a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
The bus 540 may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus 540 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 5, but not only one bus or type of bus.
Embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, may implement part or all of the steps of any one of the method embodiments described above, and implement the functions of any one of the functional units described in fig. 4 above.
Embodiments of the present application also provide a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, may implement part or all of the steps of any one of the method embodiments described above, and implement the functions of any one of the functional units described in fig. 4 above.
Embodiments of the present application also provide a computer program product, which when run on a computer or a processor, causes the computer or the processor to perform one or more steps of any of the methods described above. The respective constituent elements of the above-mentioned apparatus may be stored in the computer-readable storage medium if they are implemented in the form of software functional units and sold or used as independent products.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
It should also be understood that, in the various embodiments of the present application, the sequence numbers of the above-mentioned processes do not mean the execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application or portions thereof that substantially contribute to the prior art may be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (17)
1. A method of data synchronization, the method comprising:
a follower partition receives an offset selection strategy;
the follower partition determines a target offset in a leader partition of the topic to be synchronized based on the offset selection strategy;
the follower partition synchronizes data in the leader partition to the follower partition starting from the determined target offset of the leader partition.
2. The method of claim 1, wherein the method further comprises:
the follower partition sends a query request to the leader partition, wherein the query request is used for querying the offset in the leader partition at the current time;
and the follower partition calculates a difference value between the offset in the follower partition and the offset in the follower partition at the current moment, and determines to synchronize the data in the follower partition to the follower partition when the difference value is greater than a preset threshold value.
3. The method of claim 1 or 2, wherein the offset selection policy is an offset adaptive determination policy, and wherein the determining of the offset in the leader partition of the topic to be synchronized based on the offset selection policy comprises:
the follower partition acquires the average data writing speed of the leader partition;
and the follower partition calculates to obtain the target offset according to the average data writing speed of the leader partition.
4. The method of claim 1 or 2, wherein the offset selection policy is an offset adaptive determination policy, and wherein the determining of the offset in the leader partition of the topic to be synchronized based on the offset selection policy comprises:
the follower partition acquires the initial offset of the leader partition and the data average writing speed of the leader partition;
and the follower partition calculates the target offset according to the initial offset of the leader partition and the data average writing speed of the leader partition.
5. The method of claim 3 or 4,
when the average data writing speed of the leader partition is greater than the synchronous data transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time;
and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed, the target offset is the initial offset of the leader partition.
6. The method of claim 3 or 4,
when the average data writing speed of the leader partition is greater than the synchronous data transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time;
and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed and the synchronization time required by the current data total amount of the leader partition is less than a preset time length, the target offset is the starting offset of the leader partition.
7. The method of claim 1 or 2, wherein the offset selection policy is a data value-based selection policy, and the determining of the offset in the leader partition of the topic to be synchronized based on the offset selection policy comprises:
the follower partition obtains consumption offsets for a plurality of consumer groups, the follower partition determines the smallest of the consumption offsets as the target offset;
wherein the consumer groups are configured to consume data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is configured to indicate a number of data in the leader partition consumed by the consumer group.
8. A data synchronization apparatus, comprising:
the acquisition unit is used for acquiring an offset selection strategy;
the processing unit is used for determining a target offset in the leader partition of the topic to be synchronized based on the offset selection strategy;
a synchronization unit, configured to synchronize data in the leader partition from the determined target offset of the leader partition.
9. The data synchronization apparatus of claim 8, wherein the data synchronization apparatus further comprises a transmission unit,
the sending unit is configured to send a query request to the leader partition, where the query request is used to query an offset in the leader partition at the current time;
the processing unit is further configured to calculate a difference between an offset in the leader partition at the current time and an offset in the data synchronization apparatus, and determine to synchronize data in the leader partition when the difference is greater than a preset threshold.
10. The data synchronization apparatus according to claim 8 or 9, wherein the offset selection policy is an offset adaptive determination policy, and the processing unit is specifically configured to:
acquiring the average data writing speed of leader partition;
and calculating to obtain the target offset according to the average data writing speed of the leader partition.
11. The data synchronization apparatus according to claim 8 or 9, wherein the offset selection policy is an offset adaptive determination policy, and the processing unit is specifically configured to:
acquiring the initial offset of the leader partition and the average data writing speed of the leader partition;
and calculating to obtain the target offset according to the initial offset of the leader partition and the data average writing speed of the leader partition.
12. The data synchronization apparatus according to claim 10 or 11,
when the average data writing speed of the leader partition is greater than the synchronous data transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time;
and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed, the target offset is the initial offset of the leader partition.
13. The data synchronization apparatus according to claim 10 or 11,
when the average data writing speed of the leader partition is greater than the synchronous data transmission speed of the leader partition, the target offset is the maximum offset of the leader partition at the current time;
and when the average data writing speed of the leader partition is less than or equal to the synchronous data transmission speed and the synchronization time required by the current data total amount of the leader partition is less than a preset time length, the target offset is the starting offset of the leader partition.
14. The data synchronization apparatus according to claim 8 or 9, wherein the offset selection policy is a selection policy based on a data value, and the processing unit is specifically configured to:
acquiring consumption offset of a plurality of consumer groups, and determining the minimum consumption offset as the target offset;
wherein the consumer groups are configured to consume data in the topic to be synchronized, each consumer group includes at least one consumer, and the consumption offset is configured to indicate a number of data in the leader partition consumed by the consumer group.
15. A computing device, comprising a processor and a memory, the processor executing computer instructions stored by the memory to cause the computing device to perform the method of any of claims 1-7.
16. A computer storage medium, characterized in that the computer storage medium stores a computer program which, when executed by a computing device, implements the method of any one of claims 1-7.
17. A computer program product comprising computer instructions which, when executed by a computing device, may carry out the method of any of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911423080.0A CN113055430A (en) | 2019-12-27 | 2019-12-27 | Data synchronization method and related equipment |
| PCT/CN2020/136716 WO2021129477A1 (en) | 2019-12-27 | 2020-12-16 | Data synchronization method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911423080.0A CN113055430A (en) | 2019-12-27 | 2019-12-27 | Data synchronization method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113055430A true CN113055430A (en) | 2021-06-29 |
Family
ID=76507531
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911423080.0A Pending CN113055430A (en) | 2019-12-27 | 2019-12-27 | Data synchronization method and related equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113055430A (en) |
| WO (1) | WO2021129477A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114385081A (en) * | 2021-12-27 | 2022-04-22 | 联通智网科技股份有限公司 | Disk protection method for kafka cluster and related equipment |
| CN115037745A (en) * | 2022-05-18 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Method and device for election in distributed system |
| CN115052006A (en) * | 2022-08-12 | 2022-09-13 | 佛山赛思禅科技有限公司 | Data synchronization method and system based on leader node |
| CN116955427A (en) * | 2023-09-18 | 2023-10-27 | 北京长亭科技有限公司 | Method and device for processing real-time multi-rule dynamic expression data based on Flink frame |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113872888B (en) * | 2021-09-27 | 2023-05-19 | 南威软件股份有限公司 | Method, system and medium for catch-up of Kafka backlog queue |
| CN114827049B (en) * | 2022-03-02 | 2023-05-09 | 厦门服云信息科技有限公司 | Pile-up data consumption method based on kafka, terminal equipment and storage medium |
| CN115150471B (en) * | 2022-06-27 | 2024-03-29 | 北京百度网讯科技有限公司 | Data processing method, apparatus, device, storage medium, and program product |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10262032B2 (en) * | 2016-02-24 | 2019-04-16 | Salesforce.Com, Inc. | Cache based efficient access scheduling for super scaled stream processing systems |
| CN105912628B (en) * | 2016-04-07 | 2019-05-28 | 北京奇虎科技有限公司 | The synchronous method and device of master-slave database |
| CN106790378A (en) * | 2016-11-11 | 2017-05-31 | 北京奇虎科技有限公司 | The full synchronous method of data of equipment room, apparatus and system |
| CN109388677B (en) * | 2018-08-23 | 2022-10-11 | 顺丰科技有限公司 | Method, device and equipment for synchronizing data among clusters and storage medium thereof |
-
2019
- 2019-12-27 CN CN201911423080.0A patent/CN113055430A/en active Pending
-
2020
- 2020-12-16 WO PCT/CN2020/136716 patent/WO2021129477A1/en active Application Filing
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114385081A (en) * | 2021-12-27 | 2022-04-22 | 联通智网科技股份有限公司 | Disk protection method for kafka cluster and related equipment |
| CN115037745A (en) * | 2022-05-18 | 2022-09-09 | 阿里巴巴（中国）有限公司 | Method and device for election in distributed system |
| CN115037745B (en) * | 2022-05-18 | 2023-09-26 | 阿里巴巴（中国）有限公司 | Method and device for electing in distributed system |
| CN115052006A (en) * | 2022-08-12 | 2022-09-13 | 佛山赛思禅科技有限公司 | Data synchronization method and system based on leader node |
| CN116955427A (en) * | 2023-09-18 | 2023-10-27 | 北京长亭科技有限公司 | Method and device for processing real-time multi-rule dynamic expression data based on Flink frame |
| CN116955427B (en) * | 2023-09-18 | 2023-12-15 | 北京长亭科技有限公司 | Method and device for processing real-time multi-rule dynamic expression data based on Flink frame |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2021129477A1 (en) | 2021-07-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113055430A (en) | Data synchronization method and related equipment | |
| US11379461B2 (en) | Multi-master architectures for distributed databases | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| KR20170110708A (en) | Opportunistic Resource Migration for Resource Deployment Optimization | |
| US20140074810A1 (en) | Managing a search index | |
| TWI701562B (en) | Database flexible scheduling method and device | |
| CA3044156C (en) | Adaptive service timeouts | |
| CN110648178A (en) | Method for increasing kafka consumption capacity | |
| WO2020042612A1 (en) | Method and device for storing and reading a message, server, and storage medium | |
| US11256719B1 (en) | Ingestion partition auto-scaling in a time-series database | |
| CN110673928A (en) | Thread binding method, thread binding device, storage medium and server | |
| US11593496B2 (en) | Decentralized data protection system for multi-cloud computing environment | |
| CN105760391B (en) | Method, data node, name node and system for dynamically redistributing data | |
| US11886225B2 (en) | Message processing method and apparatus in distributed system | |
| CN114518948A (en) | Large-scale microservice application-oriented dynamic perception rescheduling method and application | |
| US9893972B1 (en) | Managing I/O requests | |
| CN111400241B (en) | Data reconstruction method and device | |
| Shu et al. | Dynamic load balancing and channel strategy for apache flume collecting real-time data stream | |
| US11263184B1 (en) | Partition splitting in a distributed database | |
| WO2022252631A1 (en) | Method and apparatus for distributed application, and computer device and storage medium | |
| CN113239059B (en) | Switching method and device of distributed lock, server and storage medium | |
| CN115221116A (en) | Data writing method, device and equipment and readable storage medium | |
| CN111352916B (en) | Data storage method, system and storage medium based on NAS storage system | |
| Edmondson et al. | QoS-enabled distributed mutual exclusion in public clouds | |
| Meng et al. | A network load sensitive block placement strategy of HDFS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220218 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Applicant after: Huawei Cloud Computing Technology Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Applicant before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |