CN112559628A - Multi-cluster message synchronization method, device, medium and electronic equipment - Google Patents
Multi-cluster message synchronization method, device, medium and electronic equipment Download PDFInfo
- Publication number
- CN112559628A CN112559628A CN202011454591.1A CN202011454591A CN112559628A CN 112559628 A CN112559628 A CN 112559628A CN 202011454591 A CN202011454591 A CN 202011454591A CN 112559628 A CN112559628 A CN 112559628A
- Authority
- CN
- China
- Prior art keywords
- message queue
- data
- updating
- clusters
- cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000004590 computer program Methods 0.000 claims description 8
- 238000012216 screening Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims 1
- 239000002609 medium Substances 0.000 description 13
- 230000008569 process Effects 0.000 description 9
- 239000012634 fragment Substances 0.000 description 8
- 230000006870 function Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000013500 data storage Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000012120 mounting media Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000013468 resource allocation Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application discloses a method, a device, a medium and an electronic device for synchronizing multi-cluster messages. The method comprises the following steps: if the data updating events of a plurality of ES clusters are monitored, acquiring the object ID of a data producer; determining a target message queue for sending the updating data according to the object ID; controlling consumer groups of a plurality of ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group. By executing the scheme, the consistency of the data can be kept under the condition of updating the messages in multiple clusters.
Description
Technical Field
The embodiment of the application relates to the technical field of computers, in particular to a method, a device, a medium and electronic equipment for synchronizing multi-cluster messages.
Background
With the rapid development of economic society, the storage and the update of mass source data are often involved for some platforms, such as the platform of a freight transportation system. If the platform adopts a storage form of distributed cluster, the consistency of the storage and update processes is particularly important. Once consistency is in problem, the products which are already off-shelf may be updated, so that false data is generated, and the operation order of the platform is greatly influenced. Therefore, a technical solution for ensuring consistency in the storage and update processes under the condition of multiple clusters is urgently needed.
Disclosure of Invention
The embodiment of the application provides a method, a device, a medium and an electronic device for synchronizing multi-cluster messages, which can keep data consistency under the condition of message updating in multiple clusters.
In a first aspect, an embodiment of the present application provides a method for synchronizing multiple cluster messages, where the method includes:
if the data updating events of a plurality of ES clusters are monitored, acquiring the object ID of a data producer;
determining a target message queue for sending the updating data according to the object ID;
controlling consumer groups of a plurality of ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group.
Further, if a data update event for a plurality of ES clusters is monitored, obtaining an object ID of a data producer includes:
acquiring an initial message queue of data updating events of a plurality of ES clusters;
and carrying out validity check on the initial message queue to obtain the screened message, and acquiring the object ID of the data producer of the message.
Further, determining a target message queue for sending the update data according to the object ID includes:
and if the data updating event comprises goods source shelving and updating, after a target message queue is determined according to the object ID, putting the goods source shelving message and the goods source updating message into the same target message queue.
Further, determining a target message queue according to the object ID includes:
abs (object ID)% topic corresponds to the queue ID;
abs is an absolute value of an object ID, and topic is a message type.
Further, the rule of sequential consumption includes: and according to the writing sequence of the messages in the target message queue, consuming.
In a second aspect, an embodiment of the present application provides an apparatus for synchronizing multiple cluster messages, where the apparatus includes:
the object ID acquisition module is used for acquiring the object ID of the data producer if the data updating events of the plurality of ES clusters are monitored;
a target message queue determining module, configured to determine a target message queue for sending update data according to the object ID;
the consumption module is used for controlling the consumer groups of the ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group.
Further, the object ID obtaining module includes:
an initial message queue obtaining unit, configured to obtain an initial message queue of data update events for multiple ES clusters;
and the screening unit is used for carrying out validity check on the initial message queue to obtain screened messages and acquiring the object ID of the data producer of the messages.
Further, the target message queue determining module is specifically configured to:
and if the data updating event comprises goods source shelving and updating, after a target message queue is determined according to the object ID, putting the goods source shelving message and the goods source updating message into the same target message queue.
In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a synchronization method for multiple cluster messages according to the present application.
In a fourth aspect, an embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable by the processor, where the processor executes the computer program to implement the method for synchronizing multiple cluster messages according to the embodiment of the present application.
According to the technical scheme provided by the embodiment of the application, if data updating events of a plurality of ES clusters are monitored, the object ID of a data producer is obtained; determining a target message queue for sending the updating data according to the object ID; controlling consumer groups of a plurality of ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group. According to the technical scheme, the consistency of the data can be kept under the condition that the messages are updated in multiple clusters.
Drawings
Fig. 1 is a flowchart of a synchronization method for multiple cluster messages according to an embodiment of the present application;
FIG. 2 is a diagram illustrating a synchronization process for multiple cluster messages according to an embodiment of the present disclosure;
fig. 3 is a schematic structural diagram of a synchronization apparatus for multiple cluster messages according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the application and are not limiting of the application. It should be further noted that, for the convenience of description, only some of the structures related to the present application are shown in the drawings, not all of the structures.
Before discussing exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although a flowchart may describe the steps as a sequential process, many of the steps can be performed in parallel, concurrently or simultaneously. In addition, the order of the steps may be rearranged. The process may be terminated when its operations are completed, but may have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, and the like.
Fig. 1 is a flowchart of a multi-cluster message synchronization method provided in an embodiment of the present application, where the present embodiment is applicable to a situation of multi-cluster storage and update of mass source data, and the method can be executed by a multi-cluster message synchronization apparatus provided in an embodiment of the present application, where the apparatus can be implemented by software and/or hardware, and can be integrated in an electronic device running the system.
Aiming at the increase of business requirements, the scheme can perform individual operations such as an individual pricing strategy, a search display strategy, a transaction flow and the like for different owners, goods transportation routes, departure time periods and the like. For the same cargo information, different modules require different views of data to implement the corresponding functions. For search ranking, the main concern is the attributes that the source can use for searching, such as origin province, destination province, length, model, source type tag, etc. And all the source information comes from the data synchronization of the source center, in which case the success of writing and updating the source information is very important. In the above existing solutions, the execution logic and resource allocation for the internal service of the big data server are basically executed and acquired immediately, which may conflict with the actual use of the user.
In order to ensure the efficiency of searching data, mass data is stored in an ElasticSearch cluster, and due to the increasing search traffic of users, a set of clusters is gradually expanded into a plurality of ElasticSearch clusters storing the same data. But because a plurality of ElasticSearch clusters of the same storage data are introduced, the storage consistency and the query consistency of the data gradually expose different clusters.
The method aims to improve the concurrency of mass source data storage and the consistency of source data updating.
As shown in fig. 1, the method for synchronizing multiple cluster messages includes:
s110, if the data updating events of the ES clusters are monitored, the object ID of the data producer is obtained.
Among them, the ES cluster is called an ElasticSearch cluster. The ES cluster is a distributed full-text retrieval framework, a complex processing mechanism is hidden, and a fragmentation mechanism, cluster discovery and fragmentation load balancing request routing are used inside the ES cluster.
Shards represents index Shards, and the ES can divide a complete index into a plurality of Shards, so that the advantage of splitting a large index into a plurality of Shards and distributing the plurality of Shards to different nodes is realized. A distributed search is constructed. The number of slices can only be specified before index creation and cannot be changed after index creation.
The Replicas fragments represent index copies, the ES can set multiple index copies, and the copies have the function of improving the fault tolerance of the system and can be recovered from the copies when a certain fragment of a certain node is damaged or lost. And secondly, the query efficiency of the ES is improved, and the ES can automatically perform load balancing on the search request.
The core principle of the ES cluster is that each index is divided into a plurality of shards for storage, the default index creation is to allocate 5 shards for storage, each shard is deployed in a distributed manner on a plurality of different nodes, and the shard becomes a primary shards primary shard. Each main fragment has its own corresponding spare fragment in order to realize high availability, the spare fragment corresponding to the main fragment cannot be stored on the same server, and the main fragment and other spare fragments can be stored on the same node. There is no sub-sharding in a single ES server.
The number of the ES clusters may be two, three or more, and may be specifically determined according to the ES clusters required by the platform. The data updating event can be events such as addition, modification and deletion of data, and can be events such as shelving, updating and shelving of the goods source corresponding to the goods source system.
In the scheme, the data updating event of the plurality of ES clusters can be determined to be monitored by detecting that a shipper initiates a cargo source racking request, a cargo source updating request, a cargo source racking request and the like. Where the data producer may be a device used by the owner of the goods. The object ID at the time of the data producer generating the message may be acquired, and may be the source ID corresponding to the source system. It will be appreciated that the source ID of each source is uniquely present throughout the system.
In this scheme, optionally, if a data update event for a plurality of ES clusters is monitored, obtaining an object ID of a data producer includes:
acquiring an initial message queue of data updating events of a plurality of ES clusters;
and carrying out validity check on the initial message queue to obtain the screened message, and acquiring the object ID of the data producer of the message.
Specifically, after monitoring the message queue which is on shelf and updated by the source, checking the validity of the source for each message in the message queue. For example, each message in the message queue is taken out to obtain the source ID of each message, and whether each source ID is in an online state is verified, if yes, the message is valid, and if not, the message is invalid, and the message is deleted. By the arrangement, the obtained information which is put on shelf and updated by the goods source can be determined to be effective, so that the problem of the effectiveness of the information can be avoided from influencing the synchronization of the multi-cluster information.
And S120, determining a target message queue for sending the updating data according to the object ID.
Wherein, after obtaining the object ID of each message, the target message queue of the update data can be determined. It will be appreciated that there are many message queues, but how to determine the queue into which each message enters is determined here based on the object ID of the message. For example, messages with the same object ID may be entered into a queue, with messages on the shelf before messages and update messages after messages, thus ensuring that all data to be updated is in a message queue for an object.
In this scheme, specifically, determining a target message queue for sending update data according to the object ID includes:
and if the data updating event comprises goods source shelving and updating, after a target message queue is determined according to the object ID, putting the goods source shelving message and the goods source updating message into the same target message queue.
The method and the device have the advantages that the shelving messages and the updating messages of the same goods source are put into the same target message queue, and the messages of the goods source can be processed according to the message queue. It will be appreciated that there is a case where only update messages are present and no shelving messages are present, in which case the target message queue may be determined from the source ID and it is determined whether a shelving message for that source is present in the target message queue, and if not, the messages may be sorted by time of placement into the message queue, and if so, the update message may be placed into the shelving message before ensuring that the shelving message takes precedence over the update message in the message queue.
In this scheme, specifically, determining a target message queue according to the object ID includes:
abs (object ID)% topic corresponds to the queue ID;
abs is an absolute value of an object ID, and topic is a message type.
Specifically, after the preposed verification of the validity of the source data and the like is carried out, the source data information is sent to an MQ (message queue) of the user, and meanwhile, a consumption group of the user is established for each ElasticSearch cluster to consume the same source data information, so that the parallel writing of a plurality of ElasticSearch clusters is achieved, the mutual influence of the writing of the plurality of clusters is eliminated, and the concurrency of the source data storage is improved.
The producer selects the queue for message sending according to the source ID, and the shelving and updating messages for the same goods are sent to the same queue.
Abs (object ID)% topic corresponds to the queue ID;
based on the method, the original consumers are modified, and the sequential consumption attributes of the consumer groups are set, so that the consumer groups sequentially consume the messages in the same queue.
Through the arrangement, sequential processing of the messages can be ensured, so that concurrent processing of data by multiple clusters and data consistency are ensured.
S130, controlling consumer groups of a plurality of ES clusters to consume according to a sequential consumption rule for the target message queue; wherein each ES cluster is provided with one consumer group.
It can be understood that, in the process of consuming the target message queue, each ES cluster may use its own consumer group to consume, where the consumption does not sequentially consume the messages of the entire target message queue at a time, but consumes the messages according to the sequence in the target message queue. Thereby ensuring the validity and consistency of the data of multiple clusters.
Specifically, the rule of sequential consumption includes: and according to the writing sequence of the messages in the target message queue, consuming.
Through the arrangement, the sequential problems of shelf loading and updating of the goods sources can be solved, and the success of updating the data of the goods sources is ensured.
Fig. 2 is a schematic diagram of a synchronization process of multi-cluster messages provided in an embodiment of the present application, and as shown in fig. 2, after checking validity of a preposed source and the like by monitoring MQs (message queues) that are shelved and updated from the source, source information is sent to MQs of us, and a consumption group of the each ElasticSearch cluster is established to consume the same source information, so that parallel writing of multiple ElasticSearch clusters is achieved, mutual influence of writing of multiple clusters is eliminated, and concurrence of source data storage is increased.
Meanwhile, in order to solve the sequential problem of shelf and update of the goods sources, the following design is carried out for producers and consumers of MQ:
(1) the producer selects the queue for message sending according to the source id, and the shelving and updating messages for the same ticket goods are sent to the same queue.
Abs (source id)% topic corresponds to queue number;
(2) and transforming the original consumers, and setting the sequential consumption attribute of the consumer group, so that the consumer group sequentially consumes the messages in the same queue.
The technical scheme can overcome the defects in the prior art, and the prior art has the problems that the writing of a plurality of clusters is performed in a serialization mode, the writing success of one cluster can affect the writing of other clusters, the writing time is prolonged, and the writing concurrency is reduced. Meanwhile, the existing scheme has no strict sequence for the handling of the goods source shelving and goods source updating, so that a plurality of updating operations occur before the writing operation, which results in updating failure.
Fig. 3 is a schematic structural diagram of a synchronization apparatus for multiple cluster messages according to an embodiment of the present application, and as shown in fig. 3, the synchronization apparatus may include:
an object ID obtaining module 310, configured to obtain an object ID of a data producer if a data update event for multiple ES clusters is monitored;
a target message queue determining module 320, configured to determine a target message queue for sending update data according to the object ID;
a consuming module 330, configured to control, for the target message queue, a consumer group of the multiple ES clusters to consume according to a rule of sequential consumption; wherein each ES cluster is provided with one consumer group.
Further, the object ID obtaining module 310 includes:
an initial message queue obtaining unit, configured to obtain an initial message queue of data update events for multiple ES clusters;
and the screening unit is used for carrying out validity check on the initial message queue to obtain screened messages and acquiring the object ID of the data producer of the messages.
Further, the target message queue determining module 320 is specifically configured to:
and if the data updating event comprises goods source shelving and updating, after a target message queue is determined according to the object ID, putting the goods source shelving message and the goods source updating message into the same target message queue.
The multi-cluster message synchronization device provided by the embodiment of the invention can execute the multi-cluster message synchronization method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects for executing the multi-cluster message synchronization method.
Embodiments of the present application also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method for synchronization of multiple cluster messages, the method comprising:
if the data updating events of a plurality of ES clusters are monitored, acquiring the object ID of a data producer;
determining a target message queue for sending the updating data according to the object ID;
controlling consumer groups of a plurality of ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group.
Storage media refers to any of various types of memory electronics or storage electronics. The term "storage medium" is intended to include: mounting media such as CD-ROM, floppy disk, or tape devices; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Lanbas (Rambus) RAM, etc.; non-volatile memory such as flash memory, magnetic media (e.g., hard disk or optical storage); registers or other similar types of memory elements, etc. The storage medium may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the computer system in which the program is executed, or may be located in a different second computer system connected to the computer system through a network (such as the internet). The second computer system may provide the program instructions to the computer for execution. The term "storage medium" may include two or more storage media that may reside in different unknowns (e.g., in different computer systems connected by a network). The storage medium may store program instructions (e.g., embodied as a computer program) that are executable by one or more processors.
Of course, the storage medium containing computer-executable instructions provided in the embodiments of the present application is not limited to the operations of the synchronization method for multi-cluster messages described above, and may also perform related operations in the synchronization method for multi-cluster messages provided in any embodiments of the present application.
The embodiment of the present application provides an electronic device, where the synchronization apparatus for multiple cluster messages provided in the embodiment of the present application may be integrated in the electronic device, and the electronic device may be configured in a system or may be a device that performs part or all of functions in the system. Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. As shown in fig. 4, the present embodiment provides an electronic device 400, which includes: one or more processors 420; the storage device 410 is used for storing one or more programs, and when the one or more programs are executed by the one or more processors 420, the one or more processors 420 implement the method for synchronizing multiple cluster messages provided by the embodiment of the present application, the method includes:
if the data updating events of a plurality of ES clusters are monitored, acquiring the object ID of a data producer;
determining a target message queue for sending the updating data according to the object ID;
controlling consumer groups of a plurality of ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group.
Of course, those skilled in the art can understand that the processor 420 also implements the technical solution of the synchronization method for multi-cluster messages provided in any embodiment of the present application.
The electronic device 400 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present application.
As shown in fig. 4, the electronic device 400 includes a processor 420, a storage device 410, an input device 430, and an output device 440; the number of the processors 420 in the electronic device may be one or more, and one processor 420 is taken as an example in fig. 4; the processor 420, the storage device 410, the input device 430, and the output device 440 in the electronic apparatus may be connected by a bus or other means, and are exemplified by a bus 450 in fig. 4.
The storage device 410 is a computer-readable storage medium, and can be used to store software programs, computer-executable programs, and module units, such as program instructions corresponding to the synchronization method of multi-cluster messages in the embodiments of the present application.
The storage device 410 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the storage 410 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, storage 410 may further include memory located remotely from processor 420, which may be connected via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input means 430 may be used to receive input numbers, character information, or voice information, and to generate key signal inputs related to user settings and function control of the electronic device. The output device 440 may include a display screen, speakers, or other electronic equipment.
The electronic device provided by the embodiment of the application can keep the consistency of data under the condition of updating messages in multiple clusters.
The multi-cluster message synchronization device, the medium and the electronic device provided in the above embodiments may execute the multi-cluster message synchronization method provided in any embodiment of the present application, and have corresponding functional modules and beneficial effects for executing the method. Technical details not elaborated in the above embodiments may be referred to a synchronization method of a multi-cluster message provided in any of the embodiments of the present application.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (10)
1. A method for synchronizing multiple cluster messages, the method comprising:
if the data updating events of a plurality of ES clusters are monitored, acquiring the object ID of a data producer;
determining a target message queue for sending the updating data according to the object ID;
controlling consumer groups of a plurality of ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group.
2. The method of claim 1, wherein obtaining the object ID of the data producer if the data update event for the plurality of ES clusters is monitored comprises:
acquiring an initial message queue of data updating events of a plurality of ES clusters;
and carrying out validity check on the initial message queue to obtain the screened message, and acquiring the object ID of the data producer of the message.
3. The method of claim 1, wherein determining a target message queue for transmission of update data based on the object ID comprises:
and if the data updating event comprises goods source shelving and updating, after a target message queue is determined according to the object ID, putting the goods source shelving message and the goods source updating message into the same target message queue.
4. The method of claim 3, wherein determining a target message queue based on the object ID comprises:
abs (object ID)% topic corresponds to the queue ID;
abs is an absolute value of an object ID, and topic is a message type.
5. The method of claim 1, wherein the rules for sequential consumption comprise: and according to the writing sequence of the messages in the target message queue, consuming.
6. An apparatus for synchronizing multiple cluster messages, the apparatus comprising:
the object ID acquisition module is used for acquiring the object ID of the data producer if the data updating events of the plurality of ES clusters are monitored;
a target message queue determining module, configured to determine a target message queue for sending update data according to the object ID;
the consumption module is used for controlling the consumer groups of the ES clusters to consume according to a rule of sequential consumption for the target message queue; wherein each ES cluster is provided with one consumer group.
7. The apparatus of claim 6, wherein the object ID obtaining module comprises:
an initial message queue obtaining unit, configured to obtain an initial message queue of data update events for multiple ES clusters;
and the screening unit is used for carrying out validity check on the initial message queue to obtain screened messages and acquiring the object ID of the data producer of the messages.
8. The apparatus of claim 6, wherein the target message queue determining module is specifically configured to:
and if the data updating event comprises goods source shelving and updating, after a target message queue is determined according to the object ID, putting the goods source shelving message and the goods source updating message into the same target message queue.
9. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out a method for synchronizing a multi-cluster message according to any one of claims 1 to 5.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of synchronizing a multi-cluster message according to any of claims 1-5 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011454591.1A CN112559628A (en) | 2020-12-10 | 2020-12-10 | Multi-cluster message synchronization method, device, medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011454591.1A CN112559628A (en) | 2020-12-10 | 2020-12-10 | Multi-cluster message synchronization method, device, medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112559628A true CN112559628A (en) | 2021-03-26 |
Family
ID=75061838
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011454591.1A Withdrawn CN112559628A (en) | 2020-12-10 | 2020-12-10 | Multi-cluster message synchronization method, device, medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112559628A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114296910A (en) * | 2021-12-02 | 2022-04-08 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN118520051A (en) * | 2024-07-22 | 2024-08-20 | 深圳市灵智数字科技有限公司 | Data synchronization method and electronic equipment |
-
2020
- 2020-12-10 CN CN202011454591.1A patent/CN112559628A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114296910A (en) * | 2021-12-02 | 2022-04-08 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN114296910B (en) * | 2021-12-02 | 2022-11-22 | 武汉物易云通网络科技有限公司 | Multi-partition queue sequential consumption method and device |
| CN118520051A (en) * | 2024-07-22 | 2024-08-20 | 深圳市灵智数字科技有限公司 | Data synchronization method and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107391629B (en) | Method, system, server and computer storage medium for data migration between clusters | |
| US9367261B2 (en) | Computer system, data management method and data management program | |
| JP2007183904A (en) | Event processing system, method, apparatus and program | |
| JP5686034B2 (en) | Cluster system, synchronization control method, server device, and synchronization control program | |
| CN107153643B (en) | Data table connection method and device | |
| CN110941502A (en) | Message processing method, device, storage medium and equipment | |
| CN105468718B (en) | Data consistency processing method, device and system | |
| CN113760513B (en) | Distributed task scheduling method, device, equipment and medium | |
| CN112559628A (en) | Multi-cluster message synchronization method, device, medium and electronic equipment | |
| WO2023284473A1 (en) | Data management method and apparatus, computer device, and storage medium | |
| CN111475480A (en) | Log processing method and system | |
| CN111913793A (en) | Distributed task scheduling method, device, node equipment and system | |
| CN113094430A (en) | Data processing method, device, equipment and storage medium | |
| US20130318059A1 (en) | Transfer of data from transactional data sources to partitioned databases in restartable environment | |
| CN111427689B (en) | Cluster keep-alive method and device and storage medium | |
| EP3811227B1 (en) | Methods, devices and systems for non-disruptive upgrades to a distributed coordination engine in a distributed computing environment | |
| CN117076096A (en) | Task flow execution method and device, computer readable medium and electronic equipment | |
| JP5691306B2 (en) | Information processing system | |
| CN113254437A (en) | Batch processing job processing method and device | |
| CN114205354B (en) | Event management system, event management method, server, and storage medium | |
| CN115238006A (en) | Retrieval data synchronization method, device, equipment and computer storage medium | |
| CN110716814B (en) | Performance optimization method and device for inter-process large-data-volume communication | |
| CN108958967A (en) | A kind of method and server of data processing | |
| CN114020472A (en) | Data acquisition method, device, equipment and storage medium | |
| CN112463887A (en) | Data processing method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210326 |
|
| WW01 | Invention patent application withdrawn after publication |