CN113688111A - Cross-region message copying method, system, electronic equipment and readable storage medium - Google Patents
Cross-region message copying method, system, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN113688111A CN113688111A CN202111237836.XA CN202111237836A CN113688111A CN 113688111 A CN113688111 A CN 113688111A CN 202111237836 A CN202111237836 A CN 202111237836A CN 113688111 A CN113688111 A CN 113688111A
- Authority
- CN
- China
- Prior art keywords
- cloud environment
- target
- cluster
- log
- cross
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a cross-region message copying method, a system, electronic equipment and a readable storage medium, wherein the cross-region message copying method is applied to a target cloud environment and comprises the following steps: receiving a log backup sent by a child cloud environment, and storing the log backup into a target cluster; and inputting the log backup in the target cluster into a consumption service for data consumption, wherein the target cloud environment comprises the target cluster and the consumption service. The invention can solve the technical problems of non-uniform service architecture and inconvenient maintenance in cross-region message replication in the prior art.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to a method, a system, an electronic device, and a readable storage medium for cross-region message replication.
Background
In the data docking process of the cloud native era, cross-cloud, cross-machine room and cross-regional copying are often required, and the cross-regional copying refers to that clusters dispersed in different regions can be copied with each other in a certain configuration mode, and the cross-regional copying function can meet the requirements of container cross-regional disaster tolerance or user data copying. The current conventional implementation is: the cross-region replication is realized by deploying a plurality of sets of service individual bearer services in different cloud environments, but the method has the disadvantages of non-uniform service architecture, inconvenience in maintenance and risk of data loss.
Disclosure of Invention
The invention mainly aims to provide a cross-region message copying method, a cross-region message copying system, electronic equipment and a readable storage medium, and aims to solve the technical problems that service architectures in cross-region message copying are not uniform and maintenance is inconvenient in the prior art.
In order to achieve the above object, the present invention provides a cross-region message replication method, where the cross-region message replication method is applied to a target cloud environment, where the target cloud environment includes a target cluster and a consumption service, and the cross-region message replication method includes:
receiving a log backup sent by a child cloud environment, and storing the log backup into the target cluster;
and inputting the log backup in the target cluster into the consumption service for data consumption.
Optionally, the step of receiving a log backup sent by a child cloud environment includes:
detecting whether the target cloud environment fails;
if the target cloud environment fails, sending failure information of the failure of the target cloud environment to a child cloud environment, and stopping receiving log backup sent by the child cloud environment;
and if the target cloud environment fault recovery is detected, sending the fault recovery information of the target cloud environment fault recovery to the child cloud environment, and re-receiving the log backup sent by the child cloud environment.
Optionally, the step of storing the log backup in the target cluster includes:
acquiring main framework log data corresponding to a target cloud environment, and storing the main framework log data and the log backup to a target cluster.
Optionally, the cross-region message replication method is applied to a child cloud environment, and the steps of the cross-region message replication method include:
collecting cloud environment log data corresponding to a child cloud environment, and storing the cloud environment log data to a local subset group;
and if the cloud environment log data need to be copied to the target cluster, copying the cloud environment log data to obtain a log backup, and sending the log backup to the target cluster.
Optionally, the step of sending the log backup to a target cluster includes:
if fault information of a fault of the target cloud environment sent by the target cloud environment is received, stopping sending the log backup to the target cluster;
and if fault recovery information sent by the target cloud environment is received, the log backup is sent to the target cluster again.
Optionally, before the step of sending the log backup to the target cluster, the method includes:
determining all matched child cloud environments that match the child cloud environment;
and determining a target cloud environment in each matched sub-cloud environment.
Optionally, after the step of determining a target cloud environment in each of the matched sub-cloud environments, the method further includes:
determining other cloud environments in each matched sub-cloud environment except the target cloud environment;
and acquiring a target cluster in the target cloud environment, acquiring other clusters in other cloud environments, and constructing the local sub-cluster, the target cluster and the other clusters into a cluster network, so that the local sub-cluster and the other clusters respectively send log backups to the target clusters.
In addition, in order to achieve the above object, the present invention further provides a cross-regional message replication system, including a child cloud environment and the target cloud environment, wherein the target cloud environment includes a target cluster and a consumption service,
the child cloud environment is used for acquiring cloud environment log data corresponding to the child cloud environment and storing the cloud environment log data to a local sub-cluster;
the child cloud environment is used for copying the cloud environment log data to obtain a log backup if the cloud environment log data needs to be copied to a target cluster, and sending the log backup to the target cluster;
the target cloud environment is used for receiving the log backup sent by the sub-cloud environment and storing the log backup into the target cluster;
and the target cloud environment is used for inputting the log backup in the target cluster into the consumption service for data consumption.
In addition, to achieve the above object, the present invention also provides an electronic device including: the computer program product comprises a memory, a processor and a program stored on the memory and executable on the processor, wherein the program implementing the cross-region message copying method can implement the steps of the cross-region message copying method as described above when the program implementing the cross-region message copying method is executed by the processor.
In addition, to achieve the above object, the present application further provides a readable storage medium, which is a computer readable storage medium, on which a program for implementing a cross-region message copying method is stored, and when the program for implementing the cross-region message copying method is executed by a processor, the steps of the cross-region message copying method are implemented.
The invention provides a cross-region message copying method, a cross-region message copying system, electronic equipment and a readable storage medium, wherein the cross-region message copying method is applied to a target cloud environment, the target cloud environment comprises a target cluster and a consumption service, a log backup sent by a sub-cloud environment is received, the log backup is stored in the target cluster, and the log backup in the target cluster is input to the consumption service for data consumption. According to the mode, the target cloud environment receives log backups sent by all the sub-cloud environments, all the log backups are stored through the target cloud environment, cross-region message copying can be achieved without deploying one set of service for each cloud environment, so that the deployment cost is saved, the efficiency of cross-region message copying is improved, the log backups in the target cluster are further input to consumption services for data consumption, the service architecture is unified, only the service architecture of the target cluster needs to be maintained, and the convenience of service architecture maintenance is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
FIG. 1 is a schematic diagram of an apparatus architecture of a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a cross-region message replication method according to a first embodiment of the present invention;
FIG. 3 is a flowchart illustrating a cross-region message replication method according to a second embodiment of the present invention;
FIG. 4 is a schematic diagram of an application scenario in the cross-region message copying method of the present invention;
fig. 5 is a schematic diagram of a system architecture related to the cross-region message copying method of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the technical solutions in the embodiments may be combined with each other, but must be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination of the technical solutions should not be considered to exist, and is not within the protection scope of the present invention.
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, fig. 1 is a schematic device structure diagram of a hardware operating environment according to an embodiment of the present invention.
As shown in fig. 1, the device may be an electronic device, and the electronic device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and optionally, the user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory (e.g., a magnetic disk memory). The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration of the apparatus shown in fig. 1 is not intended to be limiting of the apparatus and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is a kind of computer storage medium, may include therein an operating system, a network communication module, a user interface module, and a computer program.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to invoke the computer program stored in the memory 1005 and perform the following operations:
receiving a log backup sent by a child cloud environment, and storing the log backup into a target cluster;
and inputting the log backup in the target cluster into a consumption service for data consumption.
Based on the above hardware structure, various embodiments of the cross-region message copying method of the present invention are provided.
The invention provides a cross-region message copying method.
Referring to fig. 2, fig. 2 is a flowchart illustrating a cross-region message replication method according to a first embodiment of the present invention.
In this embodiment, the cross-region message replication method is applied to a target cloud environment, where the target cloud environment includes a target cluster and a consumption service, and the cross-region message replication method includes the steps of:
step S10, receiving the log backup sent by the child cloud environment, and storing the log backup in a target cluster;
and step S20, inputting the log backup in the target cluster to a consumption service for data consumption.
In this embodiment, the cross-region message replication includes cross-cloud environment, cross-region message replication, and the like, and the cross-region message replication is implemented based on a pulsar cluster in this embodiment. The Pulsar cluster is a multi-tenant for server-to-server message delivery, supports production and consumption of messages at different regional locations, and has message replication capability. For example, an application publishes data in a certain region or market, but needs to process and consume data in other regions or markets, cross-machine-room replication of Pulsar may enable replication of persistently stored message data across multiple cross-cloud clusters. Further, in this embodiment, the target cluster is a Pulsar cluster, that is, the target cluster is a target Pulsar cluster, and in another embodiment, similarly, based on the Pulsar cluster, cross-region replication between different regions in the same cloud environment can also be achieved, and meanwhile, a single instance of the Pulsar natively supports multiple clusters, and message replication can be seamlessly completed between the clusters across a machine room.
In this embodiment, a target cloud environment may be predetermined, the target cloud environment receives the log backup sent by the child cloud environment, stores the log backup in the target cluster, and inputs the log backup in the target cluster to a consumption service in the target cloud environment for data consumption. The log backup is obtained by copying log data acquired in the sub-cloud environment. Specifically, referring to fig. 4, the cloud environment 2 in fig. 4 is taken as a target cloud environment, after receiving the log backup sent by the cloud environment 1 and the cloud environment 3, the cloud environment 2 stores the log backup in a target cluster (pulser cluster 2) corresponding to the cloud environment 2, and the target cluster pulser cluster2 sends the log backup to an output port, where the output port of this embodiment may be an output consumption service, and after data consumption, batch write and store the log backup data and the data consumption records. In this embodiment, the consumer is provided with subscription consumption according to different monitoring subjects through an output consumption service, wherein the subscription modes may include multiple topic subscription modes such as an exclusive subscription mode, a sharing subscription mode, a failover subscription mode and the like, and finally, the log backup data and the data consumption records are written in and stored in batch, so that data persistence is ensured, and the cross-region message subscription method under the log collection scene is realized.
Further, in step S10, the step of storing the log backup in the target cluster includes:
step a, acquiring main framework log data corresponding to a target cloud environment, and storing the main framework log data and the log backup to a target cluster.
In this embodiment, the target cloud environment acquires main framework log data corresponding to the target cloud environment, and the main framework log data and the log backup are stored in the target cluster in batch, so that the comprehensiveness and integrity of the log data are ensured. The main framework log data are log data corresponding to the target cloud environment acquired by the target cloud environment.
Further, in step S10, the refinement of the step of receiving the log backup sent by the child cloud environment includes:
step A, detecting whether the target cloud environment has a fault;
in this embodiment, the target cloud environment may be down or otherwise have a fault, and at this time, the child cloud environment cannot send the log backup to the target cloud environment. According to the embodiment, whether the target cloud environment fails or not can be detected by the timer every preset time, and of course, whether the target cloud environment fails or not can be detected in real time, when the target cloud environment fails or not, the target cloud environment runs normally, at this time, the log backup can be sent to the target cluster for backup through the local pulsar sub-cluster corresponding to the sub-cloud environment, and normal backup of log data can be guaranteed.
Step B, if the target cloud environment fails, sending failure information of the failure of the target cloud environment to a child cloud environment, and stopping receiving log backup sent by the child cloud environment;
and step C, if the target cloud environment fault recovery is detected, sending the fault recovery information of the target cloud environment fault recovery to the child cloud environment, and re-receiving the log backup sent by the child cloud environment.
In this embodiment, if it is detected that the target cloud environment has a failure, the failure information of the failure of the target cloud environment is sent to the child cloud environment, and the receiving of the log backup sent by the child cloud environment is stopped. So as to prevent the situation that data confusion is caused by continuously receiving log backup when the target cloud environment fails. Further, whether the target cluster is recovered to be normal or not can be detected in real time through the timer, if the target cloud environment fault recovery is detected, the target cloud environment is indicated to be normal in operation, the log data can be continuously backed up, at the moment, the fault recovery information of the target cloud environment fault recovery is sent to the sub-cloud environment, the log backup sent by the sub-cloud environment is received again, and the completeness of the log data is guaranteed. As shown in fig. 4, if it is detected that a failure occurs in the target cloud environment (cloud environment 2), both cloud environment 1 and cloud environment 3 cannot copy the log backup of the log data to cloud environment 2 normally, cloud environment 2 sends failure information to cloud environment 1 and cloud environment 3, and stops receiving the log backup sent by cloud environment 1 and cloud environment 3, and if it is detected that a failure of cloud environment 2 is recovered, failure recovery information of failure recovery of cloud environment 2 is sent to cloud environment 1 and cloud environment 3, and the log backup sent by cloud environment 1 and cloud environment 3 is received again. In an embodiment, the message replication of the log data can be completed in the local sub-cluster to obtain a log backup, and the log backup is sent to the target cluster.
In this embodiment, when a target cluster fails, the pulser-based message persistence is performed, and when any one area fails or goes down, the risk of data loss can be effectively avoided.
The embodiment provides a cross-region message copying method, which is applied to a target cloud environment, and based on pulser technology, message persistence, low delay and cross-machine room inter-cluster message copying, can receive log backups sent by a sub-cloud environment through the target cloud environment, store the log backups into a target cluster in the target cloud environment, and input the log backups in the target cluster into a consumption service in the target cloud environment for data consumption. According to the mode, the target cloud environment receives log backups sent by all the sub-cloud environments, all the log backups are stored through the target cloud environment, namely different cloud environments can copy messages to a certain cloud environment to achieve cross-regional message copying, and cross-regional message copying can be achieved without deploying a set of service to each cloud environment, so that the deployment cost is saved, the cross-regional message copying efficiency is improved, the log backups in the target cluster are further input to a consumption service to perform data consumption, the service architecture is unified, the service architecture of the target cluster only needs to be maintained, and the service architecture maintenance convenience is improved.
Further, the invention provides a cross-region message copying method.
Referring to fig. 3, fig. 3 is a flowchart illustrating a screen projection control method according to a second embodiment of the present invention.
In a second embodiment, the cross-region message replication method is applied to a child cloud environment, and includes:
step S30, collecting cloud environment log data corresponding to the child cloud environment, and storing the cloud environment log data to a local sub-cluster;
step S40, if the cloud environment log data needs to be copied to the target cluster, copying the cloud environment log data to obtain a log backup, and sending the log backup to the target cluster.
In this embodiment, the local sub-cluster is a pulsar cluster, that is, the local sub-cluster is a local pulsar sub-cluster, and the sub-cloud environment acquires cloud environment log data corresponding to the sub-cloud environment and stores the cloud environment log data to the local sub-cluster. The cloud environment log data is log data corresponding to the child cloud environment acquired by the child cloud environment. And if the cloud environment log data need to be copied to the target cluster, copying the cloud environment log data to obtain a log backup, and sending the log backup to the target cluster through the local sub-cluster corresponding to the sub-cloud environment. In an embodiment, message replication of log data can be completed in the local sub-cluster to obtain a log backup, and the log backup is sent to the target cluster. Referring to fig. 4, in fig. 4, cloud environment 1 is a child cloud environment, and pulsar cluster1 is a local sub-cluster corresponding to cloud environment 1. Taking cloud environment 1 as an example, cloud environment 1 may collect cloud environment log data corresponding to cloud environment 1 by a principle of proximity, and report the cloud environment log data to a local sub-cluster (pulsar cluster 1) corresponding to cloud environment 1, and if the cloud environment log data needs to be copied to a target cluster (pulsar cluster 2), copy the cloud environment log data by using local sub-cluster (pulsar cluster 1) to obtain a log backup, and send the log backup to a target cluster (pulsar cluster 2) to implement message copying, where the target cluster corresponding to this embodiment is a target cluster (pulsar cluster 2) corresponding to cloud environment 2 as shown in fig. 3. The target cloud environment can receive log backups sent by the plurality of cloud environments.
Specifically, in this embodiment, the cloud environment log data corresponding to the child cloud environment is acquired, the cloud environment log data is stored in the local sub-cluster, if the cloud environment log data needs to be copied to the target cluster, the cloud environment log data is copied to obtain a log backup, and a specific process of sending the log backup to the target cluster may be: through an Application integration toolkit in a child cloud environment or through a service integration toolkit, the toolkit may be sdk (a software development toolkit), cloud environment log data is collected through an API (Application Programming Interface) in sdk (Application Programming Interface) package, sdk sends the collected cloud environment log data to a local agent in the child cloud environment, the local agent may be an agent process, the agent uploads the log data to an input Interface of a log service, the input Interface may be an input collector, the input collector writes the log data into a local pulsar sub-cluster, the local pulsar sub-cluster receives the log data, and if the cloud environment log data needs to be copied to a target cluster, the log data is copied through the local pulsar sub-cluster to obtain a log backup, and the log backup is sent to the target pulsar cluster. The API is a predefined interface (e.g., a function, an HTTP interface), and the software development kit sdk is a set of development tools for creating application software for a specific software package, software framework, hardware platform, operating system, and the like.
Referring to fig. 4, taking cloud environment 1 as an example, cloud environment 1 includes a client, collects cloud environment log data through an application or service integration sdk toolkit of the client through an API in a sdk toolkit, and then sdk toolkit sends the collected cloud environment log data to a local agent in cloud environment 1, the agent uploads the log data to an input interface input collector of a log service, the input collector writes the log data into a local pulser sub-cluster (pulser cluster 1), and cluster pulser cluster1 copies the cloud environment log data to obtain a log backup, and then sends the log backup to a target cluster pulser cluster 2.
Specifically, in the step S40, the refinement of the step of sending the log backup to the target cluster includes:
step D, if fault information of a fault of the target cloud environment sent by the target cloud environment is received, stopping sending the log backup to the target cluster;
and step E, if fault recovery information sent by the target cloud environment is received, the log backup is sent to the target cluster again.
In this embodiment, if receiving fault information of a fault occurring in a target cloud environment sent by the target cloud environment, the child cloud environment indicates that the target cloud environment is running in a fault, at this time, sending the log backup to the target cluster should be stopped, the local sub-clusters corresponding to the child cloud environments can keep and store the log backup to be sent in the local sub-clusters, and if receiving fault recovery information sent by the target cloud environment, the child cloud environment sends the log backup to be sent to the target cluster again, so that message synchronization is achieved, and the integrity of log data is ensured. Wherein the sub-cloud environment may store log data or log backup reservations in a local cache or storage unit. As shown in fig. 3, if the cloud environment 1 and the cloud environment 3 receive the failure information sent by the cloud environment 2, at this time, the pulsar cluster1 and the pulsar cluster3 stop sending the log backup to the target cluster pulsar cluster2, and respectively perform the retention storage on the corresponding log data or log backup.
The embodiment provides a cross-region message copying method, which is applied to a child cloud environment, collects cloud environment log data corresponding to the child cloud environment, stores the cloud environment log data to a local sub-cluster, copies the cloud environment log data to obtain a log backup if the cloud environment log data needs to be copied to a target cluster, and sends the log backup to the target cluster. By the mode, each sub-cloud environment can back up the log data of the sub-cloud environment in the target environment, cross-region message copying is achieved, service deployment cost is reduced, and cross-region message copying efficiency is improved.
Further, based on the second embodiment of the present invention, a third embodiment of the cross-region message copying method according to the present invention is provided, and in this embodiment, before the step of sending the log backup to the target cluster in step S40, the method includes:
step F, determining all matched sub-cloud environments matched with the sub-cloud environments;
and G, determining a target cloud environment in each matched sub-cloud environment.
In this embodiment, all matching child cloud environments that match the child cloud environments are determined, and a target cloud environment in each matching child cloud environment is determined. I.e. determining all cloud environments that need to be replicated across regional messages. As one of the embodiments, for example: as shown in fig. 4, if it is determined in fig. 4 that the sub-cloud environment is cloud environment 1, it is determined that the matching sub-cloud environment matched with the sub-cloud environment is cloud environment 2 and cloud environment 3, and it is determined that the target cloud environment is in cloud environment 2 and cloud environment 3, in this embodiment, cloud environment 2 is used as a main framework environment (i.e., target cloud environment), and pulsar cluster2 in the target cloud environment is a target pulsar cluster. In the embodiment, the log data is copied to the target cluster corresponding to the target cloud environment for backup through each sub-cloud environment. Local sub-clusters pulsar cluster1 and pulsar cluster3, such as cloud environment 1 and cloud environment 3, copy log data into target cluster pulsar cluster 2. In this embodiment, each sub-cloud environment maintains symmetry of the architecture, so as to facilitate uniform maintenance of the service.
Step H, determining other cloud environments except the target cloud environment in each matched sub-cloud environment;
step I, acquiring a target cluster in the target cloud environment, acquiring other clusters in other cloud environments, and constructing the local sub-cluster, the target cluster and the other clusters into a cluster network, so that the local sub-cluster and the other clusters respectively send log backups to the target clusters.
In this embodiment, determining other cloud environments in each matching sub-cloud environment except for the target cloud environment, obtaining a target cluster in the target cloud environment, obtaining other clusters in the other cloud environments, and constructing the local sub-cluster, the target cluster, and the other clusters as a cluster network, for example, the matching sub-cloud environments are cloud environment 2 and cloud environment 3, where cloud environment 2 is the target cloud environment, determining other cloud environments in the matching sub-cloud environment except for target cloud environment 2, that is, cloud environment 3, obtaining target cluster pulsar cluster2 in target cloud environment 2, obtaining other cluster pulsar cluster3 in other cloud environment 3, constructing local sub-cluster pulsar cluster1, target cluster pulsar cluster2, and other cluster3 as a pulsar cluster network, so that the local sub-cluster and the other clusters respectively send logs to the target cluster, as shown in fig. 4, cluster1, The "pulsar cluster 2" and the "pulsar cluster 3" form a pulsar cluster network, and the cluster pulsar cluster1 and the cluster pulsar cluster3 can be implemented in the pulsar cluster network to copy log data into a target cluster pulsar cluster 2. In this embodiment, the target cluster and the plurality of local pulser sub-clusters form a cluster network, so that each local pulser sub-cluster copies the log data to the target cluster in the cluster network.
In the embodiment, messages of all cloud environments are stored in a target cluster in a centralized manner, cross-cloud, cross-machine-room and cross-region log data message copying is realized based on functions of message persistence, low delay and cross-machine-room and inter-cluster message copying of the pulsar cluster, and further a set of service does not need to be deployed in each different environment to receive log data, so that the service architecture is unified, the convenience of service architecture maintenance is improved, and the operation and maintenance cost is further reduced.
Corresponding to the first embodiment, the second embodiment, and the third embodiment, an embodiment of the present application further provides a cross-region message replication system, where the cross-region message replication system includes: the child cloud environment and the target cloud environment, specifically, refer to fig. 5, which is a schematic diagram of a system architecture related to the cross-region message replication method of the present invention, wherein the child cloud environment performs message replication of data to the target cloud environment, the data performing message replication in this embodiment is log data, wherein,
the child cloud environment is used for acquiring cloud environment log data corresponding to the child cloud environment and storing the cloud environment log data to a local sub-cluster;
the child cloud environment is used for copying the cloud environment log data to obtain a log backup if the cloud environment log data needs to be copied to a target cluster, and sending the log backup to the target cluster;
the target cloud environment is used for receiving the log backup sent by the sub-cloud environment and storing the log backup into the target cluster;
and the target cloud environment is used for inputting the log backup in the target cluster into a consumption service for data consumption.
In the embodiments of the present application, the same or corresponding contents as those in the first embodiment, the second embodiment, or the third embodiment are referred to the above description, and are not repeated herein.
The cross-region message copying system provided by the embodiment of the invention adopts the cross-region message copying method provided by the embodiment, thereby solving the technical problems of non-uniform service architecture and inconvenient maintenance in cross-region message copying. Compared with the prior art, the beneficial effect of the cross-region message copying system provided by the embodiment of the present invention is the same as that of the cross-region message copying method provided by the first embodiment, the second embodiment, or the third embodiment, and other technical features of the cross-region message copying system are the same as those disclosed in the first embodiment, the second embodiment, or the third embodiment, and thus no further description is given here.
The present embodiments provide a computer-readable storage medium having computer-readable program instructions stored thereon for performing a cross-region message duplication method of one of the above-described embodiments.
The computer readable storage medium provided by the embodiments of the present invention may be a U disk, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present embodiment, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, or device. Program code embodied on a computer readable storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer-readable storage medium provided by the invention stores the computer-readable program instruction for executing the cross-region message copying method, and solves the technical problems of non-uniform service architecture and inconvenient maintenance in cross-region message copying. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the embodiment of the present invention are the same as the beneficial effects of the cross-region message copying method provided by the first embodiment, the second embodiment, or the third embodiment, and are not described herein again.
The present application further provides an electronic device, the device comprising: at least one processor, and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the cross-region message duplication method as described above.
The electronic equipment solves the technical problems of non-uniform service architecture and inconvenient maintenance in cross-region message copying. Compared with the prior art, the beneficial effects of the electronic device provided by the embodiment of the present invention are the same as the beneficial effects of the cross-region message copying method provided by the first embodiment, the second embodiment, or the third embodiment, and are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A cross-regional message replication method is applied to a target cloud environment, the target cloud environment comprises a target cluster and a consumption service, and the cross-regional message replication method comprises the following steps:
receiving a log backup sent by a child cloud environment, and storing the log backup into the target cluster;
and inputting the log backup in the target cluster into the consumption service for data consumption.
2. The cross-region message replication method of claim 1, wherein the step of receiving a log backup sent by a child cloud environment comprises:
detecting whether the target cloud environment fails;
if the target cloud environment fails, sending failure information of the failure of the target cloud environment to a child cloud environment, and stopping receiving log backup sent by the child cloud environment;
and if the fault recovery of the target cloud environment is detected, sending the fault recovery information of the fault recovery of the target cloud environment to a child cloud environment, and re-receiving the log backup sent by the child cloud environment.
3. The cross-region message replication method of claim 1, wherein the step of storing the log backup in the target cluster comprises:
acquiring main framework log data corresponding to a target cloud environment, and storing the main framework log data and the log backup to a target cluster.
4. A cross-region message copying method is applied to a child cloud environment, and the steps of the cross-region message copying method comprise:
collecting cloud environment log data corresponding to a child cloud environment, and storing the cloud environment log data to a local subset group;
and if the cloud environment log data need to be copied to the target cluster, copying the cloud environment log data to obtain a log backup, and sending the log backup to the target cluster.
5. The cross-region message replication method of claim 4, wherein the step of sending the log backup to a target cluster comprises:
if fault information of a fault of the target cloud environment sent by the target cloud environment is received, stopping sending the log backup to the target cluster;
and if fault recovery information sent by the target cloud environment is received, the log backup is sent to the target cluster again.
6. The cross-region message replication method of claim 4, wherein the step of sending the log backup to the target cluster is preceded by:
determining all matched child cloud environments that match the child cloud environment;
and determining a target cloud environment in each matched sub-cloud environment.
7. The cross-region message replication method of claim 6, wherein the step of determining a target cloud environment in each of the matched child cloud environments is followed by further comprising:
determining other cloud environments in each matched sub-cloud environment except the target cloud environment;
and acquiring a target cluster in the target cloud environment, acquiring other clusters in other cloud environments, and constructing the local sub-cluster, the target cluster and the other clusters into a cluster network, so that the local sub-cluster and the other clusters respectively send log backups to the target clusters.
8. A cross-regional message replication system comprising a child cloud environment and a target cloud environment, wherein the target cloud environment comprises a target cluster and a consuming service,
the child cloud environment is used for acquiring cloud environment log data corresponding to the child cloud environment and storing the cloud environment log data to a local sub-cluster;
the child cloud environment is used for copying the cloud environment log data to obtain a log backup if the cloud environment log data needs to be copied to a target cluster, and sending the log backup to the target cluster;
the target cloud environment is used for receiving the log backup sent by the sub-cloud environment and storing the log backup into the target cluster;
and the target cloud environment is used for inputting the log backup in the target cluster into the consumption service for data consumption.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the steps of the cross-region message duplication method of any one of claims 1 to 3 and 4 to 7.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon a program for implementing a cross-region message copying method, the program being executed by a processor to implement the steps of the cross-region message copying method according to any one of claims 1 to 3 and claims 4 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111237836.XA CN113688111A (en) | 2021-10-25 | 2021-10-25 | Cross-region message copying method, system, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111237836.XA CN113688111A (en) | 2021-10-25 | 2021-10-25 | Cross-region message copying method, system, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113688111A true CN113688111A (en) | 2021-11-23 |
Family
ID=78587861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111237836.XA Pending CN113688111A (en) | 2021-10-25 | 2021-10-25 | Cross-region message copying method, system, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113688111A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679841A (en) * | 2015-02-11 | 2015-06-03 | 北京京东尚科信息技术有限公司 | Consumption terminal data flow copying method and system |
| CN108573006A (en) * | 2017-06-06 | 2018-09-25 | 北京金山云网络技术有限公司 | Across computer room data synchronous system, method and device, electronic equipment |
| CN111400065A (en) * | 2020-03-13 | 2020-07-10 | 百融云创科技股份有限公司 | Pulsar message long-distance multi-live method and system for separating global zookeeper |
| CN112905390A (en) * | 2021-03-31 | 2021-06-04 | 恒生电子股份有限公司 | Log data backup method, device, equipment and storage medium |
-
2021
- 2021-10-25 CN CN202111237836.XA patent/CN113688111A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104679841A (en) * | 2015-02-11 | 2015-06-03 | 北京京东尚科信息技术有限公司 | Consumption terminal data flow copying method and system |
| CN108573006A (en) * | 2017-06-06 | 2018-09-25 | 北京金山云网络技术有限公司 | Across computer room data synchronous system, method and device, electronic equipment |
| CN111400065A (en) * | 2020-03-13 | 2020-07-10 | 百融云创科技股份有限公司 | Pulsar message long-distance multi-live method and system for separating global zookeeper |
| CN112905390A (en) * | 2021-03-31 | 2021-06-04 | 恒生电子股份有限公司 | Log data backup method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108712501B (en) | Information sending method and device, computing equipment and storage medium | |
| TWI608419B (en) | Method for pre-testing software compatibility and system thereof | |
| TWI575362B (en) | Backup method, pre-testing method for enviornment updating and system thereof | |
| CN106487486B (en) | Service processing method and data center system | |
| JP2021524104A (en) | Master / Standby Container System Switching | |
| CN107682169B (en) | Method and device for sending message by Kafka cluster | |
| CN113312153B (en) | Cluster deployment method and device, electronic equipment and storage medium | |
| CN113051110A (en) | Cluster switching method, device and equipment | |
| CN112612545A (en) | Configuration hot loading system, method, equipment and medium of server cluster | |
| CN111865632A (en) | Switching method of distributed data storage cluster and switching instruction sending method and device | |
| CN112527507A (en) | Cluster deployment method and device, computer equipment and storage medium | |
| CN115658390A (en) | Container disaster tolerance method, system, device, equipment and computer readable storage medium | |
| CN108173665B (en) | Data backup method and device | |
| CN112000567A (en) | Regulation and control software test service method based on cloud platform | |
| CN111930565B (en) | Process fault self-healing method, device and equipment for components in distributed management system | |
| CN112181724A (en) | Big data disaster tolerance method and device and electronic equipment | |
| CN110096226B (en) | Disk array deployment method and device | |
| CN113688111A (en) | Cross-region message copying method, system, electronic equipment and readable storage medium | |
| CN115016917A (en) | Task control method and device and readable storage medium | |
| CN116166470A (en) | Redis cluster clone replication method and device, medium and equipment | |
| CN101242201B (en) | A master-slave system maintenance method, system and device | |
| JP2015114952A (en) | Network system, monitoring control unit, and software verification method | |
| CN112488462A (en) | Unified pushing method, device and medium for workflow data | |
| CN115145782A (en) | Server switching method, mooseFS system and storage medium | |
| CN114189429A (en) | System, method, device and medium for monitoring server cluster faults |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211123 |