CN117112181A - Data synchronization method and device, computer readable storage medium and electronic equipment - Google Patents
Data synchronization method and device, computer readable storage medium and electronic equipment Download PDFInfo
- Publication number
- CN117112181A CN117112181A CN202311262814.8A CN202311262814A CN117112181A CN 117112181 A CN117112181 A CN 117112181A CN 202311262814 A CN202311262814 A CN 202311262814A CN 117112181 A CN117112181 A CN 117112181A
- Authority
- CN
- China
- Prior art keywords
- task
- synchronization
- data
- target
- working node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 98
- 230000001360 synchronised effect Effects 0.000 claims abstract description 159
- 238000012544 monitoring process Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 6
- 238000012545 processing Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 230000002085 persistent effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000004886 process control Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Embodiments of the present disclosure relate to the field of big data technology, and more particularly, to a data synchronization method and apparatus, a computer readable storage medium, and an electronic device. The method comprises the following steps: acquiring data to be synchronized, and creating a synchronization task corresponding to the data to be synchronized; determining a target working node for scheduling service, starting a task submitting process in the target working node, and submitting the synchronous task to a target resource manager through the task submitting process; and starting the synchronous task in the target resource manager through the target working node to finish the synchronization of the data to be synchronized. The method reduces the load of the target working node.
Description
Technical Field
Embodiments of the present disclosure relate to the field of big data technology, and more particularly, to a data synchronization method and apparatus, a computer storage medium, and an electronic device.
Background
This section is intended to provide a background or context for the embodiments of the disclosure recited in the claims, which description herein is not admitted to be prior art by inclusion in this section.
In the field of big data, the data needs to be synchronized to a storage service after being processed in a plurality of bins, and currently, offline data synchronization is mainly completed through a datax (heterogeneous data source offline synchronization tool), offline tasks are started by a scheduling service, and the scheduling service is usually a high-availability architecture of multiple monitoring nodes and multiple working nodes. Each monitoring node and each working node are deployed on a separate machine, the working node is responsible for specific task starting of the datax, and in synchronous tasks, the working node of the scheduling service starts a datax process for each synchronous task.
Disclosure of Invention
However, in the prior art, the load of the working node of the scheduling service is too high, and because the synchronous task and the working process of the scheduling service are located on the same working node, when a large number of synchronous tasks are started, the working process is suspended, even the scheduling service is not available, and all offline tasks have faults; on the other hand, resources used by the synchronization task cannot be managed and limited.
To this end, there is a great need for an improved data synchronization method and apparatus, a computer readable storage medium and an electronic device to provide a data synchronization method that reduces the load of the working nodes of the dispatch service.
In this context, embodiments of the present disclosure desirably provide a data synchronization method and apparatus, a computer-readable storage medium, and an electronic device.
According to one aspect of the present disclosure, there is provided a data synchronization method including:
acquiring data to be synchronized, and creating a synchronization task corresponding to the data to be synchronized;
determining a target working node for scheduling service, starting a task submitting process in the target working node, and submitting the synchronous task to a target resource manager through the task submitting process;
and starting the synchronous task in the target resource manager through the target working node to finish the synchronization of the data to be synchronized.
In an exemplary embodiment of the present disclosure, obtaining data to be synchronized, creating a synchronization task corresponding to the data to be synchronized, includes:
acquiring a synchronization parameter and an operation configuration parameter of the data to be synchronized, and taking the synchronization parameter and the operation configuration parameter as main program parameters;
and adding a data synchronization dependency, and generating a synchronization task corresponding to the synchronization data based on the data synchronization dependency and the main program parameter.
In one exemplary embodiment of the present disclosure, when the synchronous task is submitted to a target resource manager by the task submitting process, the method includes:
acquiring a container resource limiting parameter included in the main program parameter, and applying resources for the synchronous task in the target resource manager through the container resource limiting parameter;
and acquiring a target submitting space of the target resource manager, acquiring task resource limiting parameters of the target submitting space, and limiting occupied resources of the synchronous task through the task resource limiting parameters.
In one exemplary embodiment of the present disclosure, after submitting the synchronization task to a target resource manager, the method further comprises:
and controlling the task submitting process to exit.
In an exemplary embodiment of the present disclosure, after the synchronization task is initiated in the target resource manager, the method further comprises:
acquiring a unique identifier corresponding to the synchronous task and distributed by the target resource manager, and storing the unique identifier;
and starting a monitoring thread in the target working node, and monitoring the synchronous task in the target resource manager through the monitoring thread.
In an exemplary embodiment of the present disclosure, listening, by the listening thread, for a synchronization task in the target resource manager includes:
and acquiring the synchronous state of the synchronous task in the target resource manager, and synchronizing the synchronous state into the memory of the target working node.
In an exemplary embodiment of the present disclosure, the method further comprises:
when the target working node fails, selecting a substitute working node;
reading the unique identifier of the synchronous task through the substitute working node, and restoring the unique identifier to the memory of the substitute working node;
and monitoring the synchronous task through the alternative working node.
According to one aspect of the present disclosure, there is provided a data synchronization apparatus including:
the synchronous task creating module is used for acquiring data to be synchronized and creating a synchronous task corresponding to the data to be synchronized;
the task containerization module is used for determining a target working node for scheduling service, starting a task submitting process in the target working node, and submitting the synchronous task to a target resource manager through the task submitting process;
and the data synchronization module is used for starting the synchronization task in the target resource manager through the target working node to complete the synchronization of the data to be synchronized.
According to one aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the data synchronization method described above.
According to one aspect of the present disclosure, there is provided an electronic device including:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the data synchronization method of any one of the above via execution of the executable instructions.
According to the data synchronization method of the embodiment of the disclosure, on one hand, after the data to be synchronized is obtained, a synchronization task corresponding to the data to be synchronized is created, a target working node is determined, a task submitting process is started in the target working node for the synchronization task, the task submitting process is only responsible for submitting the synchronization task to a target resource manager, and the target resource manager is used for completing the synchronization of the data to be synchronized, so that the problem that the load of the target working node responsible for scheduling service is too high due to the fact that the synchronization task is completed by datax in the working node in the related art is solved, and the load of the target working node is reduced; on the other hand, the synchronous task is submitted to the target resource manager, and the resource limitation can be carried out on the synchronous task through the target resource manager, so that the problem that the resource is not managed in the related technology is solved.
Drawings
The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:
fig. 1 schematically illustrates a related art schedule service architecture diagram according to an embodiment of the present disclosure;
FIG. 2 schematically illustrates a flow chart of a data synchronization method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a method of acquiring data to be synchronized, creating a synchronization task corresponding to the data to be synchronized, according to an embodiment of the disclosure;
FIG. 4 schematically illustrates a flow chart of a method of data synchronization when submitting a synchronization task to a target resource manager by a task submitting process, according to an embodiment of the disclosure;
FIG. 5 schematically illustrates a flow chart of a method of data synchronization after a synchronization task is initiated in a target resource manager, in accordance with an embodiment of the present disclosure;
FIG. 6 schematically illustrates a flow chart of a method of data synchronization according to an embodiment of the present disclosure;
FIG. 7 schematically illustrates a system architecture diagram for data synchronization according to an embodiment of the present disclosure;
FIG. 8 schematically illustrates a block diagram of a data synchronization apparatus according to an embodiment of the present disclosure;
FIG. 9 shows a schematic diagram of a computer storage medium according to an embodiment of the disclosure;
fig. 10 schematically illustrates a block diagram of an electronic device in accordance with a disclosed embodiment.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable one skilled in the art to better understand and practice the present disclosure and are not intended to limit the scope of the present disclosure in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
Those skilled in the art will appreciate that embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: complete hardware, complete software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to an embodiment of the present disclosure, a data synchronization method, a data synchronization apparatus, a computer-readable storage medium, and an electronic device are provided.
Any number of elements in the figures are for illustration and not limitation, and any naming is used for distinction only, and not for any limiting sense.
The principles and spirit of the present disclosure are described in detail below with reference to several representative embodiments thereof.
Summary of The Invention
The present inventors have found that offline tasks are typically initiated by a scheduling service, the system architecture of which is shown with reference to fig. 1, the scheduling service being a high availability architecture of a monitoring node 110+a working node 120, and when data needs to be synchronized, the working node 120 further includes a plurality of datax processes 130. When offline data synchronization is performed in the working node 120, the working node 120 is not only responsible for starting a data synchronization task, but also needs to start a datax process 130 to complete the data synchronization task, and the synchronization task is started through a shell script and runs on a physical machine, so that the load of the working node is too high, the resources of the synchronization task cannot be managed and limited, and when the working node fails or needs to be updated and restarted, the synchronization task is directly killed, and the synchronization task is started again at other working nodes, so that the waste of a CPU and memory resources is caused.
In view of the above, the basic idea of the present disclosure is that: according to the data synchronization method and the data synchronization device, firstly, data to be synchronized is obtained, a synchronization task is created according to the data to be synchronized, a target working node for scheduling service is determined, a submitting process for submitting the synchronization task is started in the target working node, the synchronization task is submitted to a target resource manager through the submitting process, the synchronization task is started in the target resource manager, and synchronization of the data to be synchronized is completed.
Having described the basic principles of the present disclosure, various non-limiting embodiments of the present disclosure are specifically described below.
Exemplary method
A data synchronization method according to an exemplary embodiment of the present disclosure is described below with reference to fig. 1.
Referring to fig. 2, the data synchronization method may include the steps of:
s210, acquiring data to be synchronized, and creating a synchronization task corresponding to the data to be synchronized;
s220, determining a target working node for scheduling service, starting a task submitting process in the target working node, and submitting the synchronous task to a target resource manager through the task submitting process;
s230, starting the synchronization task in the target resource manager through the target working node to complete synchronization of the data to be synchronized.
In the data synchronization method of the embodiment of the disclosure, on one hand, after the data to be synchronized is acquired, a synchronization task corresponding to the data to be synchronized is created, a target working node is determined, a task submitting process is started in the target working node for the synchronization task, the task submitting process is only responsible for submitting the synchronization task to a target resource manager, and the target resource manager is used for completing the synchronization of the data to be synchronized, so that the problem that the load of the target working node responsible for scheduling service is too high due to the fact that the synchronization task is completed by the datax in the working node in the related art is solved, and the load of the target working node is reduced; on the other hand, the synchronous task is submitted to the target resource manager, and the resource limitation can be carried out on the synchronous task through the target resource manager, so that the problem that the resource is not managed in the related technology is solved.
In step S210, data to be synchronized is acquired, and a synchronization task corresponding to the data to be synchronized is created.
In the exemplary embodiment of the present disclosure, the data to be synchronized may be processed data in several bins or unprocessed data, which is not specifically limited in this exemplary embodiment. The data to be synchronized needs to be synchronized to a storage service, where the storage service may be MySQL (relational database), HBASE (distributed storage system), or elastic search (distributed full text search engine), and this is not specifically limited in this example embodiment. After the data to be synchronized is obtained, a synchronization task is generated based on the data to be synchronized, and referring to fig. 3, the data to be synchronized is obtained, and the creation of the synchronization task corresponding to the data to be synchronized may include:
s310, acquiring a synchronization parameter and an operation configuration parameter of the data to be synchronized, and taking the synchronization parameter and the operation configuration parameter as main program parameters;
and S320, adding data synchronization dependence, and generating a synchronization task corresponding to the synchronization data based on the data synchronization dependence and the main program parameter.
Hereinafter, step S310 and step S320 will be further explained and explained. When a synchronization task is created, firstly, a synchronization parameter and an operation configuration parameter of data to be synchronized are obtained, and the synchronization parameter and the operation configuration parameter are used as main program parameters, wherein the synchronization parameter can comprise a source data source name, a source indication, a target data source name, a target table name and the like of the data to be synchronized, and the synchronization parameter is not specifically limited in the embodiment; the operation configuration parameters are configuration parameters, such as parallelism, field mapping, etc., when the synchronous task is executed, and are not particularly limited in the present exemplary embodiment; the main program may be a Spark (general cluster computing system) program written using a scale (pure object oriented programming language) or Java (object oriented programming language). Then, in order to realize the synchronization of the synchronization task, data synchronization dependence is added in the main program, and the synchronization task is generated according to the main program parameters and the data synchronization dependence; wherein, depending on representing the connection between classes in the code, i.e. the implementation of one class depends on another class to some extent, after submitting the synchronization task to the target resource manager, a data synchronization tool is needed to achieve synchronization of the data to be synchronized, and the data synchronization dependence can be a datax dependence or other data offline synchronization dependence, and in this example embodiment, the data synchronization dependence is not specifically limited; after adding the datax, when data synchronization is needed, the datax is started to perform data synchronization by calling the client of the datax.
In step S220, a target working node performing scheduling service is determined, a task submitting process is started in the target working node, and the synchronous task is submitted to a target resource manager through the task submitting process.
In the exemplary embodiment of the present disclosure, there are a plurality of working nodes in the scheduling service, one working node may be selected as a target working node from the plurality of working nodes, and a selection method of the working node is not particularly limited in this exemplary embodiment. After determining the target working node, a task submitting process is started in the target working node, the task submitting process can be a Spark submit script process, and the synchronous task is submitted to the target resource manager by starting the Spark submit script process, and the task submitting process is only used for submitting the synchronous task. The target resource manager may be Yarn (Yet another resource negotiator, a general distributed resource management system and scheduling platform) to provide uniform resource management and scheduling for upper layer applications, or K8s (Kubernetes, a distributed architecture solution based on container technology), to support deploying, expanding and managing containerized applications anywhere, and is not specifically limited in this example embodiment.
Referring to fig. 4, when a synchronous task is submitted to a target resource manager through a task submitting process, the data synchronization method includes:
s410, acquiring container resource limiting parameters included in the main program parameters, and applying resources for the synchronous task in the target resource manager through the container resource limiting parameters;
s420, acquiring a target submitting space of the target resource manager, acquiring task resource limiting parameters of the target submitting space, and limiting occupied resources of the synchronous task through the task resource limiting parameters.
Hereinafter, step S410 and step S420 will be further explained and explained. Specifically, in the related art, when data is synchronized, the limitation on the single resource of the synchronization task is realized through the JVM parameter, and the limitation on the total resource is not performed, so that the resource of the synchronization task is not managed. Thus, in this example embodiment, when submitting the synchronization task to the target resource manager, the container resource restriction parameter included in the main program parameter may be acquired, and the resource may be applied for the synchronization task in the target resource manager by using the container resource restriction parameter, that is, the memory resource may be applied for the synchronization task in the target resource manager; the value of the container resource limitation parameter may be 1G or 5G, which is not specifically limited in this exemplary embodiment. In addition, when submitting the synchronous task to the target resource manager, the total resources of the synchronous task may be limited, including: firstly, determining a target submitting space of a target resource manager, setting the target submitting space in the target resource manager, and acquiring task resource limiting parameters of the target submitting space; and then limiting the total resources occupied by the synchronous task according to the task resource limiting parameters. For example, when the target resource manager is Yarn, the synchronous task is submitted to the designated queue of Yarn, and the total resources of the synchronous task are limited according to the minimum resource parameter and the maximum resource parameter of the designated queue, and the minimum resource parameter and the maximum resource parameter are not specifically limited in this example embodiment. When the target resource manager is K8s, submitting the synchronous task to a designated namespace of K8s, and limiting the total resources of the synchronous task by using the resource requirement and the resource limiting parameter used by the designated namespace, wherein the resource requirement and the resource limiting parameter used by the designated namespace are not particularly limited in the present exemplary embodiment.
Further, after submitting the synchronization task to the target resource manager, the data synchronization method further includes:
and controlling the task submitting process to exit.
Specifically, after the task submitting process submits the synchronous task to the target resource manager, the task submitting process is controlled to immediately exit, and the load of the target working node is reduced.
In step S230, the synchronization task is started in the target resource manager through the target working node, so as to complete synchronization of the data to be synchronized.
In the exemplary embodiment of the present disclosure, after the synchronization task is submitted to the target resource manager, the synchronization task needs to be started, and when the synchronization task is started, because the datax dependency is added in the main program parameters of the synchronization task, the target working node may call the datax client to start the synchronization task, so as to complete synchronization of the data to be synchronized.
Referring to fig. 5, after the synchronization task is started in the target resource manager, the data synchronization method further includes:
s510, acquiring a unique identifier corresponding to the synchronous task and distributed by the target resource manager, and storing the unique identifier;
and S520, starting a monitoring thread in the target working node, and monitoring the synchronous task in the target resource manager through the monitoring thread.
Hereinafter, step S510 and step S520 will be further explained and explained. Specifically, after submitting the synchronous task to the target resource manager, the target resource manager allocates a unique identifier for the synchronous task, the target working node obtains the unique identifier, stores the unique identifier in a memory used by the target working node, and after storing the unique identifier in the memory used by the target working node, can also perform persistence storage on the unique identifier and persist the unique identifier to the database. After persisting the unique identification, the target working node also needs to start a listening thread through which to listen for the synchronization task in the target resource manager.
In an exemplary embodiment of the present disclosure, listening, by a listening thread, for a synchronization task in a target resource manager may include:
and acquiring the synchronous state of the synchronous task in the target resource manager, and synchronizing the synchronous state into the memory of the target working node.
Specifically, when the monitoring thread monitors the synchronous task in the target resource manager, the synchronous state of the synchronous task in the target resource manager is obtained, and the obtained synchronous state of the synchronous task is synchronized to the memory of the target working node.
In an exemplary embodiment of the present disclosure, referring to fig. 6, the data synchronization method further includes:
s610, selecting a substitute working node when the target working node fails;
s620, reading the unique identifier of the synchronous task through the substituted working node, and restoring the unique identifier to the memory of the substituted working node;
and S630, monitoring the synchronous task through the substitute working node.
Hereinafter, step S610 to step S630 will be further explained and explained. Specifically, after the monitoring thread synchronizes the synchronization state of the synchronization task in the target resource manager to the memory of the target working node, whether the target working node has a fault or not can be judged according to the synchronization state, and when the target working node has a fault, a working node is selected from a plurality of working nodes of the scheduling service to serve as a substitute working node; after determining the substitute working node, reading the unique identifier of the synchronous task from the persistent memory, namely the database, through the substitute working node, restoring the acquired unique identifier into the memory, and continuing to monitor the synchronous task in the target resource manager through the substitute working node.
In summary, the method provided by the present disclosure may be applied to synchronize data. Referring to the system architecture shown in fig. 7, a monitoring node 710, a target working node 720, and a target resource manager 730 are included. When data synchronization is carried out, firstly, generating a synchronization task according to synchronization parameters and operation configuration parameters of data to be synchronized; then, determining a target working node 720 in the scheduling service, starting a task submitting process in the target working node 720, submitting the synchronous task to a target resource manager 730 through the task submitting process, and starting the synchronous task in the target resource manager 730 through the target working node 720 after submitting the synchronous task to the target resource manager 730 so as to complete the synchronization of the data to be synchronized. On the one hand, after the data to be synchronized is obtained, a synchronous task corresponding to the data to be synchronized is created, a target working node is determined, a task submitting process is started in the target working node for the synchronous task, the task submitting process is only responsible for submitting the synchronous task to a target resource manager, and the target resource manager completes the synchronization of the data to be synchronized, so that the problem that the load of the target working node responsible for scheduling service is too high due to the fact that the synchronous task is completed by the datax in the working node in the related art is solved, and the load of the target working node is reduced; on the other hand, the synchronous task is submitted to a target resource manager, and the synchronous task can be subjected to resource limitation through the target resource manager, so that the problem that resources are not managed in the related technology is solved; on the other hand, when the target working node is in fault or needs to be restarted after upgrading, the synchronous task running in the container is not affected, the synchronous task is not required to be killed, and then the synchronous task is started in other working nodes, so that CPU resources and memory resources are saved.
Exemplary apparatus
Having introduced a data synchronization method of an exemplary embodiment of the present disclosure, next, a data synchronization apparatus of an exemplary embodiment of the present disclosure will be described with reference to fig. 8.
Referring to fig. 8, a data synchronization device 8 of an exemplary embodiment of the present disclosure may include: a sync task creation module 801, a task containerization module 802, and a data synchronization module 803; wherein:
a synchronization task creation module 801, configured to obtain data to be synchronized, and create a synchronization task corresponding to the data to be synchronized;
a task containerization module 802, configured to determine a target working node performing a scheduling service, start a task submitting process in the target working node, and submit the synchronous task to a target resource manager through the task submitting process;
and the data synchronization module 803 is configured to start the synchronization task in the target resource manager through the target working node, and complete synchronization of the data to be synchronized.
According to an exemplary embodiment of the present disclosure, the synchronous task creation module includes:
the parameter configuration module is used for acquiring the synchronous parameters and the operation configuration parameters of the data to be synchronized, and taking the synchronous parameters and the operation configuration parameters as main program parameters;
and the task generating module is used for adding data synchronization dependence and generating a synchronization task corresponding to the synchronization data based on the data synchronization dependence and the main program parameter.
According to an exemplary embodiment of the present disclosure, the task containerization module includes:
the resource application module is used for acquiring container resource limiting parameters included in the main program parameters, and applying resources for the synchronous task in the target resource manager through the container resource limiting parameters;
the resource limiting module is used for acquiring a target submitting space of the target resource manager, acquiring task resource limiting parameters of the target submitting space and limiting occupied resources of the synchronous task through the task resource limiting parameters.
According to an exemplary embodiment of the present disclosure, the task containerization module includes:
and the process control module is used for controlling the task submitting process to exit.
According to an exemplary embodiment of the present disclosure, the data synchronization module includes:
the unique identifier persistence module is used for acquiring the unique identifier corresponding to the synchronous task and distributed by the target resource manager and storing the unique identifier;
and the task monitoring module is used for starting a monitoring thread in the target working node, and monitoring the synchronous task in the target resource manager through the monitoring thread.
According to an exemplary embodiment of the present disclosure, the task listening module includes:
and the state acquisition module is used for acquiring the synchronous state of the synchronous task in the target resource manager and synchronizing the synchronous state into the memory of the target working node.
According to an exemplary embodiment of the present disclosure, the data synchronization apparatus further includes:
the node selection module is used for selecting a substitute working node when the target working node fails;
the node replacement module is used for reading the unique identifier of the synchronous task through the replacement working node and restoring the unique identifier to the memory of the replacement working node;
and the monitoring module is used for monitoring the synchronous task through the substitute working node.
Since each functional module of the data synchronization device in the embodiment of the present disclosure is the same as that in the above-described data synchronization method disclosure embodiment, a detailed description thereof is omitted herein.
Exemplary storage Medium
Having described the data synchronization method and apparatus of the exemplary embodiments of the present disclosure, next, a computer-readable storage medium of the exemplary embodiments of the present disclosure will be described with reference to fig. 9.
Referring to fig. 9, a program product 900 for implementing the above-described method according to an embodiment of the present disclosure is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
Exemplary electronic device
Having described the storage medium of the exemplary embodiments of the present disclosure, next, an electronic device of the exemplary embodiments of the present disclosure will be described with reference to fig. 10.
The electronic device 1000 shown in fig. 10 is merely an example and should not be construed as limiting the functionality and scope of use of the disclosed embodiments.
As shown in fig. 10, the electronic device 1000 is embodied in the form of a general purpose computing device. Components of electronic device 1000 may include, but are not limited to: the at least one processing unit 1010, the at least one memory unit 1020, a bus 1030 connecting the various system components (including the memory unit 1020 and the processing unit 1010), and a display unit 1040.
Wherein the storage unit stores program code that is executable by the processing unit 1010 such that the processing unit 1010 performs steps according to various exemplary embodiments of the present disclosure described in the above section of the present specification. For example, the processing unit 1010 may perform steps S210 to S230 as shown in fig. 1.
The memory unit 1020 may include volatile memory units such as a random access memory unit (RAM) 10201 and/or a cache memory unit 10202, and may further include a read only memory unit (ROM) 10203.
The storage unit 1020 may also include a program/utility 10204 having a set (at least one) of program modules 10205, such program modules 10205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 1030 may include a data bus, an address bus, and a control bus.
The electronic device 1000 can also communicate with one or more external devices 1100 (e.g., keyboard, pointing device, bluetooth device, etc.) through input/output (I/O) interface 1050. The electronic device 1000 also includes a display unit 1040 that is connected to an input/output (I/O) interface 1050 for displaying. Also, electronic device 1000 can communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 1060. As shown, the network adapter 1060 communicates with other modules of the electronic device 1000 over the bus 1030. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with the electronic device 1000, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.
It should be noted that although several modules or sub-modules of the audio playback device and the audio sharing device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Furthermore, although the operations of the methods of the present disclosure are depicted in the drawings in a particular order, this is not required to or suggested that these operations must be performed in this particular order or that all of the illustrated operations must be performed in order to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
While the spirit and principles of the present disclosure have been described with reference to several particular embodiments, it is to be understood that this disclosure is not limited to the particular embodiments disclosed nor does it imply that features in these aspects are not to be combined to benefit from this division, which is done for convenience of description only. The disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims (10)
1. A method of data synchronization, comprising:
acquiring data to be synchronized, and creating a synchronization task corresponding to the data to be synchronized;
determining a target working node for scheduling service, starting a task submitting process in the target working node, and submitting the synchronous task to a target resource manager through the task submitting process;
and starting the synchronous task in the target resource manager through the target working node to finish the synchronization of the data to be synchronized.
2. The method for synchronizing data according to claim 1, wherein obtaining data to be synchronized, creating a synchronization task corresponding to the data to be synchronized, comprises:
acquiring a synchronization parameter and an operation configuration parameter of the data to be synchronized, and taking the synchronization parameter and the operation configuration parameter as main program parameters;
and adding a data synchronization dependency, and generating a synchronization task corresponding to the synchronization data based on the data synchronization dependency and the main program parameter.
3. The data synchronization method according to claim 2, wherein when the synchronization task is submitted to a target resource manager by the task submitting process, the method comprises:
acquiring a container resource limiting parameter included in the main program parameter, and applying resources for the synchronous task in the target resource manager through the container resource limiting parameter;
and acquiring a target submitting space of the target resource manager, acquiring task resource limiting parameters of the target submitting space, and limiting occupied resources of the synchronous task through the task resource limiting parameters.
4. The data synchronization method of claim 1, wherein after submitting the synchronization task to a target resource manager, the method further comprises:
and controlling the task submitting process to exit.
5. The data synchronization method of claim 1, wherein after the synchronization task is initiated in the target resource manager, the method further comprises:
acquiring a unique identifier corresponding to the synchronous task and distributed by the target resource manager, and storing the unique identifier;
and starting a monitoring thread in the target working node, and monitoring the synchronous task in the target resource manager through the monitoring thread.
6. The method of claim 5, wherein listening for the synchronization task in the target resource manager by the listening thread comprises:
and acquiring the synchronous state of the synchronous task in the target resource manager, and synchronizing the synchronous state into the memory of the target working node.
7. The method of data synchronization according to claim 6, wherein the method further comprises:
when the target working node fails, selecting a substitute working node;
reading the unique identifier of the synchronous task through the substitute working node, and restoring the unique identifier to the memory of the substitute working node;
and monitoring the synchronous task through the alternative working node.
8. A data synchronization device, comprising:
the synchronous task creating module is used for acquiring data to be synchronized and creating a synchronous task corresponding to the data to be synchronized;
the task containerization module is used for determining a target working node for scheduling service, starting a task submitting process in the target working node, and submitting the synchronous task to a target resource manager through the task submitting process;
and the data synchronization module is used for starting the synchronization task in the target resource manager through the target working node to complete the synchronization of the data to be synchronized.
9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the data synchronization method of any one of claims 1-7.
10. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the data synchronization method of any one of claims 1-7 via execution of the executable instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311262814.8A CN117112181A (en) | 2023-09-27 | 2023-09-27 | Data synchronization method and device, computer readable storage medium and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311262814.8A CN117112181A (en) | 2023-09-27 | 2023-09-27 | Data synchronization method and device, computer readable storage medium and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117112181A true CN117112181A (en) | 2023-11-24 |
Family
ID=88812923
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311262814.8A Pending CN117112181A (en) | 2023-09-27 | 2023-09-27 | Data synchronization method and device, computer readable storage medium and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117112181A (en) |
-
2023
- 2023-09-27 CN CN202311262814.8A patent/CN117112181A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10735345B2 (en) | Orchestrating computing resources between different computing environments | |
| US8799477B2 (en) | Hypervisor selection for hosting a virtual machine image | |
| US9542226B2 (en) | Operating programs on a computer cluster | |
| CN111897638B (en) | Distributed task scheduling method and system | |
| WO2015057188A1 (en) | Package dependency maps for distributed computing | |
| CN109697070B (en) | Ambari-based cluster management method, device and medium | |
| US9329953B2 (en) | Reducing application downtime during failover | |
| CN113760306A (en) | Method and device for installing software, electronic equipment and storage medium | |
| CN110890987A (en) | Method, device, equipment and system for automatically creating cluster | |
| CN110781159B (en) | Ceph directory file information reading method and device, server and storage medium | |
| CN113204425B (en) | Method, device, electronic equipment and storage medium for process management internal thread | |
| CN110096543B (en) | Data operation method, device, server and medium of application program | |
| CN109491762B (en) | Container state control method and device, storage medium and electronic equipment | |
| CN116089048A (en) | Service processing method and device, storage medium and electronic equipment | |
| CN117112181A (en) | Data synchronization method and device, computer readable storage medium and electronic equipment | |
| CN111352664A (en) | Distributed machine learning task starting method, system, equipment and storage medium | |
| CN114205354A (en) | Event management system, event management method, server, and storage medium | |
| CN113760469A (en) | Distributed computing method and device | |
| CN111385334B (en) | Data distribution method, device, equipment and storage medium | |
| CN111858234A (en) | Task execution method, device, equipment and medium | |
| CN113791819A (en) | Interface document generation method and device, storage medium and electronic equipment | |
| CN113761075A (en) | Method, device, equipment and computer readable medium for switching databases | |
| CN111767113A (en) | Method and device for realizing container eviction | |
| CN114466026B (en) | Update method and device of application program interface, storage medium and computing device | |
| CN109587224B (en) | Data processing method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |