CN111881123B - Data synchronization method and system between distributed systems - Google Patents
Data synchronization method and system between distributed systems Download PDFInfo
- Publication number
- CN111881123B CN111881123B CN202010710041.5A CN202010710041A CN111881123B CN 111881123 B CN111881123 B CN 111881123B CN 202010710041 A CN202010710041 A CN 202010710041A CN 111881123 B CN111881123 B CN 111881123B
- Authority
- CN
- China
- Prior art keywords
- data
- pci
- information data
- management system
- pci device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2365—Ensuring data consistency and integrity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Abstract
The application discloses a data synchronization method between distributed systems, which is applied to a resource management system and comprises the following steps: sending a pulling instruction to a PCI equipment management system, and calling an interface of the PCI equipment management system to pull information data of each PCI equipment; after filtering the pulled information data of each PCI device according to a preset rule by the PCI device management system, receiving feedback data sent by the PCI device management system; and updating the data of the system resource management database according to the feedback data to complete data synchronization. By applying the scheme of the application, the synchronization time consumption can be effectively reduced, and the situation of network resource waste is avoided. The application also provides a data synchronization system between the distributed systems, and the data synchronization system has corresponding technical effects.
Description
Technical Field
The invention relates to the technical field of cloud computing, in particular to a data synchronization method and system among distributed systems.
Background
In cloud computing, data synchronization needs to be performed on a PCI device management system and a resource management system, and when a cloud host is created, a proper PCI device is usually selected according to information stored in the resource management system, and then the PCI device is directly connected to the cloud host, so that the purpose that the cloud host uses the PCI device is achieved.
In the current scheme, a PCI device management system periodically discovers information data of each PCI device on a physical server, and then the PCI device management system compares the data in a database of the system to generate new, deleted and modified data, and updates the data in the local database. After the updating, the system resource management system interface is called again, and the data are sent to the system resource management system to realize the data synchronization. The synchronization period is usually set to 1 minute, which is acceptable for the case of fewer PCI devices, and the period is sufficient.
However, with the increasing number of PCI devices in a large-scale scenario, such a method is too long in synchronization time. If the cycle time is increased, a good real-time effect cannot be achieved, for example, a user can see the PCI device on the PCI device management system, and the creation of the cloud host cannot be performed by using the PCI device in the resource management system, which brings great inconvenience to the user. Meanwhile, in this way, once the interface of the resource management system is in a problem, the PCI device management system still transmits data to the resource management system all the time, which causes a lot of network resource waste and huge pressure on the server.
In summary, how to perform data synchronization between distributed systems effectively reduces synchronization time consumption and avoids the situation of network resource waste is a technical problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide a data synchronization method and a data synchronization system among distributed systems, so that the time consumed by synchronization is effectively reduced and the waste of network resources is avoided when data synchronization is carried out among the distributed systems.
In order to solve the technical problems, the invention provides the following technical scheme:
a data synchronization method between distributed systems is applied to a resource management system and comprises the following steps:
sending a pulling instruction to a PCI equipment management system, and calling an interface of the PCI equipment management system to pull information data of each PCI equipment;
after the PCI equipment management system filters the pulled information data of each PCI equipment according to a preset rule, receiving feedback data sent by the PCI equipment management system;
and updating the data of the system resource management database according to the feedback data to complete data synchronization.
Preferably, the filtering, by the PCI device management system, the pulled information data of each PCI device according to a preset rule includes:
the PCI equipment management system inputs the pulled information data of each PCI equipment into a filter composed of N filtering units; n is a positive integer;
regarding the information data of any one PCI device, when the information data of the PCI device meets the requirements of each filtering unit, the information data of the PCI device is taken as the information data passing through the filter;
and using the information data of each PCI device passing through the filter as feedback data.
Preferably, the method further comprises the following steps:
the PCI equipment management system takes the information data of each PCI equipment which does not pass through the filter as second-class data and stores the second-class data in a cache;
the PCI equipment management system judges whether the difference value between the moment when the feedback data is sent and the moment when the pull instruction is received last time is larger than or equal to a preset threshold value or not;
if not, the second type of data in the cache is sent to the resource management system after the feedback data is sent, so that the resource management system updates the system resource management database according to the second type of data.
Preferably, the filter comprises at least an equipment usage filtering unit;
the device usage rate filtering unit is provided with a device usage rate threshold, and for any piece of information data of the PCI device, when the device usage rate carried in the information data of the PCI device exceeds the device usage rate threshold, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter.
Preferably, the filter at least comprises an equipment type filtering unit, an equipment model filtering unit, a cluster filtering unit where equipment is located and a node resource utilization rate filtering unit;
for any piece of information data of the PCI device, when the device type carried in the information data of the PCI device is not in the type library set by the device type filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for any piece of information data of the PCI device, when the device model carried in the information data of the PCI device is not in the model library set by the device model filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for the information data of any one PCI device, when the grade of the cluster where the device is located, which is carried in the information data of the PCI device, does not accord with the grade setting of the cluster filtering unit where the device is located, the information data of the PCI device is determined to be the information data of the PCI device which does not pass through the filter;
for any piece of information data of the PCI device, when the resource utilization rate of the node where the device is located, which is carried in the information data of the PCI device, exceeds the node resource utilization rate set by the node resource utilization rate filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter.
Preferably, the method further comprises the following steps:
after the PCI equipment management system pulls the information data of each PCI equipment, the PCI equipment management system clears the information data of each PCI equipment with incomplete information before filtering according to a preset rule.
Preferably, the method further comprises the following steps:
after the information data of each PCI device with incomplete information is cleared and before filtering is performed according to preset rules, the PCI device management system clears the information data of each PCI device with the creation time later than the update time, and clears the information data of each PCI device with the device binding time later than the unbinding time.
Preferably, the updating the data of the system resource management database according to the feedback data to complete data synchronization includes:
and based on set operation, updating data of a system resource management database according to the feedback data so as to complete data synchronization.
A data synchronization system between distributed systems is applied to a resource management system and comprises:
the pulling module is used for sending a pulling instruction to the PCI equipment management system and calling an interface of the PCI equipment management system to pull the information data of each PCI equipment;
the feedback data receiving module is used for receiving the feedback data sent by the PCI equipment management system after the PCI equipment management system filters the pulled information data of each PCI equipment according to a preset rule;
and the data synchronization module is used for updating the data of the system resource management database according to the feedback data so as to complete data synchronization.
Preferably, the filtering, by the PCI device management system, the pulled information data of each PCI device according to a preset rule includes:
the PCI equipment management system inputs the pulled information data of each PCI equipment into a filter composed of N filtering units; n is a positive integer;
regarding the information data of any one PCI device, when the information data of the PCI device meets the requirements of each filtering unit, the information data of the PCI device is taken as the information data passing through the filter;
and using the information data of each PCI device passing through the filter as feedback data.
By applying the technical scheme provided by the embodiment of the invention, the traditional push data is changed into the pull data, specifically, the resource management system actively sends the pull instruction, so that the interface of the PCI equipment management system is called to pull the information data of each PCI equipment. In the traditional scheme, the PCI equipment management system needs to update the local database and then pushes the local database to the resource management system, so that the time consumption of the link can be saved. In addition, if the resource management system has a problem, the resource management system needs to actively send the pull instruction in the scheme of the application, so that the PCI device management system does not receive the pull instruction, and a large amount of network resources are not wasted. Furthermore, the PCI equipment management system can filter the pulled information data of each PCI equipment according to a preset rule and then feed the information data back to the resource management system, so that the data with high priority can be ensured to be preferentially synchronized. In summary, according to the scheme of the application, when data synchronization is performed among distributed systems, synchronization time consumption can be effectively reduced, and the situation of network resource waste is avoided.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flow chart of an embodiment of a method for synchronizing data between distributed systems according to the present invention;
fig. 2 is a schematic structural diagram of a data synchronization system between distributed systems according to the present invention.
Detailed Description
The core of the invention is to provide a data synchronization method between distributed systems, which can effectively reduce the time consumption of synchronization and avoid the situation of network resource waste.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a data synchronization method between distributed systems according to the present invention, which is applied in a resource management system, and the data synchronization method between the distributed systems may include the following steps:
step S101: and sending a pulling instruction to the PCI equipment management system, and calling an interface of the PCI equipment management system to pull the information data of each PCI equipment.
The resource management system may generally send a pull instruction to the PCI device management system according to a preset pull period, where the pull period may be set and adjusted according to actual needs, for example, may be set to 1 minute.
After a pull instruction is sent to the PCI equipment management system each time, an interface of the PCI equipment management system can be called to pull the information data of each PCI equipment. It should be noted that, in order to ensure that only one service in the distributed service in the resource management system pulls data to the PCI device resource management system at the same time, each time a pull instruction is sent to the PCI device management system to pull the information data of each PCI device, a distributed lock is usually created.
In addition, when the interface is called, the interface of the PCI equipment management system can be called to pull the information data of all needed PCI equipment at one time, and frequent calling of the interface is avoided.
Step S102: and after filtering the pulled information data of each PCI device according to a preset rule by the PCI device management system, receiving feedback data sent by the PCI device management system.
In the solution of the application, it is considered that the PCI device management system may directly feed back the pulled information data of each PCI device to the resource management system after receiving the pull instruction, but it is further considered that the resource management system generally creates the cloud host by using the information data of each PCI device stored in the resource management database, so when the PCI device management system and the resource management system perform data synchronization, the information data of the PCI devices that are used with higher probability may be preferentially synchronized, and then the remaining low-priority data may be synchronized, so that in an occasion where the network is busy, the information data of the PCI devices that are used with high probability may be preferentially synchronized.
That is, when filtering is performed according to a preset rule, the preset rule may be adaptively adjusted according to actual needs, but after filtering is performed according to the preset rule, data that passes filtering should be data that has a higher probability of being used than data that does not pass filtering. The PCI device management system filters the pulled information data of each PCI device according to a preset rule, and uses the information data of the PCI device that is used with a higher probability as the feedback data in step S102, and accordingly, the remaining data may be placed in the cache first, and then sent after the feedback data is sent, or sent during the idle time of the pulling cycle.
In an embodiment of the present invention, the method may further include:
the PCI equipment management system takes the information data of each PCI equipment which does not pass through the filter as second-class data and stores the second-class data in a cache;
the PCI equipment management system judges whether the difference value between the moment when the feedback data is sent and the moment when the pull instruction is received last time is larger than or equal to a preset threshold value or not;
if not, the second type of data in the cache is sent to the resource management system after the feedback data is sent, so that the resource management system updates the data of the system resource management database according to the second type of data.
In this embodiment, the PCI device management system stores the information data of each PCI device that has passed through the filter as feedback data, and the remaining information data that has not passed through the filter as second-type data in the cache. That is, in this embodiment, the feedback data is preferentially fed back to the resource management system, and whether the second type of data is fed back to the resource management system in the pull cycle depends on the remaining time of the pull cycle. If the difference value between the moment when the feedback data is sent and the moment when the pulling instruction is received last time is smaller than the preset threshold value, the PCI equipment management system indicates that the residual time of the pulling period is enough, and the second-class data in the cache can be sent to the resource management system.
The implementation method can enable the feedback data with higher priority to be preferentially synchronized, and for the second type of data with lower priority, the feedback data can be synchronized into the resource management system immediately after being sent under the condition that the current pulling cycle time is enough. In practical application, as long as the short-term and sudden situation that the network is particularly busy does not occur, the feedback data and the second type of data can be synchronized into the resource management system in sequence.
In practical application, the filtering operation can be completed through the filter consisting of N filtering units, and the arrangement of the scheme is favorable for conveniently adjusting the filtering rules by adding, modifying and deleting the filtering units.
That is, in a specific embodiment of the present invention, the filtering, by the PCI device management system described in step S102, the information data of each pulled PCI device according to a preset rule may specifically include:
the PCI equipment management system inputs the information data of each pulled PCI equipment into a filter consisting of N filtering units; n is a positive integer;
regarding the information data of any one PCI device, when the information data of the PCI device meets the requirements of each filtering unit, the information data of the PCI device is taken as the information data passing through the filter;
the information data of each PCI device having passed through the filter is used as feedback data.
Of course, the number N of the filter units and the specific filter rule of each filter unit can be set and adjusted according to actual needs.
For example, in one embodiment of the present invention, the filter comprises at least an equipment usage filtering unit;
the device utilization rate filtering unit is provided with a device utilization rate threshold, and for any piece of information data of the PCI device, when the device utilization rate carried in the information data of the PCI device exceeds the device utilization rate threshold, the information data of the PCI device is determined to be the information data of the PCI device which does not pass through the filter.
In this embodiment, in consideration of that, when the cloud host is created, if the usage rate of one PCI device is higher, the probability of being selected for creating the cloud host is lower, therefore, the device usage rate filtering unit may be set with a device usage rate threshold value, for example, set to 40%, for example, the PCI device 1 is GPU1, the device usage rate is 30%, the PCI device 2 is GPU2, the device usage rate is 70%, the PCI device 3 is FPGA1, the device usage rate is 10%, the PCI device 4 is FPGA2, and the device usage rate is 35%, then the PCI device 2 fails in the device usage rate filtering unit, that is, the information data of the PCI device 2 is determined to be the information data of the PCI device that fails in the filter.
In a specific embodiment of the present invention, the filter at least includes an equipment type filtering unit, an equipment model filtering unit, a cluster filtering unit where the equipment is located, and a node resource utilization rate filtering unit;
for any piece of information data of the PCI device, when the device type carried in the information data of the PCI device is not in the type library set by the device type filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for any piece of information data of the PCI device, when the device model carried in the information data of the PCI device is not in the model library set by the device model filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for the information data of any one PCI device, when the grade of the cluster where the device is located carried in the information data of the PCI device does not accord with the grade setting of the cluster filtering unit where the device is located, the information data of the PCI device is determined to be the information data of the PCI device which does not pass through the filter;
for any piece of information data of the PCI device, when the resource utilization rate of the node where the device is located, which is carried in the information data of the PCI device, exceeds the node resource utilization rate set by the node resource utilization rate filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter.
In this embodiment, the filter at least includes an equipment type filtering unit, an equipment model filtering unit, a cluster filtering unit where the equipment is located, and a node resource utilization rate filtering unit. The type library of device type filter unit settings may include, for example: a domestic GPU, an england GPU, an inpur FPGA, an inpur SSD, or the like, when the device type carried in the information data of a certain PCI device is not in the type library set by the device type filtering unit, the information data of the PCI device is determined as the information data of the PCI device that does not pass through the filter.
Correspondingly, the model library set by the equipment model filtering unit can be preset and adjusted according to actual needs.
The class of the cluster where the device is located generally includes 4 classes, which respectively correspond to the high-performance cluster, the medium-performance cluster, the low-performance cluster, and the normal cluster, although there may be other classes in different occasions. Generally, the level of the high-performance cluster is taken as a high priority level, that is, the level of the cluster filter unit where the device is located is set to pass filtering in a level that the cluster where the device is located carried in the information data of the PCI device is taken as the high-performance cluster, and the remaining levels are not passed filtering. Of course, other configurations are possible, such as high performance clusters, and neutral performance clusters capable of filtering.
The resource usage rate of the node where the device is located, which is carried in the information data of the PCI device, is generally composed of a CPU usage rate of the node where the device is located, a memory usage rate of the node where the device is located, and a disk usage rate of the node where the device is located, and the higher the resource usage rate is, the lower the probability of creating the cloud host at the node is, and therefore, when the resource usage rate of the node where the device is located, which is carried in the information data of the PCI device, exceeds the node resource usage rate set by the node resource usage rate filtering unit, the information data of the PCI device is determined to be information data of the PCI device that does not pass through the filter.
It should be emphasized that, the fact that the resource utilization rate of the node where the device is located carried in the information data of the PCI device exceeds the node resource utilization rate set by the node resource utilization rate filtering unit means that the CPU utilization rate of the node where the device is located exceeds a set CPU utilization rate threshold, the memory utilization rate of the node where the device is located exceeds a set memory utilization rate threshold, and the disk utilization rate of the node where the device is located exceeds a set disk utilization rate threshold, where any one or more of the three items are true.
Step S103: and updating the data of the system resource management database according to the feedback data to complete data synchronization.
In the scheme of the application, after the resource management system sends the pull instruction to the PCI device management system each time, since the PCI device management system filters the information data of each pulled PCI device according to the preset rule and then feeds back the information data to the resource management system, that is, the data is divided into 2 parts, i.e., the feedback data with higher priority and the second-class data with lower priority and without direct feedback. Therefore, when the resource management system updates the system resource management database according to the feedback data, the resource management system can be set to perform three items of data insertion, data deletion and data modification, or set to perform only two items of data insertion and data modification, and when the second type of data is received in the pull cycle, the resource management system performs three items of data insertion, data deletion and data modification by combining the second type of data and the feedback data.
Specifically, when the data of the system resource management database is updated according to the feedback data, if the data is set to be inserted, deleted, and modified, the original data of the system resource management database is replaced with the feedback data. And only two items of data insertion and data modification are set, so that the three operations of data insertion, data deletion and data modification are performed on the system resource management database in combination with the feedback data when the second type of data is received in the pull cycle, considering that part of the data is filtered, that is, the feedback data originally does not contain the information data of each PCI device.
Further, in an embodiment of the present invention, the updating the data of the system resource management database according to the feedback data to complete data synchronization may specifically include:
and based on the set operation, updating the data of the system resource management database according to the feedback data to complete data synchronization.
In the implementation mode, the time consumption is shorter compared with dictionary operation on the basis of set operation, and the method is beneficial to improving the synchronization efficiency of the scheme.
In the set operation, the IDs of the PCI devices may be traversed to generate a set, for example, referred to as set a, while the IDs of the PCI devices may be traversed to generate a set, for example, referred to as set B, while the data in the system resource management database is traversed.
Performing intersection operation on the sets a and B to obtain a result that the PCI device is common in the two sets, and comparing specific contents carried in information data of the PCI device, the method generally includes: and if the information of the PCI equipment such as the ID, the name, the manufacturer, the product number, the equipment type, the equipment bus address, the equipment characteristic value, the used state, the equipment creating time, the updating time, the binding time, the unbinding time, the attribute creating time, the attribute updating time and the like is inconsistent, the corresponding content of the system resource management database can be updated according to the feedback data.
Correspondingly, the difference between the set A and the set B can be obtained to obtain the newly added PCI equipment, and the newly added PCI equipment can be added into the list to be inserted, namely, each PCI equipment object to be inserted is obtained from the feedback data and is inserted into the system resource management database.
And (4) subtracting the set B from the set A, obtaining the PCI equipment object to be deleted, and deleting the corresponding data in the system resource management database. Of course, as described above, when the data synchronization is completed by updating the system resource management database according to the feedback data, the data insertion, data deletion, and data modification may be performed, or the data deletion may not be performed first, and after the pull cycle acquires the second type of data, the deletion operation is performed on the system resource management database by combining the second type of data and the feedback data.
In performing step S103, atomicity of the database operation data may be guaranteed using the database transaction.
In addition, it should be emphasized that, in the conventional scheme, the PCI device management system needs to update the local database and then push data to the resource management system, which can save time consumption of this link. In practical application, a separate process may be set for the PCI device management system, and the PCI device management system updates the local database based on the process, that is, the data synchronization process between the PCI device management system and the distributed system of the present application is not affected.
In an embodiment of the present invention, the method may further include:
after the PCI equipment management system pulls the information data of each PCI equipment, the PCI equipment management system clears the information data of each PCI equipment with incomplete information before filtering according to a preset rule.
It should be emphasized that, in the foregoing embodiment, the PCI device management system filters the information data of each pulled PCI device, and what is described is a process of determining the priority of the information data of different types of PCI devices, that is, the information data with higher priority is preferentially sent as feedback data, and when the remaining time of the pull cycle is sufficient, the second type of data that has not passed the filtering is sent. In this embodiment, the data is purged, which refers to the purging of unreasonable, unproductive data.
Specifically, when information of information data of one PCI device is incomplete, for example, if the content of one attribute is null, the information data of the PCI device has no value of synchronization, and therefore, the PCI device management system will perform data cleanup before performing filtering.
Further, in an embodiment of the present invention, the method may further include:
after the information data of each PCI device with incomplete information is cleared and before filtering is performed according to preset rules, the PCI device management system clears the information data of each PCI device with the creation time later than the update time, and clears the information data of each PCI device with the device binding time later than the unbinding time.
In this embodiment, it is not reasonable to consider that the creation time is later than the update time or the device binding time is later than the unbinding time in addition to the incomplete information, and therefore, the data is also erased in either case. The method and the device for clearing the unreasonable data are beneficial to guaranteeing that the data synchronized to the resource management system are reasonable data.
By applying the technical scheme provided by the embodiment of the invention, the traditional push data is changed into the pull data, specifically, the resource management system actively sends the pull instruction, so that the interface of the PCI equipment management system is called to pull the information data of each PCI equipment. In the traditional scheme, the PCI equipment management system needs to update the local database and then pushes the local database to the resource management system, so that the time consumption of the link can be saved. In addition, if the resource management system has a problem, the resource management system needs to actively send the pull instruction in the scheme of the application, so that the PCI device management system does not receive the pull instruction, and a large amount of network resources are not wasted. Furthermore, the PCI equipment management system can filter the pulled information data of each PCI equipment according to a preset rule and then feed the information data back to the resource management system, so that the data with high priority can be ensured to be preferentially synchronized. In conclusion, according to the scheme of the application, when data synchronization is performed among distributed systems, the time consumed by synchronization can be effectively reduced, and the situation of network resource waste is avoided.
Corresponding to the above method embodiments, the embodiments of the present invention further provide a data synchronization system between distributed systems, which can be referred to in correspondence with the above.
Referring to fig. 2, a schematic structural diagram of a data synchronization system between distributed systems in the present invention is applied to a resource management system, and includes:
a pulling module 201, configured to send a pulling instruction to the PCI device management system, and call an interface of the PCI device management system to pull information data of each PCI device;
the feedback data receiving module 202 is configured to receive feedback data sent by the PCI device management system after the PCI device management system filters the pulled information data of each PCI device according to a preset rule;
and the data synchronization module 203 is used for performing data update on the system resource management database according to the feedback data to complete data synchronization.
In a specific embodiment of the present invention, the filtering, by the PCI device management system, the pulled information data of each PCI device according to a preset rule includes:
the PCI equipment management system inputs the information data of each pulled PCI equipment into a filter consisting of N filtering units; n is a positive integer;
regarding the information data of any one PCI device, when the information data of the PCI device meets the requirements of each filtering unit, the information data of the PCI device is taken as the information data passing through the filter;
using the information data of each PCI device passing through the filter as feedback data;
in a specific embodiment of the present invention, the PCI device management system further includes:
the second-class data acquisition module is used for taking the information data of each PCI device which does not pass through the filter as second-class data and storing the second-class data in a cache;
the judging module is used for judging whether the difference value between the moment when the feedback data is sent and the moment when the pull instruction is received last time is larger than or equal to a preset threshold value or not; if not, triggering a second type data sending module;
and the second type data sending module is used for sending the second type data in the cache to the resource management system after the feedback data is sent, so that the resource management system updates the data of the system resource management database according to the second type data.
In one embodiment of the invention, the filter comprises at least an equipment usage filtering unit;
the device utilization rate filtering unit is provided with a device utilization rate threshold, and for any piece of information data of the PCI device, when the device utilization rate carried in the information data of the PCI device exceeds the device utilization rate threshold, the information data of the PCI device is determined to be the information data of the PCI device which does not pass through the filter.
In a specific embodiment of the present invention, the filter at least includes an equipment type filtering unit, an equipment model filtering unit, a cluster filtering unit where the equipment is located, and a node resource utilization rate filtering unit;
for any piece of information data of the PCI device, when the device type carried in the information data of the PCI device is not in the type library set by the device type filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for any piece of information data of the PCI device, when the device model carried in the information data of the PCI device is not in the model library set by the device model filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for the information data of any one PCI device, when the grade of the cluster where the device is located carried in the information data of the PCI device does not accord with the grade setting of the cluster filtering unit where the device is located, the information data of the PCI device is determined to be the information data of the PCI device which does not pass through the filter;
for any piece of information data of the PCI device, when the resource utilization rate of the node where the device is located, which is carried in the information data of the PCI device, exceeds the node resource utilization rate set by the node resource utilization rate filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter.
In a specific embodiment of the present invention, the PCI device management system further includes:
and the data clearing module is used for clearing the information data of each PCI device with incomplete information before filtering according to a preset rule after the PCI device management system pulls the information data of each PCI device.
In an embodiment of the present invention, the data clearing module is further configured to:
after information data of each PCI device with incomplete information is cleared, before filtering is carried out according to a preset rule, the information data of each PCI device with the creation time later than the updating time is cleared, and the information data of each PCI device with the device binding time later than the unbinding time is cleared.
In an embodiment of the present invention, the data synchronization module 203 is specifically configured to:
and based on the set operation, updating the data of the system resource management database according to the feedback data to complete data synchronization.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The principle and the implementation of the present invention are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (7)
1. A data synchronization method between distributed systems is applied to a resource management system, and comprises the following steps:
sending a pulling instruction to a PCI equipment management system, and calling an interface of the PCI equipment management system to pull information data of each PCI equipment;
after the PCI equipment management system filters the pulled information data of each PCI equipment according to a preset rule, receiving feedback data sent by the PCI equipment management system;
updating the data of the system resource management database according to the feedback data to complete data synchronization;
the PCI equipment management system filters the pulled information data of each PCI equipment according to a preset rule, and comprises the following steps:
the PCI equipment management system inputs the pulled information data of each PCI equipment into a filter composed of N filtering units; n is a positive integer;
regarding the information data of any one PCI device, when the information data of the PCI device meets the requirements of each filtering unit, the information data of the PCI device is taken as the information data passing through the filter;
using the information data of each PCI device passing through the filter as feedback data;
further comprising:
the PCI equipment management system takes the information data of each PCI equipment which does not pass through the filter as second-class data and stores the second-class data in a cache;
the PCI equipment management system judges whether the difference value between the moment when the feedback data is sent and the moment when the pull instruction is received last time is larger than or equal to a preset threshold value or not;
if not, the second type of data in the cache is sent to the resource management system after the feedback data is sent, so that the resource management system updates the system resource management database according to the second type of data.
2. The method of data synchronization between distributed systems of claim 1, wherein the filter comprises at least a device usage filtering unit;
the device usage rate filtering unit is provided with a device usage rate threshold, and for any piece of information data of the PCI device, when the device usage rate carried in the information data of the PCI device exceeds the device usage rate threshold, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter.
3. The data synchronization method among the distributed systems according to claim 1, wherein the filter at least includes a device type filtering unit, a device model filtering unit, a cluster filtering unit where the device is located, and a node resource utilization rate filtering unit;
for any piece of information data of the PCI device, when the device type carried in the information data of the PCI device is not in the type library set by the device type filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for any piece of information data of the PCI device, when the device model carried in the information data of the PCI device is not in the model library set by the device model filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter;
for the information data of any one PCI device, when the grade of the cluster where the device is located, which is carried in the information data of the PCI device, does not accord with the grade setting of the cluster filtering unit where the device is located, the information data of the PCI device is determined to be the information data of the PCI device which does not pass through the filter;
for any piece of information data of the PCI device, when the resource utilization rate of the node where the device is located, which is carried in the information data of the PCI device, exceeds the node resource utilization rate set by the node resource utilization rate filtering unit, the information data of the PCI device is determined to be the information data of the PCI device that does not pass through the filter.
4. The method for data synchronization between distributed systems according to claim 1, further comprising:
after the PCI equipment management system pulls the information data of each PCI equipment, the PCI equipment management system clears the information data of each PCI equipment with incomplete information before filtering according to a preset rule.
5. The method of data synchronization between distributed systems of claim 4, further comprising:
after the information data of each PCI device with incomplete information is cleared and before filtering is performed according to preset rules, the PCI device management system clears the information data of each PCI device with the creation time later than the update time, and clears the information data of each PCI device with the device binding time later than the unbinding time.
6. The method for data synchronization between distributed systems according to claim 1, wherein the performing data update on the system resource management database according to the feedback data to complete data synchronization comprises:
and based on set operation, updating data of a system resource management database according to the feedback data so as to complete data synchronization.
7. A data synchronization system between distributed systems is applied to a resource management system, and comprises:
the pulling module is used for sending a pulling instruction to the PCI equipment management system and calling an interface of the PCI equipment management system to pull the information data of each PCI equipment;
the feedback data receiving module is used for receiving the feedback data sent by the PCI equipment management system after the PCI equipment management system filters the pulled information data of each PCI equipment according to a preset rule;
the data synchronization module is used for updating the data of the system resource management database according to the feedback data so as to complete data synchronization;
the PCI equipment management system filters the pulled information data of each PCI equipment according to a preset rule, and comprises the following steps:
the PCI equipment management system inputs the pulled information data of each PCI equipment into a filter composed of N filtering units; n is a positive integer;
regarding the information data of any one PCI device, when the information data of the PCI device meets the requirements of each filtering unit, the information data of the PCI device is taken as the information data passing through the filter;
using the information data of each PCI device passing through the filter as feedback data;
further comprising:
the second-class data acquisition module is used for taking the information data of each PCI device which does not pass through the filter as second-class data and storing the second-class data in a cache;
the judging module is used for judging whether the difference value between the moment when the feedback data is sent and the moment when the pull instruction is received last time is larger than or equal to a preset threshold value or not; if not, triggering a second type data sending module,
and the second type data sending module is used for sending the second type data in the cache to the resource management system after the feedback data is sent, so that the resource management system updates the data of the system resource management database according to the second type data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010710041.5A CN111881123B (en) | 2020-07-22 | 2020-07-22 | Data synchronization method and system between distributed systems |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010710041.5A CN111881123B (en) | 2020-07-22 | 2020-07-22 | Data synchronization method and system between distributed systems |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111881123A CN111881123A (en) | 2020-11-03 |
| CN111881123B true CN111881123B (en) | 2022-04-22 |
Family
ID=73155993
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010710041.5A Active CN111881123B (en) | 2020-07-22 | 2020-07-22 | Data synchronization method and system between distributed systems |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111881123B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255958A (en) * | 2011-06-27 | 2011-11-23 | 北京星网锐捷网络技术有限公司 | Data synchronization method and system thereof |
| CN106126347A (en) * | 2016-08-17 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of method effectively realizing mainframe cluster scheduling of resource under cloud platform |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9842152B2 (en) * | 2014-02-19 | 2017-12-12 | Snowflake Computing, Inc. | Transparent discovery of semi-structured data schema |
-
2020
- 2020-07-22 CN CN202010710041.5A patent/CN111881123B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102255958A (en) * | 2011-06-27 | 2011-11-23 | 北京星网锐捷网络技术有限公司 | Data synchronization method and system thereof |
| CN106126347A (en) * | 2016-08-17 | 2016-11-16 | 浪潮电子信息产业股份有限公司 | A kind of method effectively realizing mainframe cluster scheduling of resource under cloud platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111881123A (en) | 2020-11-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109493076B (en) | Kafka message unique consumption method, system, server and storage medium | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| CN107045422A (en) | Distributed storage method and equipment | |
| CN106445644A (en) | Distributed transaction processing method and device based on improved one-phase commit | |
| CN104601366B (en) | It is a kind of control, service node configuration service method and device | |
| CN103581332A (en) | HDFS framework and pressure decomposition method for NameNodes in HDFS framework | |
| WO2019076236A1 (en) | Data synchronization method and device, super controller, domain controller, and storage medium | |
| CN111400041A (en) | Server configuration file management method and device and computer readable storage medium | |
| CN100518132C (en) | Distributing traffic information storage file system | |
| CN109032587A (en) | A kind of data stream management frame, method, terminal device and storage medium | |
| CN112416969A (en) | Parallel task scheduling system in distributed database | |
| CN111694517A (en) | Distributed data migration method and system and electronic equipment | |
| CN114090179A (en) | Migration method and device of stateful service and server | |
| CN109189431A (en) | A kind of client side upgrading method, device, equipment and readable storage medium storing program for executing | |
| CN111881123B (en) | Data synchronization method and system between distributed systems | |
| CN112214288A (en) | Pod scheduling method, device, equipment and medium based on Kubernetes cluster | |
| CN112965839A (en) | Message transmission method, device, equipment and storage medium | |
| CN115964176B (en) | Cloud computing cluster scheduling method, electronic equipment and storage medium | |
| CN111897643A (en) | Thread pool configuration system, method, device and storage medium | |
| CN113901141B (en) | Distributed data synchronization method and system | |
| CN108984105B (en) | Method and device for distributing replication tasks in network storage device | |
| CN109491768A (en) | A kind of distributed transaction method for dynamically processing and system based on subtransaction stream | |
| CN107342944B (en) | A kind of distribution of ID mark and persistence method and system | |
| CN111176848B (en) | Cluster task processing method, device, equipment and storage medium | |
| CN114328285A (en) | Heap memory allocation management method and device of embedded operating system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |