CN117271666A - Data synchronization method, distributed system, storage medium and electronic device - Google Patents

Abstract

The present disclosure relates to the field of data processing technology, and relates to a data synchronization method, a distributed system, a computer-readable storage medium, and an electronic device. The data synchronization method is applied to a subsystem in a distributed system, the subsystem comprises a plurality of sub-processes and a shared memory among the plurality of sub-processes, and the data synchronization method comprises the following steps: responding to the data synchronization request, and determining a data processing strategy by the first subprocess according to the request type; the first subprocess performs data synchronization on the shared memory according to a data processing strategy; the first subprocess monitors the read state of the old data corresponding to the data synchronization based on the timing mechanism so as to perform invalidation processing on the old data according to the read state. The data synchronization method disclosed by the invention avoids resource competition among the multi-core systems, realizes the data consistency of each core in the multi-core system, and ensures the performance of the distributed system.

Description

Data synchronization method, distributed system, storage medium and electronic device

Technical Field

The present disclosure relates to the field of data processing technology, and more particularly, to a data synchronization method, a distributed system, a computer-readable storage medium, and an electronic device.

Background

Along with the trend of the distributed micro-service architecture, in practical application, due to complex network conditions and dependency relationships among multiple subsystems in the distributed system, inconsistent data among multiple applications of each subsystem can cause abnormal system functions and reduce system performance. Therefore, achieving data synchronization in a distributed system becomes particularly important.

However, when the related art performs data synchronization in the distributed system, there is a problem that data synchronization of each core in the multi-core system is inconsistent, which affects performance of the multi-core system to a certain extent.

It should be noted that the information of the present invention in the above background section is only for enhancing understanding of the background of the present disclosure, and thus may include information that does not form the prior art that is already known to those of ordinary skill in the art.

Disclosure of Invention

The disclosure aims to provide a data synchronization method, a distributed system, a computer readable storage medium and electronic equipment, so as to avoid resource competition among multi-core systems, realize data consistency of the multi-core systems and guarantee stability of the distributed system.

Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.

According to one aspect of the present disclosure, there is provided a data synchronization method applied to a subsystem in a distributed system, the subsystem including a plurality of sub-processes and a shared memory between the plurality of sub-processes, the data synchronization method including: responding to the data synchronization request, and determining a data processing strategy by the first subprocess according to the request type; the first subprocess performs data synchronization on the shared memory according to the data processing strategy; the first sub-process monitors the read state of the old data corresponding to the data synchronization based on a timing mechanism, so as to perform failure processing on the old data according to the read state.

In an exemplary embodiment of the present disclosure, the first sub-process performs data synchronization on the shared memory according to the data processing policy, including: after the first subprocess acquires the security lock of the shared memory, carrying out data synchronization on the shared memory according to the data processing strategy; monitoring a read lock count of old data corresponding to the data synchronization based on a timing mechanism, wherein the read lock count is determined according to the number of second sub-processes which have read the old data; and if the read lock count is zero, performing invalidation processing on the old data.

In an exemplary embodiment of the present disclosure, if the request type is modified data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, including: the first subprocess modifies old data in the shared memory to obtain target data, wherein the target data is used for subsequent data reading; and adding failure time to the old data, and performing failure processing on the old data according to the read state when the failure time expires.

In an exemplary embodiment of the present disclosure, the method further comprises: configuring a first identifier for the target data and a second identifier for the old data, wherein the first identifier has a higher priority than the second identifier; and before the target data and the expiration time are obtained, preferentially reading the target data with the first identifier.

In an exemplary embodiment of the present disclosure, if the request type is delete data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, including: and adding an expiration time to the old data, and deleting the old data according to the read state when the expiration time expires.

In one exemplary embodiment of the present disclosure, the determining the time to failure includes: based on the read state of the old data, acquiring the number of sub-processes of the read old data and the service types corresponding to the sub-processes; and determining the failure time according to the number of the sub-processes and the service type.

In an exemplary embodiment of the present disclosure, if the request type is add data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, including: and adding the data to be added into the shared memory.

According to one aspect of the present disclosure, there is provided a distributed system comprising a data processing module and a plurality of subsystems; the data processing module is configured to process the data to be synchronized and update a database according to the obtained standard data; the data processing module is further configured to sequentially send a data synchronization request to the subsystem; any of the subsystems is configured to perform a data synchronization process according to the data synchronization method of any of the above exemplary embodiments to obtain subsystem data consistent with the database.

According to one aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of any one of the above.

According to one aspect of the present disclosure, there is provided an electronic device including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method of any of the above via execution of the executable instructions.

In a subsystem in a distributed system, a data synchronization request is received and responded through a first sub-process, a data processing strategy is determined according to a request type, and data synchronization is performed on a shared memory according to the data processing strategy. The first subprocess monitors the read state of the old data corresponding to the data synchronization based on the timing mechanism, and performs failure processing on the old data according to the read state. On one hand, the first subprocess performs data synchronization on the shared memory according to a data processing strategy, so that data in the subsystem is consistent with data of the distributed system, the problem of abnormal system functions or performance reduction caused by inconsistent data of each system is avoided, and the advantages of the distributed system are fully exerted; on the other hand, when the first subprocess performs data synchronization on the shared memory, the read state of the old data corresponding to the data synchronization is monitored based on a timing mechanism, and as other subprocesses which have read the old data may exist during the data synchronization, in order to prevent the processing progress and the result of the other subprocesses from being influenced, the first subprocess performs failure processing on the old data according to the read state of the old data, so that the problem of system function abnormality or performance caused by early failure of the old data is avoided, namely, the performance of the distributed system is ensured while the data synchronization is realized.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which:

FIG. 1 illustrates a process model schematic of an Nginx subsystem according to an exemplary embodiment of the present disclosure;

FIG. 2 illustrates a flow chart of a data synchronization method according to an exemplary embodiment of the present disclosure;

FIG. 3 illustrates a flowchart of an implementation of a first sub-process to data synchronize a shared memory, according to an example embodiment of the present disclosure;

FIG. 4 illustrates a flowchart of one implementation of determining a time to failure according to an exemplary embodiment of the present disclosure;

FIG. 5 illustrates a complete flow chart of a data synchronization according to an exemplary embodiment of the present disclosure;

FIG. 6 illustrates a schematic diagram of a distributed system according to an exemplary embodiment of the present disclosure;

FIG. 7 illustrates a schematic diagram of another distributed system according to an exemplary embodiment of the present disclosure;

Fig. 8 shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure.

In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

Detailed Description

Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus detailed descriptions thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the disclosure. One skilled in the relevant art will recognize, however, that the disclosed aspects may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known structures, methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the disclosure.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, or in one or more software-hardened modules, or in different networks and/or processor devices and/or microcontroller devices.

In applications of distributed systems, data between multiple applications of each system may be inconsistent due to complex network conditions and dependencies between subsystems, which may cause system dysfunction or reduce system performance.

In the data synchronization, if the subsystem is a single core or single process system, the data of the subsystem is consistent with the data of the main system, and if the subsystem is a multi-process (or multi-core) subsystem, the data synchronization request will be received and processed by a certain sub-process, and other sub-processes which do not receive the data synchronization request will continue to use old data, which causes the problem of inconsistent data among a plurality of sub-processes in the subsystem.

By way of example, fig. 1 shows a schematic process model diagram of an nmginx subsystem according to an exemplary embodiment of the present disclosure, where the nmginx subsystem includes a master core and a plurality of worker cores, and data consistency needs to be maintained between the plurality of worker cores. However, only one worker core currently receives and processes the data synchronization request, and the user data of other worker cores is still old data.

Based on this, in an exemplary embodiment of the present disclosure, a data synchronization method is first provided. The method is applicable to any subsystem in a distributed system that includes a plurality of sub-processes and a shared memory between the plurality of sub-processes. Referring to fig. 2, there is shown a flowchart of a data synchronization method according to an exemplary embodiment of the present disclosure, including steps S210 and S220:

step S210: responding to the data synchronization request, and determining a data processing strategy by the first subprocess according to the request type;

step S220: the first subprocess performs data synchronization on the shared memory according to the data processing strategy; the first sub-process monitors the read state of the old data corresponding to the data synchronization based on a timing mechanism, so as to perform failure processing on the old data according to the read state.

Based on the data synchronization method in the exemplary embodiment of the disclosure, on one hand, the first sub-process performs data synchronization on the shared memory according to the data processing strategy, so that data in the sub-system is consistent with data of the distributed system, the problem of abnormal system functions or performance reduction caused by inconsistent data of each system is avoided, and the advantages of the distributed system are fully exerted; on the other hand, when the first subprocess performs data synchronization on the shared memory, the read state of the old data corresponding to the data synchronization is monitored based on a timing mechanism, and as other subprocesses which have read the old data may exist during the data synchronization, in order to prevent the processing progress and the result of the other subprocesses from being influenced, the first subprocess performs failure processing on the old data according to the read state of the old data, so that the problem of abnormal system functions or performance caused by early failure of the old data is avoided, namely, the performance of the distributed system is ensured while the data synchronization is realized.

The steps described above are explained in detail below.

In step S210, in response to the data synchronization request, the first sub-process determines a data processing policy according to the request type.

In the exemplary embodiment of the present disclosure, the data synchronization request is issued by a data processing module of the distributed system, and the data processing module may process the data to be synchronized to obtain standard data, and store the standard data in the database. The data to be synchronized is usually configuration data of a user, and the data processing module processes the configuration data of the user and stores the processed configuration data in the database.

The data processing module may perform preprocessing, such as format conversion, data screening, etc., on the data to be synchronized to obtain data that is required by the subsystem and is suitable for the subsystem.

The first sub-process may be any of a plurality of sub-processes of the subsystem. For example, a currently idle sub-process may be determined to be the first sub-process; for another example, a rotation mechanism may be employed to rotate one process at a time from among a plurality of sub-processes as the first sub-process. Of course, the embodiments of the present disclosure may also determine the first sub-process from the plurality of sub-processes by using other manners, which is not limited in particular.

The request type of the embodiment of the disclosure comprises modifying data, deleting data and adding data, and the first subprocess determines a corresponding data processing strategy according to the request type. The correspondence between different request types and the data processing policies may be preset, so that the first sub-process determines the data processing policies based on the correspondence.

In step S220, the first sub-process performs data synchronization on the shared memory according to the data processing policy; the first subprocess monitors the read state of the old data corresponding to the data synchronization based on the timing mechanism so as to perform invalidation processing on the old data according to the read state.

In the exemplary embodiment of the disclosure, the shared memory can be accessed by a plurality of sub-processes in the subsystem, the plurality of sub-processes can read or process the data in the shared memory, and the data accessed by the plurality of sub-processes can be consistent with the data of the database of the distributed system by synchronizing the data in the shared memory.

The data synchronization of the shared memory refers to operations of adding data, deleting data and modifying data to the shared memory according to a data processing policy, so that the data in the shared memory is data which is required by a subsystem and is consistent with the data in a database of the distributed system.

In some possible embodiments, when the operation of adding data to the shared memory according to the data processing policy is performed, the data synchronization does not have corresponding old data, i.e. the read state of the old data does not need to be monitored based on a timing mechanism.

In some possible embodiments, when modifying or deleting data in the shared memory according to the data processing policy, because of the modification or deletion of old data, in order to avoid affecting the data processing process of other sub-processes that have read old data, the first sub-process needs to monitor the read state of the corresponding old data based on the timing mechanism to synchronize the data, so as to perform invalidation processing on the old data according to the read state.

It should be noted that, after modifying the data of the shared memory according to the data processing policy, the subsequent sub-process accesses the modified data when accessing the shared memory, so as to ensure timeliness of the data consistency of the sub-process and the data of the distributed system.

Wherein, the old data is in the read state, which means that the old data is accessed by the process and is used for data processing, such as message processing, etc. If the old data is no longer in the read state, the process accessing the old data is indicated to have finished processing the message.

The invalidation processing of the old data can be to delete the old data in the shared memory and release the corresponding memory space, so that the shared memory of the subsystem is consistent with the database of the distributed system.

When the data synchronization is performed on the shared memory, the read state of the old data corresponding to the data synchronization is monitored through the first subprocess, so that the old data is subjected to invalidation processing according to the read state, the influence on the system performance of the subsystem is avoided while the data consistency of the subsystem and the database of the distributed system is ensured, namely, new data can be accessed in time, the old data is reserved to the subprocess to finish data processing, and the stability of the system is maintained.

In an exemplary embodiment, an implementation of data synchronization of a shared memory by a first sub-process is provided. As shown in fig. 3, the first sub-process performs data synchronization on the shared memory according to the data processing policy, which includes steps S310 to S330:

step S310: and after the first subprocess acquires the security lock of the shared memory, carrying out data synchronization on the shared memory according to a data processing strategy.

In the embodiment of the disclosure, after the first sub-process determines the data processing policy in response to the data synchronization request, the first sub-process obtains the secure lock of the shared memory, and may read or process (e.g. delete, adjust, etc.) the data in the shared memory, while in the current data synchronization process, other sub-processes may read the data in the shared memory, but cannot modify the data.

Step S320: the first subprocess monitors the read lock count of the old data corresponding to the data synchronization based on a timing mechanism, and the read lock count is determined according to the number of the second subprocess which has read the old data.

When the first sub-process performs data synchronization on the shared memory, the read lock count of the old data corresponding to the data synchronization can be continuously monitored. The read lock count of the subsystem in the disclosed embodiments is determined based on the number of second sub-processes that have read old data.

In an exemplary embodiment, in an nmginx system, the read lock count is determined according to the number of worker cores that acquire an old data block in the shared memory, when a sub-process acquires the read lock of the old data block, the read lock count of the old data block is incremented by 1, and when the sub-process releases the read lock of the old data block, the read lock count of the old data block is decremented by 1.

The read status of the old data block may be determined from the read lock count of the old data. And if the read lock count is non-zero, indicating that the sub-process still exists to read the old data block to perform a corresponding data processing process.

Step S330: and if the read lock count is zero, performing invalidation processing on the old data.

When the read lock count of the old data corresponding to the data synchronization is zero, the old data is indicated that the current failure of the old data cannot cause the influence on the system performance or cause the system error, and the old data can be processed for failure.

According to the embodiment of the disclosure, based on the read lock count of the old data, whether the read lock is completely released by the subprocess or not can be judged, so that the invalidation processing of the old data can be carried out, and the stability of the system performance is ensured.

In an exemplary embodiment, the timing mechanism may be based on determining a dead time, and before the dead time expires, the dead processing is temporarily not performed on the old data, where the dead time is the time for ensuring that a sub-process that acquires the old data read lock after the data synchronization processing has processed the message and releases the read lock.

In one possible embodiment, if the request type is modified data, the first sub-process performing data synchronization on the shared memory according to the data processing policy may include:

the first subprocess firstly modifies old data in the shared memory to obtain target data, wherein the target data is used for subsequent data reading; the first sub-process then adds a dead time to the old data, and performs a dead process on the old data according to the read state when the dead time expires.

Optionally, after the expiration of the expiration time, if the old data is no longer in the read state, the old data is subjected to an invalidation process, so as to avoid occurrence of misoperation. Optionally, before the expiration of the invalidation time, if it is monitored that the old data is no longer in the read state for more than a preset duration, invalidation processing can be performed on the old data in advance, so as to reduce occupation of the shared memory.

Further, in an exemplary embodiment, before the expiration time has not expired, there will be old data and target data in the shared memory, based on which an implementation of determining read data is also provided. The first identifier can be configured for the target data, and the second identifier can be configured for the old data, wherein the first identifier has a higher priority than the second identifier; and before the target data is obtained and the expiration time expires, preferentially reading the target data with the first identifier.

For example, for the nmginx system, before the target data is obtained and the expiration time expires, there are two pieces of data with the same index in the shared memory, that is, the same index corresponds to the old data and the target data. And a first identifier, such as an invalidation identifier bit 0, can be configured to target data, a second identifier, such as an invalidation identifier bit 1, can be configured to old data, and when the subsequent worker core acquires the data of the shared memory, the data with the invalidation identifier bit 0 can be preferentially queried based on the index, namely, the data after consistency processing is used for message processing.

It should be noted that the first identifier and the second identifier are merely exemplary, and may be set according to actual requirements, which is not particularly limited in the embodiments of the present disclosure.

In practical implementation, the time of adding the old data can be controlled accurately by adding a timer to the old data, wherein the time pointer of the timer is stored in the old data.

In an exemplary embodiment, if the request type is delete data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, which may be to add an expiration time to the old data, and delete the old data according to the read state when the expiration time expires.

In the present exemplary embodiment, the old data may also be deleted according to the read state of the old data when the expiration time expires by adding a timer to the old data, the time pointer of which is stored in the old data. If the last worker core in the Nginx system releases the read lock of the shared memory and the timer expires, the old data is subjected to invalidation processing.

Based on the method, the sub-process which has read the old data can be ensured to finish data processing, and the influence on the system stability caused by deleting the data in the shared memory is avoided.

In an exemplary embodiment, an implementation of determining a time to failure is also provided. As shown in fig. 4, the process of determining the expiration time may include steps S410 and S420:

Step S410: based on the read state of the old data, the number of sub-processes of the read old data and the service types corresponding to the sub-processes are obtained.

The read state of the old data may include the number of sub-processes that read the old data, as well as the type of traffic each sub-process is handling. Normally, the time that the old data is in the read state continuously is related to the number of sub-processes for reading the old data, the number of the sub-processes is large, and the failure time is required to be delayed; correspondingly, if the service type corresponding to the sub-process is a type which takes longer time, the failure time is also required to be delayed.

Step S420: and determining the failure time according to the number of sub-processes and the service type.

In the embodiment of the disclosure, the failure time can be determined by combining the number of sub-processes of the read old data and the corresponding service types.

The first correspondence between the number of different sub-processes and the time may be preset, so that the first time is determined according to the number of sub-processes based on the first correspondence. And a second corresponding relation between the service type and the time is preset, so that the second time is determined according to the service type of the sub-process of the read old data based on the second corresponding relation, and the failure time is determined according to the first time and the second time.

The first correspondence and the second correspondence may be determined according to history information of each sub-process of the subsystem, and the manner of determining the first correspondence and the second correspondence is not particularly limited in the embodiments of the present disclosure.

Alternatively, when determining the dead time according to the first time and the second time, weight information may be respectively allocated to the first time and the second time, so as to perform weighted summation on the first time and the second time based on the weight information, thereby obtaining the dead time. The second time is the second time with the largest value in the second time determined according to the service type in the plurality of sub-processes for reading the old data, so that the old data can be fully utilized.

Alternatively, when the dead time is determined according to the first time and the second time, the time with a larger value in the first time and the second time can be taken as the dead time, so that the accessed time of the old data is fully ensured.

Of course, the embodiments of the present disclosure may also combine the first time and the second time by other manners to obtain the dead time, and the embodiments of the present disclosure include, but are not limited to, the above-mentioned combining manners.

In an exemplary embodiment, if the request type is to add data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, which may be to add the data to be added to the shared memory. That is, if there is added data, the data to be added is directly added to the shared memory, and the monitoring process of the old data is not involved.

Fig. 5 shows a complete flowchart of data synchronization according to an exemplary embodiment of the present disclosure, and in conjunction with fig. 5, a data synchronization method according to an embodiment of the present disclosure will be described below by taking a subsystem as an nmginx system as an example.

Step S510: the data processing module in the distributed system receives the configuration data of the user, processes the configuration data to obtain standard data, and stores the standard data in the database.

Step S520: the data processing module sends a data synchronization request to the subsystem Nginx system.

The distributed system can send a data synchronization request to the Nginx system through an http message.

Step S530: the worker A in the Nginx system receives the data synchronization request and responds.

The worker A determines a data processing strategy according to the request type of the data synchronization request. If the request type is add data, the process goes to step S540, if the request type is modify data, the process goes to step S550, and if the request type is delete data, the process goes to step S560.

Step S540: and adding the data to be added into the shared memory.

The data to be added can be integrated to obtain data adapted to the Nginx system and stored in the shared memory.

Step S550: modifying old data in the shared memory to obtain target data, adding a timer to the old data, configuring a first identifier for the target data and configuring a second identifier for the old data.

And before the target data is obtained and the timer expires, preferentially reading the target data with the first identifier. And deleting the old data after the timer expires.

Step S560: and adding the expiration time to the old data, and deleting the old data according to the read state when the timer expires.

Before the expiration of the expiration time, the sub-process that reads the data to be deleted (old data) continues to process the message by using the old data.

Therefore, the system problem caused by modification or deletion of the data in the shared memory is avoided while the new data is synchronized to the shared memory, and the deletion of the old data is delayed by adopting a timing mechanism, so that the competition between read-write locks is reduced, the new data is ensured to be accessed in time when the data are synchronized by the multi-core subsystem, the old data is fully utilized, and the stability of the system is ensured.

In a subsystem in a distributed system, a data synchronization request is received and responded through a first sub-process, a data processing strategy is determined according to a request type, and data synchronization is performed on a shared memory according to the data processing strategy. The first subprocess monitors the read state of the old data corresponding to the data synchronization based on the timing mechanism, and performs failure processing on the old data according to the read state. On one hand, the first subprocess performs data synchronization on the shared memory according to a data processing strategy, so that data in the subsystem is consistent with data of the distributed system, the problem of abnormal system functions or performance reduction caused by inconsistent data of each system is avoided, and the advantages of the distributed system are fully exerted; on the other hand, when the first subprocess performs data synchronization on the shared memory, the read state of the old data corresponding to the data synchronization is monitored based on a timing mechanism, and as other subprocesses which have read the old data possibly exist during the data synchronization, in order to prevent the processing progress and the result of the other subprocesses from being influenced, the first subprocess performs failure processing on the old data according to the read state of the old data, so that the problem of abnormal system functions or performance caused by early failure of the old data is avoided, namely, the performance of a distributed system is ensured while the data synchronization is realized; on the other hand, aiming at the currently popular distributed system, the consistency management of the multi-core subsystem can be realized through the scheme, the data consistency of each sub-process or core in the multi-core subsystem is realized under the condition that the existing forwarding performance and function are not influenced, and a data basis is provided for fully utilizing the advantages of the distributed system.

Further, in accordance with an exemplary embodiment of the present disclosure, there is also provided a distributed system, as shown in fig. 6, the distributed system 600 includes a data processing module 610 and a plurality of subsystems 620.

A data processing module 610 configured to process the data to be synchronized and update the database according to the obtained standard data;

the data processing module 610 is further configured to send data synchronization requests to the subsystem 620 in turn.

Any subsystem 620 configured to perform data synchronization processing according to the data synchronization method of any of the above-described exemplary embodiments to obtain subsystem data consistent with a database of the distributed system.

Specifically, the data processing module 610 processes the data to be synchronized to obtain standard data, and stores the standard data in the database. The data to be synchronized is usually user-configured data, and the data processing module processes the user-configured data and stores the processed data in the database.

The data processing module 610 may perform preprocessing, such as format conversion, data filtering, etc., on the data to be synchronized to obtain data suitable for the subsystem, and the specific mode of preprocessing is not limited in the embodiment of the present disclosure.

The data processing module 610 sends a data synchronization request to the subsystem 620 by means of an http (Hypertext Transfer Protocol ) message to request the subsystem 620 to complete a consistency update with the database. The subsystem 620 may include a subsystem of a distributed system such as an nginnx system, an AI system, and the like, which is not particularly limited.

Note that, in each subsystem 620, if the subsystem 620 is a single core or single process system, the data of the subsystem is consistent with the database of the distributed system, which is not described in detail.

Wherein each subsystem 620 may have a processing priority, the data processing module 610 sequentially sends data synchronization requests to each subsystem 620 according to the processing priority.

In an exemplary embodiment, the distributed system may further include a data forwarding module, such as shown in fig. 7, which is a schematic diagram of another distributed system according to an exemplary embodiment of the present disclosure.

As shown in fig. 7, the user may perform data configuration through the web configuration front end to obtain configuration data, and send the configuration data to the data processing module 610, so that standard data obtained by processing by the data processing module 610 is stored in the database. That is, standard data in the database needs to be consistent with data of each subsystem.

The data processing module 610 may call an interface function of the data forwarding module 630 to complete synchronization of forwarding data, and determine whether to continue data synchronization for each subsystem according to a response result of the data forwarding module 630.

Specifically, if the response of the data forwarding module 630 is failure, and the synchronization of forwarding data may be failed due to communication between systems or illegal user data, the data processing module 610 receives the failed response message, and the data processing module 610 rolls back the user data of the database and returns the reason and the result of the failure of synchronizing data to the web front end of the user.

If the response of the data forwarding module 630 is successful, the data processing module 610 sends a data synchronization request to the subsystem (such as the nginnx system, the AI system, etc.) by using an http message, so that a certain sub-process of the subsystem determines a data processing policy according to the request type and executes the data processing policy, so that the data in the shared memory of the subsystem is consistent with the database.

Further, after the first subsystem completes data synchronization, the data processing module 610 will continue to send data synchronization requests to the second subsystem, requesting the second subsystem to perform data synchronization until all the sub-processes complete data synchronization.

In an exemplary embodiment, subsystem 620 is configured to perform: after the security lock of the shared memory is obtained through the first subprocess, carrying out data synchronization on the shared memory according to the data processing strategy; monitoring a read lock count of old data corresponding to the data synchronization based on a timing mechanism, wherein the read lock count is determined according to the number of second sub-processes which have read the old data; and if the read lock count is zero, performing invalidation processing on the old data.

In an exemplary embodiment, if the request type is modification data, subsystem 620 is configured to perform: modifying old data in the shared memory through a first subprocess to obtain target data, wherein the target data is used for subsequent data reading; and adding failure time to the old data, and performing failure processing on the old data according to the read state when the failure time expires.

In an exemplary embodiment, subsystem 620 is configured to perform: configuring a first identifier for target data through a first subprocess, and configuring a second identifier for old data, wherein the first identifier has higher priority than the second identifier; and before the target data and the expiration time are obtained, preferentially reading the target data with the first identifier.

In an exemplary embodiment, if the request type is delete data, subsystem 620 is configured to perform: and adding an expiration time to the old data through a first subprocess, and deleting the old data according to the read state when the expiration time expires.

In an exemplary embodiment, subsystem 620 is configured to perform: acquiring the number of subprocesses which read the old data and the service types corresponding to the subprocesses through a first subprocess based on the read state of the old data; and determining the failure time according to the number of the sub-processes and the service type.

In an exemplary embodiment, if the request type is add data, subsystem 620 is configured to perform: and adding the data to be added into the shared memory through the first subprocess.

In the embodiment of the disclosure, the data processing module sequentially sends the data synchronization request to the subsystems so as to synchronize the data of each subsystem, thereby ensuring the consistency of the data in each subsystem in the distributed system, improving the reliability and expandability of the distributed system, preventing the system problem caused by inconsistent data, fully showing the high performance advantage of the distributed system, realizing the storage application of fast read-write speed, facilitating sharing, realizing large-scale, high reliability, high performance and super fusion, and being suitable for the application scene with continuously improved requirements on the expandability and the storage performance.

Since specific details of the respective functional modules (subsystems) of the distributed system according to the exemplary embodiment of the present disclosure have been described in detail in the above embodiment of the data synchronization method, a detailed description thereof will be omitted.

It should be noted that although several modules or units of a distributed system are mentioned in the above detailed description, this partitioning is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.

Furthermore, in exemplary embodiments of the present disclosure, a computer storage medium capable of implementing the above-described method is also provided. On which a program product is stored which enables the implementation of the method described above in the present specification. In some possible embodiments, the various aspects of the present disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the disclosure as described in the "exemplary methods" section of this specification, when the program product is run on the terminal device.

The disclosed embodiments also provide a program product for implementing the above method, which may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The computer readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. Those skilled in the art will appreciate that the various aspects of the present disclosure may be implemented as a system, method, or program product. Accordingly, various aspects of the disclosure may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" system.

An electronic device 800 according to such an embodiment of the present disclosure is described below with reference to fig. 8. The electronic device 800 shown in fig. 8 is merely an example and should not be construed to limit the functionality and scope of use of embodiments of the present disclosure in any way.

As shown in fig. 8, the electronic device 800 is embodied in the form of a general purpose computing device. Components of electronic device 800 may include, but are not limited to: the at least one processing unit 810, the at least one storage unit 820, a bus 830 connecting the different system components (including the storage unit 820 and the processing unit 810), and a display unit 840.

Wherein the storage unit stores program code that is executable by the processing unit 810 such that the processing unit 810 performs steps according to various exemplary embodiments of the present disclosure described in the above section of the present specification.

Storage unit 820 may include readable media in the form of volatile storage units such as Random Access Memory (RAM) 821 and/or cache memory unit 822, and may further include Read Only Memory (ROM) 823.

The storage unit 820 may also include a program/utility 824 having a set (at least one) of program modules 825, such program modules 825 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.

Bus 830 may be one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

The electronic device 800 may also communicate with one or more external devices 900 (e.g., keyboard, pointing device, bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 800, and/or any device (e.g., router, modem, etc.) that enables the electronic device 800 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 850. Also, electronic device 800 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 over bus 830. It should be appreciated that although not shown, other hardware and/or software modules may be used in connection with electronic device 800, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage systems, and the like.

From the above description of embodiments, those skilled in the art will readily appreciate that the example embodiments described herein may be implemented in software, or in combination with the necessary hardware. Thus, the technical solution according to the embodiments of the present disclosure may be embodied in the form of a software product, which may be stored in a non-volatile storage medium (may be a CD-ROM, a U-disk, a mobile hard disk, etc.) or on a network, and includes several instructions to cause a computing device (may be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

Furthermore, the above-described figures are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It will be readily appreciated that the processes shown in the above figures do not indicate or limit the temporal order of these processes. In addition, it is also readily understood that these processes may be performed synchronously or asynchronously, for example, among a plurality of modules.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This disclosure is intended to cover any adaptations, uses, or adaptations of the disclosure following the general principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

Claims (10)

1. A data synchronization method, applied to a subsystem in a distributed system, the subsystem including a plurality of sub-processes and a shared memory between the plurality of sub-processes, the data synchronization method comprising:

responding to the data synchronization request, and determining a data processing strategy by the first subprocess according to the request type;

the first subprocess performs data synchronization on the shared memory according to the data processing strategy; the first sub-process monitors the read state of the old data corresponding to the data synchronization based on a timing mechanism, so as to perform failure processing on the old data according to the read state.

2. The method of claim 1, wherein the first sub-process performs data synchronization on the shared memory according to the data processing policy, comprising:

after the first subprocess acquires the security lock of the shared memory, carrying out data synchronization on the shared memory according to the data processing strategy;

monitoring a read lock count of old data corresponding to the data synchronization based on a timing mechanism, wherein the read lock count is determined according to the number of second sub-processes which have read the old data;

and if the read lock count is zero, performing invalidation processing on the old data.

3. The method of claim 1, wherein if the request type is modified data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, comprising:

the first subprocess modifies old data in the shared memory to obtain target data, wherein the target data is used for subsequent data reading;

and adding failure time to the old data, and performing failure processing on the old data according to the read state when the failure time expires.

4. A method according to claim 3, characterized in that the method further comprises:

configuring a first identifier for the target data and a second identifier for the old data, wherein the first identifier has a higher priority than the second identifier;

and before the target data and the expiration time are obtained, preferentially reading the target data with the first identifier.

5. The method of claim 1, wherein if the request type is delete data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, comprising:

and adding an expiration time to the old data, and deleting the old data according to the read state when the expiration time expires.

6. The method of claim 3 or 5, wherein determining the time to failure comprises:

based on the read state of the old data, acquiring the number of sub-processes of the read old data and the service types corresponding to the sub-processes;

and determining the failure time according to the number of the sub-processes and the service type.

7. The method of claim 1, wherein if the request type is add data, the first sub-process performs data synchronization on the shared memory according to the data processing policy, comprising:

and adding the data to be added into the shared memory.

8. A distributed system comprising a data processing module and a plurality of subsystems;

the data processing module is configured to process the data to be synchronized and update a database according to the obtained standard data;

the data processing module is further configured to sequentially send a data synchronization request to the subsystem;

any of the subsystems is configured to perform a data synchronization process according to the data synchronization method of any of claims 1 to 7 to obtain subsystem data consistent with the database.

9. A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method according to any of claims 1 to 7.

10. An electronic device, comprising:

a processor; and

a memory for storing executable instructions of the processor;

wherein the processor is configured to perform the method of any one of claims 1 to 7 via execution of the executable instructions.