JP2020194217A - Redundant system and data synchronization method - Google Patents

Abstract

To maintain data integrity while avoiding a problem caused by synchronization of abnormal data among a plurality of controllers when recovering from split brain.SOLUTION: A redundant system 1 is composed of two controllers 10A and 10B. The controller 10A or 10B includes: a determination unit 31A or 31B for determining whether abnormality has occurred in a database 25A or 25B when its own device is an active system; a detection unit 32A or 32B for detecting whether two or more controllers are active systems when its own device is an active system; a starting unit 33A or 33B for rebooting the own device and making it a standby system when the detection unit detects that two or more controllers are active systems and the determination unit determines that abnormality has occurred in the database; and a synchronization unit 35A or 35B that synchronizes data of the own device with a standby controller when the detection unit detects that two or more controllers are not active systems and the determination unit determines that abnormality has not occurred in the database.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for synchronizing data between a plurality of controllers constituting a redundant system.

Conventionally, in a process control system used for process control of a plant or the like, when high reliability is required, a configuration having a redundant structure in which a plurality of controllers are provided has been used. In this way, by multiplexing the controllers and making the system redundant, the system can operate more stably. A conventional plant control system having such a redundant structure includes, for example, a first controller that operates as an active system and a second controller that operates as a standby system, and when an abnormality occurs in the first controller, it stands by. Switch the second controller, which was a system, to the current system.

By the way, in such a redundant system, a state in which a plurality of active nodes exist is called "split brain". Generally, in a redundant system, a split brain occurs when the network connecting the active node and the standby node is interrupted due to a failure or the like, and the active node and the standby node operate independently. ..

In split brain, the active node and the standby node operate independently, so that each node holds different data. Therefore, if the system recovers from split brain in this state, the data will not be consistent between the active node and the standby node. Therefore, a technique for recovering the system from split brain has been proposed so that there is no data difference between the active node and the standby node.

For example, in the remote cluster system described in Patent Document 1, when recovering from split brain, the active node (primary server) receives the difference data updated by the local server during communication interruption to the standby node (secondary). Send to server). The secondary server merges the difference data received from the primary server with the difference data updated on the local server during communication interruption, and creates the transmission data to be sent to the primary server and the reflection data to be reflected on the local server. To do. Then, the secondary server reflects the reflected data on the own server and transmits the transmitted data to the primary server. The primary server reflects the transmission data sent from the secondary server on its own server. This remote cluster system thus recovers the system from split brain so that there is no difference in the data held by both servers.

Japanese Unexamined Patent Publication No. 2006-146299

By the way, split brain may also occur in a facility monitoring system that monitors a plurality of monitored devices installed in a facility such as a building. For example, a facility monitoring system generally has a dual configuration consisting of two controllers, an active system and a standby system, but if the network connecting both controllers is interrupted due to a failure or the like, a split brain occurs. In this case, since the active controller and the standby controller operate independently, each controller holds different data. However, in the conventional facility monitoring system, when recovering from split brain, processing for ensuring data consistency between both controllers is not performed, and improvement is required.

Further, in the remote cluster system described in Patent Document 1, it is not confirmed whether or not there is an abnormality in the data held by each of the primary server and the secondary server. Therefore, if there is any abnormality in the data held by the primary server, the abnormal data may be synchronized with the secondary server, causing a problem such as the system going down.

The present invention has been made to solve the above-mentioned problems, and data matching while avoiding the occurrence of a problem due to abnormal data synchronization between a plurality of controllers when recovering from split brain. The purpose is to maintain sex.

The redundant system according to the present invention is configured to include a plurality of controllers, and each controller has a determination unit for determining whether or not an abnormality has occurred in the database when the own machine is an active system, and the own machine. In the case of an active system, a detection unit that detects whether or not two or more controllers have become active systems, and a detection unit that detects that two or more controllers have become active systems and a judgment unit. However, when it is determined that an error has occurred in the database, the startup unit that restarts the machine and makes it a standby system and the detection unit detect that two or more controllers are not in the active system. In addition, the determination unit is provided with a synchronization unit that synchronizes the data of the own machine with the controller of the standby system when it is determined that no abnormality has occurred in the database.

According to the present invention, since it is configured as described above, it is possible to maintain data consistency while avoiding the occurrence of a problem due to abnormal data synchronization between a plurality of controllers when recovering from split brain. it can.

It is a figure which shows the configuration example of the redundant system (redundant system) which concerns on Embodiment 1. FIG. It is a figure which shows the image of the data stored in the SSD of the controller in Embodiment 1. FIG. FIG. 3A is a diagram showing a configuration example of the active system controller according to the first embodiment, and FIG. 3B is a diagram showing a configuration example of the standby system controller according to the first embodiment. It is a flowchart which shows the operation example when the split brain occurs in the duplex system which concerns on Embodiment 1. It is a timing chart which shows the operation example when the split brain occurs in the duplex system which concerns on Embodiment 1. FIG. It is a flowchart which shows the flow of the process which the controller in Embodiment 1 determines whether or not an abnormality has occurred in the database of own machine.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Embodiment 1.
FIG. 1 shows a configuration example of a redundant system according to the first embodiment of the present invention. The redundant system 1 is configured to include a plurality of (here, two) controllers 10 (10A, 10B), a plurality of monitoring points 200, and a monitoring device (not shown). The controllers 10A and 10B, each monitoring point 200, and the monitoring device are connected to each other via the system bus 150 so as to be able to communicate with each other. Here, since the controller is composed of two controllers, the redundant system 1 will be described as the redundant system 1. However, the number of controllers does not necessarily have to be two, and may be three or more as long as any one of the controllers can operate as an active system (active).

The duplication system 1 has a duplication function. The duplication function is a function of monitoring and controlling the monitoring point 200 with two controllers 10A and 10B. That is, in the duplex system 1, the controller 10A is normally operated as an active system and the controller 10B is operated as a standby system (standby), and the active controller 10A performs processing such as monitoring, control, and data collection of the monitoring point 200. Do. The controller 10A includes an SSD (Solid State Drive) 20A in which the database 25A is constructed, and the controller 10A stores the data collected from the monitoring point 200 in the database 25A in the SSD 20A.

The duplication feature also manages database synchronization. For example, in the duplex system 1, when the standby system controller 10B is activated while the active system controller 10A is activated, the standby system controller 10B requests the active system controller 10A to copy the database 25A. The controller 10A that has received this request transmits a copy of the database 25A to the controller 10B. When the controller 10B receives a copy of the database 25A from the controller 10A, the controller 10B constructs the database 25B in the SSD 20B provided in the own machine by using the copy.

After that, each time the controller 10A completes the update of the database 25A, the controller 10A transmits a copy of the updated data of the database 25A to the controller 10B. The controller 10B writes a copy of the update data received from the controller 10A to the database 25B, and completes the update of the database 25B. In this way, the duplication system 1 synchronizes the databases 25A and 25B.

Further, the duplex system 1 switches the standby system controller 10B to the active system when a problem such as a failure occurs in the active system controller 10A. The controller 10B switched to the active system continues processing such as monitoring, control, and data collection of the monitoring point 200, and stores the data collected from the monitoring point 200 in the database 25B in the SSD 20B.

In the first embodiment, as shown in FIG. 2, the controllers 10A and 10B are not stored in the databases 25A and 25B but are connected to the other controller in addition to the data stored in the databases 25A and 25B. The data 26A and 26B to be synchronized are stored in the SSD20A and SSD20B.

<Configuration example of controllers 10A and 10B>
Next, a configuration example of the controllers 10A and 10B will be described with reference to FIG. FIG. 3A shows a configuration example of the controller 10A, and FIG. 3B shows a configuration example of the controller 10B.

As shown in FIG. 3A, the controller 10A includes an SSD (Solid State Drive) 20A and a CPU (Central Processing Unit) 30A. The database 25A described above is constructed in the SSD 20A.

The CPU 30A functions as a determination unit 31A, a detection unit 32A, an activation unit 33A, a notification unit 34A, and a synchronization unit 35A by executing a predetermined program stored in the SSD 20A in advance.

The determination unit 31A determines whether or not an abnormality has occurred in the database 25A of the own machine when the own machine (controller 10A) is an active system. For example, the determination unit 31A transmits a predetermined signal (for example, Ping) to the database 25A of its own machine, and if there is a response within a predetermined time (for example, 1 minute), it is determined that the database 25A can be connected (normal). To do. On the other hand, if there is no response within the predetermined time, the determination unit 31A determines that an abnormality has occurred in the database 25A. The determination unit 31A transmits a ping to the database 25A and executes an abnormality determination of the database 25A at least when the controller 10A is started as an active system.

When the own machine (controller 10A) is a working system, the detection unit 32A detects whether or not both of the two controllers constituting the duplex system 1 are working systems, that is, whether or not they are split brains. .. For example, the detection unit 32A monitors each other's operating status at any time by transmitting and receiving signals to and from the controller 10B, and when the own machine (controller 10A) is operating as the active system, the controller 10B is in use from the standby system. Detects split brain when switching to the system. A specific example in which the detection unit 32A detects split brain will be described later.

In the activation unit 33A, the above-mentioned detection unit 32A detects that both of the two controllers have become active systems (split brain), and the above-mentioned determination unit 31A is the database 25A of the own machine (controller 10A). If it is determined that an abnormality has occurred in the system, restart the own machine and switch from the active system to the standby system.

Further, the activation unit 33A detects that both of the two controllers have become active systems by the detection unit 32A, and determines that no abnormality has occurred in the database 25A by the determination unit 31A. Even if there is, if the notification by the notification unit 34B described later of the controller 10B is received and the priority of the own machine is low with the controller 10B of the notification source, the own machine is restarted to be a standby system. ..

The priority indicates the degree of which controller is preferentially operated as the active system, and when the priority is lower than that of other controllers, the controller is switched from the active system to the standby system. The priority may be set in advance by the user (administrator) in each controller.

When the determination unit 31A determines that an abnormality has not occurred in the database 25A, the notification unit 34A notifies another controller (here, the controller 10B) that no abnormality has occurred.

When the synchronization unit 35A determines that an abnormality has not occurred in the database 25A of its own machine by the determination unit 31A, and detects that two or more controllers are not in the active system by the detection unit 32A. In addition, the data held by the own machine is synchronized with the standby controller.

Next, a configuration example of the controller 10B will be described with reference to FIG. 3B. As shown in FIG. 3B, the controller 10B includes an SSD (Solid State Drive) 20B and a CPU (Central Processing Unit) 30B. The database 25B described above is constructed in the SSD 20B.

The CPU 30B functions as a determination unit 31B, a detection unit 32B, an activation unit 33B, a notification unit 34B, and a synchronization unit 35B by executing a predetermined program stored in the SSD 20B in advance. The functions of the determination unit 31B, the detection unit 32B, the activation unit 33B, the notification unit 34B, and the synchronization unit 35B are the above-mentioned determination unit 31A, detection unit 32A, activation unit 33A, notification unit 34A, and synchronization unit 35A. Since it is basically the same as, the description thereof is omitted here.

<Operation example of duplex system 1>
Next, in the duplex system 1 according to the first embodiment, an operation example when a split brain occurs will be described with reference to the flowchart of FIG. 4 and the timing chart of FIG. Here, as a specific example, an operation example will be described from a state in which the power supply of the controller 10A is OFF and the controller 10B is operating as an active system. Further, here, it is assumed that the database 25B of the controller 10B operating as the active system has already been determined to be normal by the determination unit 31B.

Further, here, in FIG. 4, the controller 10A is simply referred to as “A system”, and the controller 10B is simply referred to as “B system”. Further, in FIG. 5, the numbers in parentheses correspond to the step numbers in the flowchart of FIG. 4, “A” indicates that the controller is the active system, and “S” indicates that the controller is the standby system.

First, the duplex system 1 starts the controller 10A as a standby system (step ST11).

Next, the controller 10A requests the controller 10B operating as an active system to copy the data (database 25B and the synchronized data 26B (not shown)) held by the controller 10B. Upon receiving this request, the controller 10B transmits a copy of the data held by the own machine to the controller 10A, and synchronizes the data held by the controller 10B with the data held by the controller 10A (step ST12).

Here, it is assumed that the network connecting the controller 10A and the controller 10B is disconnected due to a failure or the like during this synchronization. Then, the synchronization process is interrupted in the duplex system 1, and incomplete data in the middle of synchronization is left in the database 25A of the controller 10A (step ST13). In addition, in FIG. 5, the fact that the display of the database 25A is partially broken lines indicates that incomplete data is left in the database 25A.

Next, the duplex system 1 temporarily turns off the power of the controller 10A and starts the controller 10A as an active system in order to operate the controller 10A independently when the network is disconnected (step ST14). At this time, the controller 10A determines whether or not an abnormality has occurred in the database 25A of the own machine. The processing in this case will be described with reference to the flowchart of FIG.

FIG. 6 is a flowchart showing a flow of processing in which the controller 10A determines whether or not an abnormality has occurred in the database 25A of the own machine. First, when the controller 10A starts starting (step ST41), the controller 10A starts starting the database 25A (step ST42). Then, the controller 10A transmits a ping to the database 25A by the determination unit 31A (step ST43).

Next, the determination unit 31A confirms whether or not there is a response from the database 25A within a predetermined time (for example, 1 minute) (step ST44). As a result, if there is a response from the database 25A within a predetermined time (step ST44; YES), the determination unit 31A determines that the database 25A has started up normally (step ST45). Then, the controller 10A normally completes the startup (step ST46).

On the other hand, if there is no response from the database 25A within a predetermined time (step ST44; NO), the determination unit 31A determines that an abnormality has occurred in the database 25A (step ST47). In this case, the controller 10A becomes an error state and stops starting (step ST48).

In the examples of FIGS. 4 and 5, since the network of the duplex system 1 was disconnected during the synchronization process, incomplete data during the synchronization is left in the database 25A of the controller 10A. Therefore, the determination unit 31A cannot obtain a response from the database 25A to the transmitted Ping, and determines that an abnormality has occurred in the database 25A. As a result, the controller 10A becomes an error state and stops starting (step ST15).

After that, in the duplex system 1, the disconnected network is restored (step ST16). As a result, the duplex system 1 resumes communication between the controller 10A and the controller 10B.

After that, in the duplex system 1, the detection unit 32A of the controller 10A detects the split brain. Similarly, the detection unit 32B of the controller 10B also detects split brain (step ST17).

Next, the controller 10A notifies the controller 10B that an abnormality has occurred in the database 25A of its own machine by communication. As a result, the controller 10B detects that an abnormality has occurred in the database 25A of the controller 10A (step ST18). The process of step ST18 does not necessarily have to be executed, but it is desirable that the controller 10B be executed in order to reliably detect an abnormality in the database 25A.

After that, the controller 10A is restarted by the activation unit 33A with its own machine as a standby system (step ST19).

Next, the controller 10B detects that the two controllers are not in the active system, that is, it is not a split brain, because the controller 10A has become a standby system (step ST20). Then, the controller 10B overwrites the data held by the controller 10B (database 25B and synchronized data 26B) with the data held by the controller 10A (database 25A and synchronized data 26A) by the synchronization unit 35B. The data held by the controller 10B is synchronized with the controller 10A (step ST21). As a result, the duplication system 1 maintains data integrity between the controller 10A and the controller 10B, and restores the duplication system to a normal duplication state.

In the above example, in step ST14, after the determination unit 31A determines that an abnormality has occurred in the database 25A in step ST14, the detection units 32A and 32B detect split brain in step ST17. However, in the duplex system 1, after the detection units 32A and 32B detect the split brain, the determination unit 31A may determine that an abnormality has occurred in the database 25A, or perform these processes in parallel. You may do it.

Further, in the above example, an operation example from a state in which the power supply of the controller 10A is turned off and the controller 10B is operating as an active system has been described. However, on the contrary, even when the controller 10A operates as an active system and the power supply of the controller 10B is OFF, each controller may recover from the split brain and synchronize data in the same flow as described above. ..

In this way, in the duplex system 1, when recovering the system from the split brain, it is determined whether or not an abnormality has occurred in the databases 25A and 25B of the controllers 10A and 10B, and it is determined that an abnormality has occurred in the database. If it is determined, the controller (controller 10A in this case) is restarted to switch from the active system to the standby system. Then, the duplex system 1 discards the data held by the controller 10A that has been switched to the standby system, and synchronizes the data of the controller 10B that is continuing to operate as the active system with the controller 10A that has been switched to the standby system. It was configured in.

In particular, conventionally, when recovering from split brain, it has not been confirmed whether or not there is an abnormality in the data held by each of the active controller and the standby controller. Therefore, if there is any abnormality in the data of the controller that is the source of the data to be synchronized, there is a possibility that the abnormal data may be synchronized and the system may go down.

On the other hand, when recovering from split brain, the duplex system 1 first determines whether or not an error has occurred in each of the databases of the active controller and the standby controller, and the database has an error. Switch the controller determined to be present from the active system to the standby system. Then, the data held by the controller switched to the standby system is discarded, and the data held by the controller that continues to operate as the active system is synchronized with the controller switched to the standby system. As a result, the duplex system 1 ensures data integrity while avoiding system malfunction due to abnormal data synchronization between the controllers 10A and the controller 10B when recovering from split brain. Can be kept.

In addition, in FIGS. 4 and 5, an example in which it was determined that an abnormality occurred in the database 25A of the controller 10A was described. For example, there is no abnormality in the database 25A of the controller 10A, and there is no abnormality in the database 25B of the controller 10B. It is also assumed that this is the case. In that case, the two controllers 10A and 10B mutually notify each other by the notification units 34A and 34B that it is determined that there is no abnormality in the database of their own machine.

Then, the controllers 10A and 10B detect that the detection units 32A and 32B have become the active system of two or more controllers, and the determination units 31A and 31B have an abnormality in the databases 25A and 25B. Even if it is determined that there is no notification, if the notification by the notification units 34A and 34B is received and the priority of the own unit is low with the controller of the notification source, the activation unit 33A and 33B will use the activation unit 33A and 33B to control the own unit. Reboot and make it a standby system.

For example, in step ST15 of FIG. 4, it is assumed that the controller 10A determines by the determination unit 31A that there is no abnormality in the database 25A. In this case, the notification unit 34A notifies the controller 10B that there is no abnormality in the database 25A. On the other hand, also in the controller 10B, since the determination unit 31B determines that there is no abnormality in the database 25B, the notification unit 34B notifies the controller 10A that the database 25B is determined to have no abnormality.

At this time, even if the controller 10A determines that no abnormality has occurred in the database 25A of the own machine, if the priority of the own machine is lower than the priority of the controller 10B, the activation unit 33A causes the own machine. Is restarted to make it a standby system. Then, when the controller 10B determines that no abnormality has occurred in the database 25B by the determination unit 31B and the detection unit 32B detects that the two controllers are not in the active system, the synchronization unit 35B This synchronizes the data of the own machine with the standby controller 10A.

On the contrary, for example, when the priority of the controller 10B is lower than the priority of the controller 10A, the controller 10B is switched to the standby system and the data of the controller 10A is synchronized with the controller 10B in the same flow as described above. Just let me do it.

Further, even if the redundant system is composed of, for example, three or more controllers and there are two or more controllers that are judged to have no abnormality in the database, the controller that switches to the standby system may be determined in the same flow as above. .. For example, if the redundant system consists of three controllers and it is determined that there is no abnormality in the database of all three controllers, each controller mutually indicates that the database of its own machine is determined to be normal by the notification unit. Notify each other. In this case, each controller compares the priority between the own machine and the controller of the notification source, and if the priority of the own machine is low, the starter restarts the own machine to make it a standby system. .. Then, the controller having the highest priority (that is, the controller that maintains the active system) may synchronize the data of its own machine with the controller of the standby system by the synchronization unit.

The method of determining the controller to switch to the standby system based on the priority is only an example, and the controller to switch to the standby system may be determined by another method. For example, the number of data stored in the database of each controller may be compared, and the controller other than the controller having the largest number of data may be switched to the standby system.

As described above, according to the first embodiment, the redundant system 1 is configured to include a plurality of controllers 10A and 10B, and the controllers 10A and 10B are stored in the databases 25A and 25B when the own machine is an active system. Judgment units 31A and 31B for determining whether or not an abnormality has occurred, and detection units 32A and 32B for detecting whether or not two or more controllers have become active systems when the own machine is an active system. When the determination units 31A and 31B determine that an abnormality has occurred in the databases 25A and 25B, and the detection units 32A and 32B detect that two or more controllers have become active systems, they themselves. The activation units 33A and 33B and the detection units 32A and 32B, which restart the machine and use it as a standby system, detect that two or more controllers are not in the active system, and the judgment units 31A and 31B When it is determined that no abnormality has occurred in the databases 25A and 25B, the synchronization units 35A and 35B for synchronizing the data of the own machine with the controller of the standby system are provided. As a result, the redundant system 1 can maintain data integrity while avoiding a system malfunction due to abnormal data synchronization between a plurality of controllers when recovering from split brain. ..

Further, according to the first embodiment, when the determination units 31A and 31B determine that no abnormality has occurred in the databases 25A and 25B, the redundant system 1 informs another controller that no abnormality has occurred. Notifying units 34A and 34B are provided, the activation units 33A and 33B detect that two or more controllers have become active systems, and the determination units 31A and 31B have databases. Even if it is determined that no abnormality has occurred in 25A or 25B, if the notification by the notification units 34A and 34B is received and the priority of the own machine is low with the controller of the notification source, Restart your own machine to make it a standby system. As a result, in the redundant system 1, even if there are a plurality of controllers that can provide data to be synchronized, the controller having a lower priority than the others is switched to the standby system, so that the data consistency between the controllers can be maintained promptly. Can be done.

Further, according to the first embodiment, the determination unit transmits a predetermined signal to the database, and when there is no response to the signal within the predetermined time, the database is said to have an abnormality. to decide. As a result, the redundant system 1 can easily determine whether or not an abnormality has occurred in the database.

In the present invention, within the scope of the invention, it is possible to modify any component of the embodiment or omit any component in the embodiment.

1 Redundant system (redundant system)
10 Controller 10A Active controller 10B Standby controller 20A, 20B SSD
25A, 25B Database 26A, 26B Synchronized data 30A, 30B CPU
31A, 31B Judgment unit 32A, 32B Detection unit 33A, 33B Activation unit 34A, 34B Notification unit 35A, 35B Synchronization unit 150 System bus 200 Monitoring point

Claims (4)

A redundant system that includes multiple controllers
Each of the above controllers
A judgment unit that determines whether or not an error has occurred in the database when the own machine is a working system,
When the own machine is a working system, a detector that detects whether or not two or more controllers have become a working system, and
When the detection unit detects that two or more controllers have become active systems and the judgment unit determines that an abnormality has occurred in the database, the own machine is restarted. The startup part that is the standby system and
When the detection unit detects that two or more controllers are not in the active system and the judgment unit determines that no abnormality has occurred in the database, it waits for the data of its own machine. A synchronization unit that synchronizes with the system controller,
A redundant system characterized by being equipped with. When the determination unit determines that an abnormality has not occurred in the database, it is provided with a notification unit for notifying another controller that no abnormality has occurred.
The starter
Even if the detection unit detects that two or more controllers have become active systems and the determination unit determines that no abnormality has occurred in the database, the notification unit will be used. The redundant system according to claim 1, wherein when a notification is received and the priority of the own machine is low with the controller of the notification source, the own machine is restarted to be a standby system. A claim characterized in that the determination unit transmits a predetermined signal to the database, and determines that an abnormality has occurred in the database when there is no response to the signal within a predetermined time. The redundant system according to claim 1 or 2. A data synchronization method using multiple controllers that make up a redundant system.
Each of the above controllers
The step that the judgment unit determines whether or not an error has occurred in the database when the own machine is a working system,
The step that the detection unit detects that two or more controllers have become the active system when the own machine is the active system,
When the detection unit detects that two or more controllers have become active systems and the judgment unit determines that an abnormality has occurred in the database, the activation unit activates the own machine. The step of restarting and making it a standby system,
When the detection unit detects that two or more controllers are not in the active system and the judgment unit determines that an abnormality has not occurred in the database, the synchronization unit causes the own machine. Steps to synchronize the data of the standby system with the controller of the standby system,
A data synchronization method characterized by being equipped with.

Images