JP5836423B2 - Data matching confirmation method and system - Google Patents

Description

  The present invention generally relates to a data matching confirmation method, and more particularly, whether or not data stored in a source database matches data stored in a duplicate database at high speed and low cost. It relates to a method and a system for checking.

  For data backup, database systems comprising a source database and a duplicate database are widely applied.

  In this type of database system, data stored in the source database is periodically copied to the duplicate database.

  Recently, utilities for high-speed copying such as bulk copying shown in Non-Patent Documents 1 to 3, for example, have been prepared, which enables data copying in a shorter time and with higher reliability. It came to be executed.

Microsoft Developer Network home page, http://msdn.microsoft.com/en-us/library/7ek5da1a(v=vs.110).aspx Oracle Data Provider for .NET Developer's Guide home page, http://docs.oracle.com/cd/E16635_01/win.111/e06104/OracleBulkCopyClass.htm IBM DB2 Database for Linux, UNIX, and Windows Information Center home page, http://pic.dhe.ibm.com/infocenter/db2luw/v9r7/index.jsp?topic=%2Fcom.ibm.swg.im.dbclient. adonet.ref.doc% 2Fdoc% 2FDB2BulkCopyClass.html

  In this way, when data is copied from the source database to the duplicate database, the data stored in the source database matches the data copied to the duplicate database after the copy is completed. It is necessary to confirm.

  In general, this type of confirmation takes a great deal of time. Of course, the larger the amount of data stored in the database, the longer it takes. Therefore, in such a case, there is a risk of hindering the original operation of the database.

  However, due to recent advances in copy technology, the reliability of copying has improved, and in practice, copying is performed correctly, and after copying, the data stored in the source database and the copy database are stored. It is very likely that the data is consistent.

  Considering the above, it is desirable that the data coincidence confirmation after copying is simpler than performing it in detail over a long time, so that it can be confirmed in a short time. For the time being, if it can be confirmed that there is no data mismatch, it can be quickly returned to the original use of the database by determining that the copy has been executed correctly.

  On the other hand, if it is determined that there is a mismatch in the data, it is only necessary to find out in detail which data has a mismatch using a known tool even if it takes some time.

  The present invention has been made in view of such circumstances. When a data record stored in a source database is copied to a duplicate database, the data stored in the source database is copied to the duplicate database. An object of the present invention is to provide a data matching confirmation method and system capable of simply and quickly determining whether or not the data matches.

  In order to achieve the above object, the present invention takes the following measures.

  That is, the invention of claim 1 stores data stored in a source database that stores a large number of data records each consisting of a plurality of data items each having data, and a copy of the data stored in the source database. This is a system for confirming whether or not the data stored in the replication database for matching is the same.

  This system includes a data acquisition unit, a memory, a hash calculation unit, and a comparison unit.

The data acquisition means executes a first process of acquiring any one of the data records stored in the source database from the source database and storing it in the memory.

The hash calculation means obtains a hash value obtained by applying a predetermined algorithm to each data corresponding to each data item of the data record stored in the memory, and obtains the hash value. Each hash value obtained is stored in the memory in association with the corresponding data item, and thereafter, a second process of deleting the data record from the memory is executed in order to reduce the memory usage. .

The data acquisition means executes a third process of acquiring any one unacquired data record from the source database and storing it in the memory.

The hash calculation means obtains each hash value obtained by applying an algorithm to each data corresponding to each data item of the data record stored in the memory, and determines each obtained By adding the hash value to each stored hash value in association with the corresponding data item, the hash value is added by one data record so as to update each stored hash value. The data is updated and stored in the memory, and then a fourth process for deleting the data record from the memory is executed in order to reduce the memory usage .

  By executing the third process by the data acquisition unit and the fourth process by the hash calculation unit up to the last data record stored in the source database, the hash calculation unit performs the above process in the source database. For each data item, a fifth process for acquiring the sum of hash values is executed.

  The data acquisition means and the hash calculation means execute the first to fifth processes on the duplicate database instead of the source database, thereby calculating the sum of hash values for each data item in the duplicate database. The sixth process to be acquired is executed.

The comparison means compares the sum of the hash values acquired in the fifth process and the sum of the hash values acquired in the sixth process for each data item, and the sum of the hash values does not match. If there is a data item, it is determined that the data stored in the source database and the data stored in the duplicate database do not match, and there is no data item whose sum of hash values does not match Performs a seventh process for determining that the data stored in the source database and the data stored in the duplicate database match , thereby causing the data acquisition means to Both the acquisition of data records and the acquisition of data records from the duplicate database are performed in any order, while the source database is Thereby enabling the data record stored, the confirmation of coincidence of the data records stored in the copy database.

  The invention of claim 2 is a method applied to the system of claim 1.

  According to the method and system for data matching according to the present invention, when a data record stored in a source database is copied to a duplicate database, the data stored in the source database is copied to the duplicate database. It is possible to easily and quickly determine whether or not the data matches.

It is a conceptual diagram which shows the structural example of the system by which the data matching confirmation method which concerns on embodiment of this invention was applied. It is a figure which shows an example of the data structure of the data stored in the source database. It is a flowchart which shows the operation example of the system to which the data matching confirmation method which concerns on embodiment of this invention was applied. It is a figure which shows the example of the hash value corresponding to the data shown in FIG. It is a conceptual diagram which shows the structural example of the system by which the data matching confirmation method which concerns on another embodiment of this invention was applied.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  FIG. 1 is a conceptual diagram showing a configuration example of a system to which a data matching confirmation method according to an embodiment of the present invention is applied.

  This system 10 is a system for confirming the coincidence of data between the source database 12 and the duplicate database 14, and is provided between the source database 12 and the duplicate database 14. , A memory 18, a hash calculation unit 20, and a comparison unit 22.

  Note that the source database 12 and the system 10 and / or the replication database 14 and the system 10 may be connected by a communication network (not shown) such as the Internet. In this case, the communication network includes a LAN such as Ethernet (registered trademark) or a WAN to which a plurality of LANs are connected via a public line or a dedicated line. In the case of a LAN, it is composed of a number of subnets via routers as necessary. In the case of a WAN, a firewall or the like for connecting to a public line is provided as appropriate, but illustration and detailed description thereof are omitted here.

  FIG. 2 is a diagram illustrating an example of the data structure 24 of data stored in the source database 12. In other words, the data structure 24 is formed in two dimensions of a data item and a data record.

  In FIG. 2, (1) personal number, (2) name, and (3) address are illustrated as data items. Of course, the number of data items is not limited to three, and there may be more or less data items. Further, the contents of the data items are not limited to these. A data record corresponds to a number given to a set of data corresponding to each data item.

  Accordingly, the source database 12 has a plurality of pieces of data (for example, “00010”, “Taro Nihon”, “1-2-3 Toranomon Minato-ku, Tokyo” in the case of the data record 1 shown in FIG. 2). A large number of data records including data items (for example, (1) personal number, (2) name, and (3) address) in the example shown in FIG. 2 are stored.

  The duplicate database 14 is a database for storing a duplicate of data stored in the source database 12, and the data is stored in a data structure 24 as shown in FIG. Is done.

  The processing executed by the system 10 in order to determine whether or not the same data is stored in the source database 12 and the duplicate database 14 will be described below with reference to the flowchart of FIG.

  First, the data acquisition unit 16 acquires data of one data record (for example, data record 1) stored in the source database 12 from the source database 12 and stores it in the memory 18 (step S1). S1). Note that the order of the data records acquired by the data acquisition unit 16 from the source database 12 may be arbitrary, and it is not always necessary to acquire the data record 1 first.

  Subsequent to step S1, the hash calculation unit 20 performs each data item (for example, (1) personal number, (2) name, data record) of the data record (for example, data record 1) stored in the memory 18 in step S1. (3) By applying a predetermined algorithm to each data corresponding to (address) (for example, “00010”, “Taro Nihon”, “1-2-3 Toranomon, Minato-ku, Tokyo”), Each hash value is obtained (step S2a). This algorithm outputs a predetermined numerical value as a hash value for a certain numerical value (for example, “0010”) or a certain character (for example, “day”).

  By using such an algorithm, the hash calculation unit 20 uses the data item (1) personal number data (“00010”) and the data item (2) name data of the data record (for example, data record 1). (“Nippon Taro”), data item (3) Hash value is obtained for each address data (“Toranomon 1-2-3 Toranomon, Minato-ku, Tokyo”).

  FIG. 4 shows that the hash values obtained for the data items (1), (2), and (3) of the data record 1 are “23”, “121”, and “323”, respectively. As an example, “121” is obtained as a hash value corresponding to “Nihon Taro”, which is the data item (2) of the data record 1. In addition, “323” is obtained as an example of the hash value corresponding to “Toranomon 1-2-3 Toranomon, Minato-ku, Tokyo” which is the data item (3) of the data record 1.

  The hash calculation unit 20 further uses the hash values obtained for the data items (1), (2), and (3) of the data record (for example, the data record 1) in this way as the corresponding data. The information is stored in the memory 18 in association with the items (1), (2), and (3). When the storage in the memory 18 is completed, the data record (for example, data record 1) used for the calculation of the hash value is deleted from the memory 18 (step 2b).

  The data acquisition unit 16 acquires the data of the next one data record (for example, data record 2) from the source database 12 and stores it in the memory 18 (step S3). As described above, the order of the data records acquired by the data acquisition unit 16 from the source database 12 may be arbitrary. For example, it is necessary to acquire the data record 2 next to the data record 1. There is no.

  The hash calculation unit 20 then stores each data item (for example, (1) personal number, (2) name, (3) address) of this data record (for example, data record 2) stored in the memory 18. Applicable to the previous data record (for example, data record 1) for each corresponding data (for example, “00200”, “Hanako Utsunomiya”, “1-2-3 Yamadacho, Utsunomiya City, Tochigi Prefecture)” Each hash value is obtained by applying the same algorithm as described above (step 4a). FIG. 4 shows that the hash values obtained for data corresponding to the data items (1), (2), and (3) of the data record 2 are “31”, “225”, and “198”, respectively. Is shown.

  The hash calculation unit 20 further corresponds to each hash value (“31”, “225”, “198”) obtained for each data item (1), (2), (3) of the data record 2 respectively. Hash values already stored in the memory 18 in association with the data items (1), (2), (3) to be performed (for example, the hash of the data items (1), (2), (3) of the data record 1) Each hash value stored in the memory 18 is updated by adding to each value. For example, “23” is stored as the hash value of the data item (1) of the data record 1, “121” is stored as the hash value of the data item (2), and “323” is stored as the hash value of the data item (3). This time, “31” is stored as the hash value of the data item (1), “225” is stored as the hash value of the data item (2), and “198” is stored as the hash value of the data item (3). If calculated, the hash value of the data item (1) is updated to 23 + 31 = 54, the hash value of the data item (2) is updated to 121 + 225 = 346, and the hash value of the data item (3) is updated. Is updated to 323 + 198 = 521 and stored in the memory 18. When the hash value is updated and stored in this manner, the hash calculation unit 20 deletes the data record (for example, data record 2) on which the hash calculation has been performed from the memory 18 (step S4b).

  By executing the processing of step S3 by the data acquisition unit 16 and the processing of steps S4a and S4b by the hash calculation unit 20 up to the last data record stored in the source database 12, the hash calculation unit 20 Obtains the sum of hash values for each data item in the source database 12 (step S5). In FIG. 4, the sum of hash values of the data item (1) is “584”, the sum of hash values of the data item (2) is “2324”, and the sum of hash values of the data item (3) is “ 432 ″.

  By performing the process including steps S1 to S5 on the duplicate database 14 in the same manner, all data stored in the duplicate database 14 is similarly hashed for each data item. Get the sum of values. In this way, the system 10 according to the present embodiment performs the processing as shown in the flowchart of FIG. 3 on the source database 12 and the duplicate database 14 respectively, so that the source database 12 and the duplicate database 14 are processed. For each of and, the total of hash values for each data item is acquired. Such processing may be performed first for the source database 12 or first for the duplicate database 14. Further, as shown in FIG. 5, by providing two systems of the data acquisition unit 16, the memory 18, and the hash calculation unit 20, such a process may be performed on both databases in parallel. good.

  In this way, when the sum of hash values of each data item is obtained for each of the source database 12 and the duplicate database 14, the comparison unit 22 obtains each data item obtained for each data item of the source database 12. The sum of hash values and the sum of hash values acquired for each data item in the replication database 14 are compared for each corresponding data item. That is, the comparison unit 22 calculates the sum of hash values acquired for the data stored in the source database 12 for each data item (1), data item (2), and data item (3), and the duplicate database. 14 is compared with the sum of the hash values acquired for the data stored in 14. If any of the data items (1), (2), and (3) has a data item whose sum of hash values does not match, the data stored in the source database 12 and a copy It is determined that there is data that does not match the data stored in the database 14. On the other hand, when there is no data item in which the sum of hash values does not match, that is, for any of the data items (1), (2), and (3), the hash value acquired for the data stored in the source database 12 And the sum of the hash values acquired for the data stored in the replication database 14 match, the data stored in the source database 12 and the data stored in the replication database 14 Is determined to match. Thereby, it can be confirmed that the data stored in the source database 12 is correctly copied to the duplicate database 14.

  As described above, in the system 10 to which the data matching check method according to the present embodiment is applied, does the data stored in the source database 12 match the data stored in the duplicate database 14? It is possible to determine in a short time because not all data are checked against each other.

  Further, in this system 10, a hash value corresponding to data is calculated for each data item with respect to one data record. When the calculation of the hash value is completed, the data of this data record is deleted from the memory 18, and the memory 18 holds only the hash value for each data item.

  Also for the next data record, a hash value corresponding to the data is calculated for each data item, and when the calculation of the hash value is completed, the data of this data record is deleted from the memory 18. Then, the calculated hash value is added to the hash value already stored in the memory 18 for each data item and updated, and only the updated hash value is held in the memory 18. Such a process is repeated until the last data record.

  As described above, in the system 10 to which the data matching confirmation method according to the present embodiment is applied, the data record used for the update of the hash value is deleted from the memory 18, and the memory 18 includes only the updated hash value. Is stored for each data item. Thus, for example, as shown in FIG. 2, when a data record including three data items is stored in the database, the memory 18 holds only three hash values. Therefore, in the present system 10, the capacity of the memory 18 may be low, and no matter how much the number of data records increases, if the number of data items does not change, it is not necessary to increase the capacity. There is no extra cost.

  In addition, since all the data stored in the source database 12 and all the data stored in the duplicate database 14 are confirmed, they are acquired from the source database 12 by the data acquisition unit 16. The order of the data records to be acquired and the order of the data records acquired from the duplicate database 14 to the data acquisition unit 16 may be arbitrary, and even if the order is different between the two databases, There is no effect.

  The best mode for carrying out the present invention has been described above with reference to the accompanying drawings, but the present invention is not limited to such a configuration. Within the scope of the invented technical idea of the scope of claims, a person skilled in the art can conceive of various changes and modifications. The technical scope of the present invention is also applicable to these changes and modifications. It is understood that it belongs to.

  The data matching confirmation method and system according to the present invention comprises a source database and a duplicate database, for example, a database system for resident data management of local governments, databases of financial institutions, securities companies, insurance companies, etc. -It is applied suitably also to a system.

DESCRIPTION OF SYMBOLS 10 System 12 Source database 14 Duplicated database 16 Data acquisition part 18 Memory 20 Hash calculation part 22 Comparison part 24 Data structure

Claims (2)

Data stored in a source database that stores a large number of data records each consisting of a plurality of data items each having data, and stored in a replica database for storing a copy of the data stored in the source database A system for confirming whether or not the data matches
A data acquisition means, a memory, a hash calculation means, and a comparison means;
The data acquisition means executes a first process of acquiring any one of the data records stored in the source database from the source database and storing it in the memory And
The hash calculation means obtains a hash value obtained by applying a predetermined algorithm to each data corresponding to each data item of the data record stored in the memory. Each of the obtained hash values is stored in the memory in association with a corresponding data item, and then the data record is deleted from the memory in order to reduce the memory usage . Execute the process of
The data acquisition means executes a third process of acquiring any one unacquired data record from the source database and storing it in the memory,
The hash calculation means obtains each hash value obtained by applying the algorithm to each data corresponding to each data item of the data record for the data record stored in the memory, and Add one data record to update each stored hash value by adding each determined hash value to each stored hash value associated with the corresponding data item A hash value is updated in a manner and stored in the memory, and then, in order to reduce the memory usage, a fourth process of deleting the data record from the memory is performed,
By executing the third process by the data acquisition unit and the fourth process by the hash calculation unit up to the last data record stored in the source database, the hash calculation unit For each data item in the source database, execute a fifth process for obtaining the sum of hash values,
The data acquisition means and the hash calculation means execute the first to fifth processes on the duplicate database instead of the source database, thereby for each data item in the duplicate database. , Execute a sixth process for obtaining the sum of hash values,
The comparison means compares the sum of the hash values acquired in the fifth process with the sum of the hash values acquired in the sixth process for each same data item, and calculates the sum of the hash values. If there is a data item that does not match, it is determined that the data stored in the source database and the data stored in the duplicate database do not match, and the sum of hash values does not match If there is no item, execute a seventh process for determining that the data stored in the source database matches the data stored in the duplicate database ;
Data stored in the source database while the data acquisition unit performs acquisition of data records from the source database and acquisition of data records from the duplicate database in any order. A system capable of confirming the consistency between a record and a data record stored in the duplicate database . Data stored in a source database that stores a large number of data records each consisting of a plurality of data items each having data, and stored in a replica database for storing a copy of the data stored in the source database A method for confirming whether or not the data is consistent,
A first process of acquiring any one of the data records stored in the source database from the source database and storing it in a memory;
For each data record stored in the memory, a hash value obtained by applying a predetermined algorithm to each data corresponding to each data item of the data record is obtained. Storing a hash value in association with each corresponding data item in the memory, and then deleting the data record from the memory to reduce the memory usage ;
A third process of acquiring any one unrecorded data record from the source database and storing it in the memory;
For each data record stored in the memory, a hash value obtained by applying the algorithm to each data corresponding to each data item of the data record is obtained, and each obtained hash value is obtained. The hash value is updated by an addition method for each data record so as to update each stored hash value by adding to each stored hash value in association with the corresponding data item. A fourth process for deleting the data record from the memory in order to reduce the amount of the memory used .
By executing the third process and the fourth process up to the last data record stored in the source database, a sum of hash values is obtained for each data item in the source database. A fifth process of acquiring
A sixth process for obtaining a sum of hash values for each data item in the duplicate database by executing the first to fifth processes on the duplicate database instead of the source database. When,
A data item in which the sum of hash values acquired in the fifth process and the sum of hash values acquired in the sixth process are compared for each same data item, and the sum of hash values does not match. If there is no data item in which the data stored in the source database and the data stored in the duplicate database do not match and the sum of hash values does not match includes a data stored in the source database, seventh processing <br/> viewed free determines that the data stored in the replicated database is matched,
The data record stored in the source database and the replica are obtained while performing acquisition of the data record from the source database and acquisition of the data record from the duplicate database in an arbitrary order. A method that allows you to check the consistency of data records stored in a database .

Images