JP2014041501A - Fast reading method for batch processing target data and batch management system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which, in a conventional batch processing system targeting a large amount of data, it takes time to transfer batch processing target data stored in a DB to a batch processing cluster at a time, so that the whole batch processing time becomes long.SOLUTION: Therefore, the present invention employs a method for allowing a batch management system to extract update information on data stored in a database server a plurality of times during a batch execution cycle, to store and manage update data in a period from the last update processing to the current update processing on the basis of the update information, and to sequentially transfer the update data to a distributed file system.

Description

  The present invention relates to a technique for speeding up reading processing of batch processing target data into a batch processing system.

  In recent years, there has been an increasing demand for big data storage, batch processing, and analysis technologies. This is because a cluster configuration with a plurality of general-purpose servers is possible, and data archiving and parallel processing are realized at low cost. Currently, this type of system is adopted by many companies.

  By the way, the processing work executed by the enterprise system includes a processing work for batch processing by transferring data generated and handled by a business system such as a core system from a database on the business system side to a database on the back end side.

  In the present specification, a system that executes batch processing on the back end side is referred to as a “batch processing system”. When the batch processing system reads data from the database on the business system side, the batch processing system executes the set batch process and saves the batch processing result on the database in its own system. Currently, efforts are being made to reduce the time required for batch processing and improve the efficiency of operations.

  For example, Patent Document 1 proposes a technique for reducing the number of times of data I / O with respect to an external storage medium in order to reduce a load when data is input / output. This technology reduces the number of I / Os to an external storage medium by reusing data loaded in memory when multiple search requests for a database are accepted. We are trying to improve performance and batch processing performance.

Japanese Patent Laid-Open No. 08-6829

  However, the invention described in Patent Document 1 requires that a large amount of data be cached in a memory at the time of batch processing. The batch method that consumes a large amount of memory in this way increases the cost of the system. Further, as a real problem, there is a possibility that not all data to be batched can be written to the memory in the batch processing system.

  Therefore, the inventor paid attention to a parallel batch processing technique using a general-purpose PC cluster as a technique capable of executing batch processing at high speed while being low in cost. However, at present, even if a batch processing system is configured simply by applying open source software such as Hadoop / HDFS, it will not achieve high speed batch processing, and the reliability of system operation is also guaranteed. I can't.

  Accordingly, the inventor has come to realize that a mechanism capable of reading data at high speed is necessary for a parallel batch processing system using a general-purpose PC cluster.

  The inventor proposes a method for transferring data from a database server as a data source to a distributed file system according to the following procedure.

  That is, during the batch execution cycle, the batch management system extracts the update information of the data stored in the database server multiple times, and the batch management system performs the current update process based on the update information. We propose a method for executing processing for storing and managing update data up to the update processing, and processing for sequentially transferring update data to the distributed file system by the batch management system.

  According to the present invention, the update status (update data) in the database server as a data source can be regularly reflected in the distributed file system during the batch execution cycle. As a result, the amount of data transferred to the distributed file system at the start of the execution of batch processing can be reduced, and the time required for batch processing can be greatly reduced. Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

The figure which shows the whole structure of the system which concerns on an Example. The figure which shows the whole process sequence of the batch process which concerns on an Example. The figure explaining the outline | summary regarding the hash value management of a table. The figure which shows the table example of table difference management DB. The figure which shows the example of a table of hash value management DB. The flowchart explaining a difference data extraction process procedure. The flowchart explaining a batch processing procedure.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment of the present invention is not limited to the embodiments described later, and various modifications are possible within the scope of the technical idea.

[Example]
(System configuration)
FIG. 1 shows the overall configuration of a system assumed in this embodiment. The user terminal 101 is connected to the business server 103 via the LAN 102. The user uses the user terminal 101 to access the business service 106. Here, the business server 103 has a computer as a basic configuration, and includes an arithmetic device, a large-scale storage device such as a hard disk, a communication interface, and a peripheral device. The business service 106 is realized through program processing by a computer.

  A batch management server 104 that manages parallel batch processing executed in the background and a batch processing cluster 115 that actually executes parallel batch processing are connected to the business server 103 via the LAN 102. Note that the batch processing cluster 115 includes a plurality of batch processing servers 105. In this specification, the batch management server 104 alone or a system configured by the batch management server 104 and the batch processing cluster 115 is referred to as a “batch management system”.

  The storage server of the business server 103 stores a business service DB 107 and its transaction log 108. The business service DB 107 stores data used by the business service. The stored data here becomes the target of batch processing. The business service DB 107 is operated in a mode for outputting and saving the transaction log 108. The transaction log 108 is stored in a public directory of the business server 103 and can be downloaded from another server permitted to access.

  The batch management server 104 also has a computer as a basic configuration, and includes a storage device such as a computing device and a hard disk, a communication interface, and peripheral devices. In the batch management server 104, a batch management service 109 is operating. The batch management service 109 is realized through program processing by a computer.

  The batch management service 109 manages and controls batch processing in the entire system, for example, batch processing target data in the business service DB 107 (data related to a specific column of a table subject to batch processing) and data at the previous access Manage updates to. The storage device of the batch management server 104 stores a table difference management DB 110 and a hash value management DB 111.

  The batch processing server 105 constituting the batch processing cluster 115 also has a computer as a basic configuration. Each batch processing server 105 has a batch application 112 and a distributed processing framework 113 installed therein. The distributed processing framework 113 is a program that executes a coordinated data store and parallel processing between cluster nodes, and also controls the execution of the batch application 112. Further, the batch processing server 105 includes a secondary storage device 114. The secondary storage device 114 is configured by a storage device such as a hard disk device.

  Here, the distributed processing framework 113 serves as a base for configuring a distributed file system for sharing data between cluster nodes. An example of a distributed processing framework is Apache Hadoop. Hadoop can store data on HDFS, a distributed file system, and perform parallel distributed processing with the MapReduce framework. In this embodiment, Hadoop is assumed as the distributed processing framework, but other distributed processing frameworks having a similar mechanism may be used.

(Batch processing flow)
FIG. 2 explains the flow of the batch processing flow executed in the system shown in FIG. FIG. 2 shows the flow of processing operations among the business server 103, the batch management server 104, and the batch processing cluster 115.

  In the business server 103, the transaction log 108 is output at any time when access to the business service DB 107 occurs.

  In the batch management server 104, the batch management service 109 periodically accesses the business server 103 and downloads the transaction log 108. In this embodiment, the transaction log 108 is periodically downloaded regardless of the generation timing of the transaction log 108. However, the batch management server 104 may execute the download of the transaction log 108 with the occurrence of the transaction log 108 as a trigger.

  In the batch management service 109, it is defined in advance which table and column data are fetched for batch processing in the table group constituting the business service DB 107. Based on this transaction log 108, the batch management service 109 detects the update part from the previous time, and updates the data in the table difference management DB 110 that manages only the column data of the specified table.

  In the case of the present embodiment, the batch management service 109 calculates the hash value of the data of the entry group including the update part and registers it in the hash value management DB 111. In this embodiment, the hash value is calculated every time the difference entry is updated. Here, the reason for registering and managing the hash value of the data of the entry group including the update part is that when batch processing is actually executed on the distributed processing framework, the batch processing input data is stored on the actual business service DB 107. This is because it is possible to efficiently confirm whether or not the data matches the data existing in.

  This calculation of the hash value is effective when reliability is not ensured in the distributed file system provided by the distributed processing framework. For example, open source software such as Hadoop / HDFS is difficult to use at present as a base for batch processing. However, if the consistency check operation described later is adopted, the reliability of the distributed file system configured by open source software such as Hadoop / HDFS can be raised to the same level as the conventional system.

  The batch management service 109 sequentially transfers only the updated entries to the batch processing cluster 115. Thus, the transferred data is only the updated entry. For this reason, the amount of data transferred to the batch processing cluster 115 is much smaller than the amount of data in the business service DB 107. The updated entry data is also transferred sequentially during the batch execution cycle in the batch processing cluster 115. The batch processing cluster 115 reflects the sequentially received data on the batch processing input data on the distributed file system configured by the distributed processing framework.

  As described above, this embodiment employs a method of sequentially updating only the difference data. By adopting this method, it is not necessary to transfer a large amount of data at a time when the batch processing cluster 115 actually executes batch processing. For this reason, compared with the conventional system, the data reading time at the start of the batch is shortened. As a result, the time required from the start to the end of batch processing can be greatly reduced as compared with the conventional system.

  Prior to the start of batch processing, the batch processing cluster 115 confirms the consistency of batch processing input data with the batch management server 104 (specifically, the hash value calculated from the batch processing input data and The hash value registered in the hash value management DB 111 is collated), and batch processing is executed when the consistency is confirmed.

  The batch processing cluster 115 calculates the total value after storing the batch processing and stores it in the storage device. The batch processing cluster 115 transfers the calculated total value as the batch processing output data 202 to the business service DB 107 of the business server 103. The batch processing cluster 115 reflects the batch processing result in the business service DB 107 based on the batch processing output data 202.

(Hash value management)
Here, a method for managing the hash value of the table will be described with reference to FIG. The table shown in FIG. 3 is data stored in the table difference management DB 110. When the table data is given, the batch management service 109 extracts the entry group having the designated number of rows from the top, calculates a hash value for a value obtained by combining the individual values, and calculates the hash value as the hash value. Register in the management DB 111.

  For example, in the example of FIG. 3, the batch management service 109 extracts data by 10 rows, and for the first 10 rows, “001-00001”, “Ichiro Sato”, “34,000”, “001-00002” "Yamada Hanako", "8,200", ..., "001-00010", "Taro Suzuki", "102,100" are simply combined as binary values to calculate one binary value, and for that binary value Calculate and register the hash value h1. Similarly, the batch management service 109 calculates and registers the hash value h2 for the data from the next 11th line to the 20th line. Similarly, the batch management service 109 calculates a hash value and registers it in the hash value management DB 111.

  Here, the batch management service 109 only needs to recalculate the hash value only for the data portion including the updated entry (data for 10 rows in the embodiment) when the difference entry is updated. It is not necessary to calculate hash values for all data. With this calculation method, it is possible to realize a high-speed hash value calculation process.

  Here, it is desirable to consider the block size (64 MB by default) managed by HDFS for the number of specified rows when dividing table data. For example, the specified number of rows is set so that a constant multiple of the data size of the specified number of rows matches the block size of HDFS. By setting the specified number of rows in this way, it is possible to avoid the presence of table data with the specified number of rows straddling the block file. As a result, efficient reading and writing of batch processing input data on HDFS can be realized.

(Table structure example)
Here, a table example of the table difference management DB 110 and the hash value management DB 111 managed by the batch management server 104 will be described.

  FIG. 4 shows a table example of the table difference management DB 110. FIG. 4 is a diagram assuming a DB related to member information on an EC (electronic commerce) site. The business service DB 107 on the business server 103 side stores a master table of member information.

  The member number 401, name 402, and monthly purchase amount 403 are attributes required for batch processing in the master table stored in the business service DB 107. The table difference management DB 110 holds a partial table consisting of these attributes. By adopting this structure, data update time and data amount in the table difference management DB 110 can be reduced. The table difference management DB 110 also holds an update flag 404. The update flag 404 is an attribute used to identify an entry that has been updated between the previous data extraction from the business service DB 107 and the current data extraction. A value “1” is set for an entry that has been updated, and a value “0” is set for an entry that has not been updated. The batch management service 109 determines whether to recalculate the hash value based on the value of the update flag 104.

  FIG. 5 shows a table example of the hash value management DB 111. In this embodiment, the table attribute values include a table ID 501, a start entry ID 502, an end entry ID 503, and a hash value 504. A table ID 501 indicates the ID of a table stored in the table difference management DB 110. In this embodiment, the table ID of the table shown in FIG. 4 is represented by “T1”. The start entry ID 502 and the end entry ID 503 indicate the key values of the first and last entries of the entry group for calculating the hash value. The main key of the table T1 is a member number, and this member number is used as a base, and a start entry ID and an end entry ID are defined. The hash value 504 stores the hash value calculated by the method described with reference to FIG. 3 for the specified entry group.

(Differential data extraction processing procedure)
FIG. 6 shows a flowchart of the difference data extraction processing procedure executed in the batch management service 109. The batch management service 109 acquires the transaction log 108 output from the business service DB 107 on the business server 103 (step 601).

  Next, the batch management service 109 extracts the entry data updated in the business service DB 107 based on the acquired transaction log (step 602).

  When the extracted data includes data corresponding to the attribute value of the batch processing table stored in the table difference management DB 110, the batch management service 109 reflects the value of the update entry in the table difference management DB 110. (Step 603). At the time of reflecting this data, the batch management service 109 sets the value of the update flag 404 to “1”.

  When the processing of reflecting all the update entry values in the table difference management DB 110 is completed, the batch management service 109 searches for the update entry with reference to the value of the update flag 404 and calculates the hash value of the designated entry group including the update entry. (Step 604). When the hash value is calculated, the batch management service 109 registers the calculated hash value in the hash value management DB 111 (step 605), and then reflects the update data in the batch processing input data on the distributed file system (step 605). 606).

  This series of processes is periodically executed by the batch management service 109. Thereby, the update data of the business service DB 107 is sequentially reflected in the batch processing input data even during the batch execution cycle. As a result, when batch processing is executed, the entire batch processing can be shortened without causing data migration at a time.

(Batch processing procedure)
FIG. 7 shows a flowchart of a batch processing procedure executed in the batch processing cluster 115. When batch processing is started, the batch processing cluster 115 first checks the consistency of the batch processing input data (step 701). Specifically, the batch processing cluster 115 calculates a hash value for each block data group and compares it with the corresponding value stored in the hash value management DB 111. Here, the block data is partial data of batch processing input data, and is data corresponding to entry data for a specified number of rows.

  Next, the batch processing cluster 115 checks whether there is a block data group whose hash values do not match (step 702). When there is a block data group that does not match the hash value (Yes in step 702), the batch processing cluster 115 extracts the mismatched portion from the table difference management DB 110 to correct the batch input data (step 703), and then Then, batch processing is executed (step 704). On the other hand, if there is no block data whose hash values do not match (negative result in step 702), the batch processing cluster 115 determines that consistency has been confirmed and executes the actual batch processing (step 704).

  Finally, the batch processing cluster 115 updates the business service DB 107 based on the batch processing output data stored on the distributed file system (step 705).

(Summary)
As described above, the batch management server 104 periodically extracts update data of the business service DB 107 as a data source and reflects it in the table difference management DB 110 during the batch execution cycle. This is reflected in the distributed file system of the processing cluster 115. As a result, the amount of data transferred to the distributed file system at the start of the execution of batch processing can be reduced, and the time required for batch processing can be greatly reduced.

  In the case of this embodiment, the batch processing cluster 115 calculates a hash value for the batch processing input data before starting the batch processing, compares it with the hash value managed by the batch management server 104, and if there is a problem. Performs correction processing in advance. Thereby, even when there is a problem in the reliability of the distributed file system constituting the batch processing cluster 115, the reliability of the batch processing input data can be guaranteed.

  As described above, it is possible to realize an inexpensive batch processing system capable of accurately and rapidly batching database data in a business system.

[Other embodiments]
The invention proposed in this specification is not limited to the above-described embodiments, and various modifications are conceivable. For example, the above-described embodiments have been described in detail in order to explain the invention in an easy-to-understand manner, and it is not always necessary to have all the configurations and functions described. In addition, a known configuration can be added to the configuration of the embodiment, or a part of the configuration of the embodiment can be deleted or replaced.

  Moreover, you may implement | achieve some or all of each structure, a function, a process part, a process means, etc. which were mentioned above as an integrated circuit or other hardware, for example. Further, although cases have been described with the above embodiments where functions are realized through the execution of programs by computers constituting each server, each function may be realized through hardware resources such as a module board or an ASIC.

  Control lines and information lines indicate what is considered necessary for the description, and do not represent all control lines and information lines necessary for the product. In practice, it can be considered that almost all components are connected to each other.

  DESCRIPTION OF SYMBOLS 101 ... User terminal, 102 ... LAN, 103 ... Business server, 104 ... Batch management server, 105 ... Batch processing server, 106 ... Business service, 107 ... Business service DB, 108 ... Transaction log, 109 ... Batch management service, 110 ... Table difference management DB, 111 ... Hash value management DB, 112 ... Batch application, 113 ... Distributed processing framework, 114 ... Secondary storage device, 115 ... Batch processing cluster, 201 ... Batch processing input data, 202 ... Batch processing output Data: 401 ... Member number, 402 ... Name, 403 ... Monthly purchase amount, 404 ... Update flag, 501 ... Table ID, 502 ... Start entry ID, 503 ... End entry ID, 504 ... Hash value.

Claims (8)

In a method for high-speed reading of data subject to batch processing by a batch management system connected to a database server,
A process in which a batch management system extracts update information of data stored in the database server a plurality of times during a batch execution cycle;
The batch management system, based on the update information, a process for storing and managing update data between the last update process and the current update process;
The batch management system includes a process of sequentially transferring the update data to a distributed file system. In the high-speed reading method of the batch processing target data according to claim 1,
The batch management system includes a process for storing and managing a hash value of the update data. In the batch processing target data high-speed reading method according to claim 2,
A process in which the distributed file system updates batch process input data based on the update data;
Before starting the batch process, the distributed file system checks the consistency by comparing the batch process input data and the hash value;
A process in which the distributed file system executes a batch process on the batch process input data whose consistency has been confirmed;
The distributed file system includes a process of reflecting data after batch processing as batch processing output data to the database server. In the high-speed reading method of the batch processing target data according to claim 1,
The batch management system acquires a transaction log output from a database constituting the database server as the update information. A method for reading batch processing target data at high speed. In the high-speed reading method of the batch processing target data according to claim 1,
The batch management system extracts the update data collectively for each specified number of entries, combines the values of the extracted entries, calculates a first hash value, stores and updates,
The distributed file system extracts a data portion corresponding to the specified number of entries from the batch processing input data, calculates a second hash value, and collates with the first hash value. High-speed reading method of processing target data. The batch processing target data high-speed reading method according to claim 5,
The data size of the specified number of entries is set so as to coincide with the file size in the batch processing. The batch processing target data high-speed reading method according to claim 5,
If the first hash value and the second hash value do not match, the distributed file system acquires an entry group corresponding to the matching result from the batch management system and reflects it in the batch processing input data, A high-speed method for reading data subject to batch processing, wherein batch processing is executed after consistency is ensured. A batch management system connected to a database server,
A database for storing and managing update data from the previous update process to the current update process every time update information of the data stored in the database server is extracted during the batch execution cycle;
A batch management system comprising: a processing unit that sequentially transfers the update data to a distributed file system that executes batch processing.

Images