JP2012123680A - Distributed database management system and distributed database management method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To quicken data search processing in a distributed database system.SOLUTION: A distributed database management system includes a front server 21 configured to transfer schemer information showing the distribution destination of table data to backend servers 22, 23, and 24, and to, when there exists item data which are not included in a connection key column stored in each of the backend servers 22, 24, and 24, complementarily store the item data as duplicate data in the corresponding backend servers. Each of the backend servers 22, 23 and 24 is configured to, when a search request is transmitted, extract column data for search which becomes necessary with the search content from the connection key column by connecting column data stored so as to be distributed in the backend servers, and to generate an intermediate search result by transmitting search column data for connection to the other servers in which data for connection corresponding to the search column data are distributed.

Description

  The present invention relates to a distributed database management system that manages data stored in different databases.

2. Description of the Related Art A memory database management system that performs calculation processing such as data retrieval after all data handled in a database is expanded on a memory is used.
In this memory database management system, in batch applications that process a large amount of data at once or in a system that creates a data mart from a large amount of data, high-speed processing as a whole system can be achieved by speeding up aggregation and join processing. Can be realized.

On the other hand, in a disk-based database system, for example, when processing a large amount of data such as tens of millions, this is possible by performing the processing during a system stop time such as at night.
However, when a large amount of data processing contents include complicated combination processing and totalization processing, the processing may not be completed within the system stop time.
For this reason, by incorporating the above-mentioned memory data management system into a disk-based database management system, a method for constructing a stable system and shortening the processing time has come to be used.

In the memory database management system, the calculation target data is read into the memory which is a semiconductor storage device, and the calculation process is performed. Therefore, the amount of data handled in the system and the amount of primary data in the middle of the calculation is provided in the system. When the memory capacity is exceeded, a virtual memory area on the disk is generally used.
However, the I / O data transfer speed on the disk is much slower than the data transfer speed on the memory, so that if the amount of data exceeds the memory capacity, the database processing performance (speed) is significantly degraded. Therefore, when processing a large amount of data that exceeds the amount of installed memory, there is a disadvantage that the memory database management system cannot be incorporated into the disk-based database system.

  For example, if a server with hundreds of gigabytes of memory installed on a server is used to process table data with hundreds of bytes of data per case and several tens to billions of data, all the data is stored on the memory of one server. Processing data cannot be expanded.

  In addition, when it is desired to handle Web access log data and sales record data in units of receipts and products, it is assumed that the amount of data (number of data items) will be exceeded. When the aggregation process is performed using the database or the virtual memory area, the processing time becomes enormous, so that there may be a disadvantage that the system requirement is not met.

In addition, a memory database server management system that performs distributed processing using a plurality of memory database servers in many cases cannot be applied as an alternative to a disk-based distributed database management system that uses a storage such as a normal hard disk. .
For example, in an ordinary distributed database management system, horizontal partitioning that divides table data in units of rows is used. In order to perform query processing across multiple servers, particularly join processing such as semi-join method, each database server It is necessary to perform processing while performing communication for data transfer between them.

  However, in the case of a memory database, the processing by this large amount of communication takes a long time compared to the arithmetic processing on the memory, so the high speed characteristic peculiar to the memory database is greatly impaired, and high speed processing performance is required. There is a disadvantage that it is not suitable for the system.

  As a related technique for this, a system is disclosed in which real data is aggregated as a table in an index format in one front memory database server, and other real data is distributed to a plurality of servers (Patent Document 1). In this case, since aggregation and join processing are performed on the front memory database server, it is possible to reduce the amount of communication between servers that is necessary for normal horizontal distribution.

  As another related technique, tables including item sequences having the same meaning from a plurality of databases having different data update timings are merged in advance and held as one database, and the latest update data is aggregated into the one database. Thus, a system is disclosed in which unnecessary data and databases are excluded from search targets, thereby realizing database operational efficiency and speedup (Patent Document 2).

JP 2010-085568 A JP 2006-268783 A

  However, in the related technique described in Patent Document 1, it is necessary to cause the front memory database server to have only the index format. In this case, although the amount of data is compressed compared to normal table data, all the table data is held in one front memory database server, so that several tens of memories are required. With such a large amount of data, there may be a disadvantage that the upper limit of the memory of the front memory server is exceeded.

In the related technology described in Patent Document 2, a set of newly generated row-by-row data and column data is preferentially updated, and a plurality of databases are integrated, so that one search request from a user is obtained. Table units can be handled as search targets.
However, when performing join processing on table data stored in different databases, the amount of data transferred between databases and the amount of data to be replicated becomes enormous. There is a disadvantage that the table is limited to a relatively small table, and further, there is a problem that communication traffic becomes enormous and the speeding up of the joining process is suppressed.

[Object of invention]
The present invention provides a distributed database management system and a distributed database management method for improving the inconvenience of the related technology and performing reference, aggregation, and search processing for data on different servers more quickly. For that purpose.

  To achieve the above object, the distributed database management system according to the present invention distributes preset table data as column data to a plurality of different database servers, and responds to a search request from an external terminal. A distributed database management system comprising a request data processing device for obtaining a search result based on each database server, wherein the request data processing device provides schema information indicating a distribution destination of the column data to each database server. When there is item data that is not stored in the combination key column stored in each database server and the schema information copy transfer unit that transfers the item data, the item data is complemented to the back-end server as copy data. Each of the database servers. When the search request is sent from the request data processing device, the search request is required from the join key sequence by combining the column data distributed in the self database server based on the join key sequence. A data combination extraction unit that extracts the column data for search to be performed, a server identification unit for combination that identifies another server to which the data for combination corresponding to the extracted search column data is distributed based on the schema information, A search result combination generation unit configured to generate the search result by transmitting the search string data to the specified database server and combining it with the combination data.

  Also, in the distributed database management method according to the present invention, a request data processing device that distributes preset table data as column data to a plurality of different database servers is based on the search request in response to a search request from an external terminal. A distributed database management method for acquiring a search result from each database server, wherein the request data processing device transfers schema information indicating a distribution destination of the column data to each database server, and transmits the schema information to each database server. When there is item data that is not stored in the stored combination key column, this item data is stored in a complementary manner to the back-end server as replicated data, and each database server receives from the request data processing device When a search request is sent, the self-database is based on the join key column. By combining with the data distributed in the server, the search column data required for the search request is extracted from the combination key column, and the combination data corresponding to the extracted search column data is The distributed server is specified based on the schema information, and the search result is generated by transmitting the search string data to the specified database server and combining it with the combining data. .

  The present invention, as described above, when there is item data that is not stored in the joint key string distributed and stored in each database server, the request data processing device that complementarily stores this item data as duplicate data, By providing a database server that extracts column data for search required for a search request from a join key column by performing a join between column data distributed in the self database server based on the join key column, It is possible to provide a distributed database management system and a distributed database management method that can more quickly perform reference, aggregation, and search processing for data on different database servers.

1 is a schematic block diagram showing an embodiment of a distributed database management system according to an embodiment of the present invention. FIG. 2A is an explanatory diagram showing an example of the A handling product table in the distributed database management system. FIG. 2B is an explanatory diagram showing an example of a sales table in the distributed database management system. FIG. 2C is an explanatory diagram showing an example of a repair sales summary table in the distributed database management system. It is explanatory drawing which shows an example of the arrangement | positioning condition of the column data in the distributed database management system shown in FIG. It is explanatory drawing which shows an example of the arrangement | positioning condition of the column data in the distributed database management system shown in FIG. It is a flowchart which shows the operation | movement process step which transfers the schema information in the distributed database management system shown in FIG. It is a flowchart which shows the operation | movement process step which arrange | positions table data with respect to the back end server in the distributed database management system shown in FIG. It is a flowchart which shows the operation | movement process step of the joint process between the back end servers in the distributed database management system shown in FIG. It is a flowchart which shows the operation | movement process step of the search process in the back end server in the distributed database management system shown in FIG.

[Embodiment]
A distributed database management system 100 according to an embodiment of the present invention includes a database client (external terminal) 1 that sends a request or a command generated based on an input from a user, and a database search according to a search request from the database client 1 A distributed memory database management system 2 that performs processing and generates search results is provided.

  The distributed memory database management system 2 includes back-end memory database servers (corresponding to database servers) 22, 23, and 24 that store data constituting pre-registered table data as actual data in column units, and the above table data. The data is divided into column data (column units) and stored in each of the back-end servers 22, 23, 24, and when a search request is sent from the database client 1, the search is performed from each of the back-end servers 22, 23, 24 A front memory database server (request data processing device) 21 is provided for acquiring results and generating search result table data in which the results are integrated.

  As described above, the back-end memory database server (hereinafter referred to as “back-end server”) 22, 23, 24 returns the search results of each of them to the front server (hereinafter referred to as “front server”) 21 as an intermediate search result. To do. As a result, the query execution unit 211 of the front server 21 generates a search result by merging the intermediate search results sent from the back-end servers 22, 23, and 24, and returns this search result to the database client 1. To do.

  In the distributed memory database management system according to this embodiment, three back-end memory database servers 22, 23, and 24 are installed in parallel to the front server 21 via the internal network 4. However, one or more back-end memory database servers are sufficient.

  As shown in FIG. 1, the database client 1 and the front memory database server 21 are connected via a communication line. Similarly, the front server 21 and the back-end servers 22, 23, 24 are connected by the internal network 3. Has been. The communication protocol via the communication line and the communication protocol between servers are arbitrary.

  The front memory database server 21 analyzes the search inquiry (search request) and message contents sent from the database client 1, and also makes an inquiry and operation request (operation request) to each back server 22, 23, 24 based on the analysis result. ), A schema information management unit 212 for storing schema information, and a temporary storage memory area 213 as a temporary storage memory for temporarily storing the sent intermediate results.

  Also, the query execution unit 211 distributes and stores (partitions) the table data to the back database servers 22, 23, and 24 via the internal network 4 based on preset schema information.

  Further, the query execution unit 211 transfers the schema information managed by the schema information management unit 212 to each back database server 22, 23, 24 (schema information transfer function). Thereby, the schema information stored in each of the schema information management units 222, 232, and 242 is common (that is, the same information is managed as the schema information).

In response to an inquiry from the query execution unit 211, the schema information management unit 212 instructs the back-end servers (22, 23, 24) to be inquired based on previously stored schema information (inquiry destination designation function).
As a result, the query execution unit 211 can make inquiries and operation requests to the back-end database servers (back-end servers) 22, 23, and 24 based on instructions from the schema information management unit.

  The temporary storage memory area 213 has an intermediate result storage function for storing intermediate results sent from the back-end servers (22, 23, 24). In addition, it is assumed that schema information to be managed by the schema information management unit 212 is also stored in the data storage memory area 213.

Each of the back-end servers 22, 23, and 24 is a computer that functions as a database, and is connected to the front server 21 via the internal network 4, and in response to a request from the query execution unit 211, a self-storage memory Search the database set in the area.
Further, since the back-end servers 22, 23, and 24 have the same internal configuration, the internal configuration of the back-end server 22 will be described here. As for this internal configuration, it is assumed that the back-end servers 23 and 24 have the same corresponding configuration content.

  The back-end server 22 analyzes the schema information in response to an inquiry from the local query execution unit 221 and a local query management unit 221 that performs data search processing in its own server in response to a search processing request from the front server 21. It has a configuration including a schema information management unit 222 and a database storage memory area 223 that holds table data generated based on pre-input schema information.

  The server includes a query execution unit 211, a schema information management unit 212, and a data storage memory area 213. The server receives a query from the database client and returns a search result to the database client.

The query execution unit 211 confirms the contents of the inquiry and operation issued from the database client, and makes an inquiry and operation to the back-end database server (22, 23, 24).
However, the “table definition” is managed by the schema information management unit 212, and the back-end database server can be inquired by inquiring the information to the schema information management unit 212.

  The schema information is also transferred to each back-end server 22, 23, 24, and the schema information management unit 211 of the front server 21 and the schema information management unit on the back-end server (22, 23, 24) share the same information. It shall be managed.

  In addition, the query execution unit 211 stores the intermediate results returned from the back-end servers 22, 23, and 24 in the temporary storage memory area 213, and merges these intermediate results at the stage when all the operation results are collected. Return to client 1.

  The schema information management unit 212 stores and manages pre-input schema information. This schema information includes (1) table definition information (particularly, join key designation information indicating a join key used in the join process), and (2) which back-end memory database server (22) the data (column data) is. , 23, 24) column data storage location information indicating whether or not (stored), (3) partitioning condition information indicating a partitioning rule such as range partition or hash partition ("partitioning rule") ”).

  The data storage memory area 213 stores intermediate results returned from the back-end servers 21, 22, and 23 via the query execution unit 211 on the memory. Each schema information to be managed by the schema information management unit 212 is also stored in this data storage memory area.

  In response to the inquiry or request sent from the front server 21 (query execution unit 211), the column data management unit 221 searches for data stored in the column data storage memory area 223 in advance, and updates the stored data. A search processing function is provided.

  It is assumed that the column data storage memory area 223 also stores and holds schema information handled by the schema information management unit 222. The column data storage memory area 223 stores the same schema information as the schema information stored in the front server 21 and the other back-end servers 23 and 24.

  In addition, the column data management unit 221 provides a local join processing function for performing join processing on column data stored in its own database, and column data for join processing to other back-end servers (23, 24). It has a communication connection processing function for transmitting and combining with column data stored in other back-end servers.

  Here, the operation of transferring the schema information in the database client 1, the front memory database server 21, and the back end memory database servers 22, 23, and 24 will be described based on the flowchart of FIG.

First, a database administrator, a user, or the like uses the database client 1 to create schema information (definition) necessary for building a distributed database and inputs this schema information to the distributed database management system 2.
Further, the database client 1 may be configured to send a command for requesting the distributed memory database management system 2 to store the actual data in a distributed manner (step S51: FIG. 5).

  Here, the schema information includes a command input by the user, a table definition that defines attributes of table information to be stored, and a range of column data constituting the table to which back-end server 22, 23, 24. The partitioning rule that specifies whether to store the data, the join key specification information that indicates the columns used during the database join process that are mutually executed between the back-end servers, the reference table, or the table to be processed Contains table information to indicate.

  The database client 1 inputs the above-described schema information set by the user to the query execution unit 211 (step S51).

The query execution unit 211 analyzes the contents of the schema information input from the database client 1 (step S51) and passes the schema information to the schema information management unit 212 (step S53).
The schema information management unit 212 stores this schema information in 213 (step S54), and notifies the query execution unit 211 of the completion of setting (step S55).

Next, the query execution unit 211 duplicates the join key designation information indicating the join key column and the schema information including the partitioning rule, and for each back-end server 22, 23, 24 that is the storage destination of the column data. Then, the schema information is transferred (to step S56).
As a result, the common schema information is held in the front server 21 and the back-end servers 22, 23, and 24.

  Here, the schema information management units 221, 231, and 241 are memory areas of the semiconductor storage devices provided in the respective back-end servers 22, 23, and 24, and store and hold the transferred schema information. And

The local query execution units 221, 231, and 241 of the back-end servers 22, 23, and 24 transmit the schema information that is sent to the schema information management units (222, 232, and 242), respectively, and the schema information management units (222, 232). , 242) stores this (step S58).
As a result, the same schema information is set in the back-end servers 22, 23, and 24.

  Next, the local query execution units 221, 231, and 241 notify the query execution unit 211 that the setting of schema information has been completed, and the query execution unit 211 has completed the setting of schema information from all back-end servers. When notified, the database client 1 is notified of the completion of storing schema information (step S60).

  Further, the front server 21 receives actual data such as table information input to the distributed database management system 2 by a database administrator or a user based on the contents of preset schema information. Each of 23 and 24 has a partitioning distribution arrangement function for arranging the processing target table data (table) in a column unit.

The schema information includes a table definition specified by a command from the database client 1, a partitioning rule indicating a distribution rule to a back-end server 22, 23, 24 in a table to be stored, and a join key string at the time of a join operation. Information (join key designation information), target table data, information indicating which table is the reference table, and the like.
The schema information may be input to the query execution unit 211 by a database administrator or user.

The query execution unit 211 analyzes the contents of the schema information stored in the temporary storage memory area 213 and expands the schema information to the schema information management units 222, 232, and 242 of the back-end servers 22, 23, and 24, respectively. To do.
As a result, the schema information having the same content as the schema information stored in the front server 21 is also stored in the memory area of each back-end server 22, 23, 24.

Next, the partitioning distributed arrangement function executed by the query execution unit 211 after the schema information is stored in the schema information management units 222, 232, and 242 will be described in detail.
The operation in the partitioning distribution function is divided into the following first and second stages.

[Partitioning distribution function]
First, in the first stage of the partitioning distributed arrangement function, the query execution unit 211 performs data arrangement on the back-end server based on the table definition and the partitioning rule in the schema information.

At this time, when distributed to each back-end server 22, 23, 24, the query execution unit 211 eliminates (normalizes) and sorts the actual data included in the same actual data string (column). It shall be stored in the state.
As a result, the back-end servers 22, 23, and 24 store the actual data in a state of being sorted exclusively and redundantly, so that even if the data is changed or modified, it is not necessary to recalculate and store the data. The integrity of the recorded data can be maintained.

In addition, the query execution unit 211 stores and stores an “actual data string”, and then generates and stores an index string indicating column data including an index that refers to each data string (index string generation function).
However, if two or more tables stored locally have the same join key according to a preset join definition, they are integrated into one actual data string.
As a result, it is possible to perform the join process locally at each back-end server by referring to the index string without performing the join (end of the first stage).

  When a join definition corresponding to a table to be partitioned is set in advance, each server is not stored by distribution among the data of the join key string in each back-end server 22, 23, 24 (that is, each server For other data, replication is performed from another server, and all data included in the join key column is retained.

As a result, it is possible to omit the operation step of collecting the combination key from each server during the combination process, and to reduce the amount of communication.
Hereinafter, the partitioning distributed arrangement function will be specifically described.

  Here, as table data to be processed, as shown in FIGS. 2A to 2C, an A handling product table (a), a sales table (b), and a repair sales summary table (c) are set. It shall be.

Here, as shown in FIG. 2A, the A handling product table is table data having “product ID”, “product name”, “category”, “manufacturing company”, and “list price” as column data columns. It is.
As shown in FIG. 2B, the sales table is table data having “sales number”, “date”, “product ID”, “number”, and “sales amount” as column data columns. is there.
Further, as shown in FIG. 2C, the repair sales summary table is table data having “year”, “period”, “product ID”, “cumulative number”, and “cumulative sales” as column data strings.

  In addition, as the partitioning rules in the schema information, it is assumed that the following contents are predetermined.

[Partitioning rules]
About A handling product table (a) and sales table (b),
Product ID: 10015 or less to the back-end server 22 Product ID: 10015 or less to the back-end server 23

For the repair sales summary table (c),
Before 2007, go to back-end server 23. After 2008, go to back-end server 24 [End of partitioning rules]

  That is, in the A handling product table and the sales table, when the value of the product ID is 10015 or less, the line item is arranged in the back end server 22 in units of columns. Further, when the value of the product ID is 10016 or more, it indicates that the line item is arranged (range partitioning) in the back-end server 23 in units of columns.

  In the repair sales summary table, if the value of the year is before 2007, the line item is arranged in the back-end server 23 in units of columns. Further, if the value of the year is 2008 or later, it indicates that the line item is arranged (range partitioning) in the back-end server 24 in units of columns.

  Further, it is assumed that the following contents are predetermined as the definition of the combination in the schema information.

[Define Join]
The sales table (b)
Reference source table: A handling product table join key column: Product ID

Repair sales summary table (c)
Reference source table: A handling product table join key column: Product ID

  That is, in the sales table, the reference source table is the A handling product table, and the combined key column is the product ID. In the repair sales summary table, it is assumed that the reference source table is the A handling product table and the combined key column is the product ID.

  The query execution unit 211 of the front server 21 partitions the tables shown in FIGS. 2A to 2C on the basis of the partitioning rule and the join definition in the schema information, and back-end servers 22, 23, 24. Placed (loaded) as real data.

  Here, FIG. 3 shows a state in which the data of the A handling product table, the sales table, and the repair sales summary table partitioned by the query execution unit 211 are loaded as actual data into the back-end servers 22, 23, and 24. 3 shows.

  Here, after the A handling product table, the sales table, and the repair sales summary table are arranged for the back-end servers 22, 23, and 24, the query execution unit 211 sends the back-end servers 23 and 24 to the back-end servers 23 and 24. Thus, the column data that is not arranged in the product ID of the A handling product table that is the combined key column of the reference source is replicated and loaded as a replica column.

  Here, based on the definition that the A handling product table and the repair sales summary table are combined with the product ID, the query execution unit 211 has a product ID that is a combination key column because the object of the partitioning rule is not the product ID. Are replicated from each backend memory server and placed.

  Thereby, all the data included in the combination key string (here, the product ID string (column)) necessary for the combination process is stored in advance for each back-end server. For this reason, it is possible to effectively reduce the amount of data transferred between the back-end servers for the join process, which occurs during the search process in the back-end server.

  On the other hand, since the back-end memory server 22 holds all data for the product ID, which is a combined key column of the A handling product table and the sales table, the query execution unit 211 does not perform replication.

  This is for product IDs whose partitioning rules of the A handling product table and the sales table are combined key columns, and further, the condition (that is, if the value of the product ID is 10015 or less, the back-end server 22 In the case where the value of the product ID is 10015 or less, the same applies to the back-end server 23), and it is not necessary to refer to the combined key for other servers.

  As described above, the present embodiment stores data in the column direction because of the characteristics of the column store database, so when there is data duplication in the column when compared with the case of horizontal division. It is possible to store data in a compressed manner, which makes it possible to effectively suppress the memory usage during data storage.

[Modification]
In the above embodiment, the column data (referred to as “target column data”) corresponding to the items (rows) included in the join key column and subject to aggregation processing and search conditions in the “join definition” in the schema information. May be set to be specified together.

  Thereby, after the tables of the A handling product table, the sales table, and the repair sales summary table are arranged for the back-end servers 22, 23, 24, the query execution unit 211 sets the back-end servers 23, 24 respectively. When data of join key columns that are not arranged is loaded to the back-end servers 23 and 24 as replica columns, not only the join key columns but also the corresponding column data specified in the join definition are replicated together. Shall be loaded.

As a result, in FIG. 3, only the combination key (product ID) is replicated. In this modification, the combination key column is designated as data subject to aggregation processing and search conditions. The “price” column, which is the corresponding column data, is also replicated at the same time.
As a result, data that can be acquired from each back-end server required at the time of combination is stored in the local server in advance, so the amount of communication data between back-end servers can be further reduced, and search processing can be performed more quickly. Can be

[Description of Operation of Embodiment]
Next, the overall operation of the above embodiment will be described.

The front server (request data processing device) 21 transfers schema information indicating the distribution destination of the table data to each of the back-end servers (database servers) 22, 23, and 24 (schema information duplication transfer process), and each back-end server When there is item data that is not stored in the combination key string stored in 22, 23, 24, the item data is stored in a complementary manner in the back-end server as duplicate data (join key data supplement storage step). .
Next, when a search request is sent from the front server (request data processing device) 21, the back-end servers 22, 23, and 24 combine column data stored in a distributed manner in their own back-end servers. The search column data required for the search content is extracted from the combination key column (search column data extraction step), and the combination data corresponding to the extracted search column data is distributed. An intermediate search result is generated by specifying a server, transmitting search string data, and combining it with the data for combination (search result combination acquisition step).

  Here, the execution contents of the schema information copy transfer process, the combined key data supplement storage process, the search column data extraction process, and the search result combination acquisition process are programmed, and the computer provided in the distributed memory database management system 2 It is good also as a structure made to perform.

Next, the operation of this embodiment will be described in detail.
Here, first, the operation of distributing (loading) column data to the back-end servers 22, 23, 24 by the query execution unit 211 will be described based on the flowchart of FIG.

  Here, it is assumed that the column data distribution function in the front server 21 is executed when a message requesting data distribution (loading) from the database client 1 to the back-end servers 22, 23, 24 is input ( Step S71: Start of data loading).

[the first stage]
Here, the query execution unit 211 of the front server 21 analyzes the input table data based on the schema information (step S72).
Here, the schema information management unit 212 confirms the table definition and partitioning rule of the schema information managed in advance (step S73).

Here, the column information divided based on the partitioning rule is transferred and arranged to each of the back-end servers 22, 23, 24 specified by the partitioning rule (step S 74: range partitioning).
It is assumed that how to divide the table data and to which back-end server the divided column information is distributed are defined in advance in the schema information partitioning rule.

  Next, the local query management units 221, 231, and 241 installed in the respective back-end servers 22, 23, and 24 acquire the distributed column data, and the database storage memory area (“ (Referred to as “storage memory”) 223, 233, 243 (step S75).

  Here, each local query management unit 221, 231, 241 saves the sent column data in the storage memory in the form (actual data string) in which the column data is sent as real data, the duplication of items is eliminated, and the items are sorted in ascending order. It shall be.

Further, the local query management units 221, 231, and 241 generate an index string (index format) composed of indexes corresponding to each item (value) of each real data string, and store it in the storage memory in each back-end server. Store.

If there is a common join key column in two or more different tables to be joined based on the join definition included in the schema information stored in each backend server, one actual data column And are arranged in each storage memory.
Thereby, it is possible to perform the joining process by referring to the index string in each back-end server.

  Next, the query execution unit 211 of the front server 21 determines whether or not the arrangement of the column data has been completed (step S76: confirmation of transmission completion). Here, when there is column data that is not arranged among the column data included in the table data, the column information divided based on the partitioning rule is used as the back-end databases 22 and 23 specified by the partitioning rule. , 24 (to step S74).

  On the other hand, if there is no unplaced column data among the column data included in the table data (no data remaining), the process proceeds to the second stage described below (end of the first stage).

  Next, the second stage of the operation in which the front server 21 211 distributes and loads the column data will be described in detail with reference to the flowchart of FIG.

[Second stage]
Here, when the join definition is set in advance in the schema information of the table data to be arranged for the back-end servers 22, 23, and 24, the query execution unit 211 selects each of the data included in the join key column. It is determined whether there is data not arranged in the back-end servers 22, 23, 24.

In a case where it is determined that there is data of the join key string that is not held (that is, there is data missing), the query execution unit 211 returns the back-end server 22, 23, or 24 with the missing placement data of the join key string. In response to this, a copy (replication) of data that is not held is generated and transferred (step S78: replication is notified of deployment).
Thereby, in each backend server 22,23,24, the data of all the lines of a joint key column are held.

  Next, the local query execution unit 221 of the back-end server 22 determines whether or not there is a join key column that is referenced from the tables arranged in the other back-end servers 23 and 24 based on the schema information. If there is a join key column referenced from the tables arranged in the end servers 23 and 24, the target column data is transferred (step S79).

Next, the query execution unit 211 stores all data included in the combination key string (product ID) necessary for the combination processing in the back-end servers 22, 23, 24 for each of the back-end servers 22, 23, 24. It is confirmed whether or not it has been done (steps S80, 81).
When it is confirmed that all data included in the combination key string (product ID) necessary for the combination process is stored in each of the back-end servers 22, 23, and 24, the query execution unit 211 returns the back-end The database client 1 is notified of the end of placement (loading) of the partitioned data with respect to the servers 22, 23, and 24 (step S82).

  The database client 1 obtains the load end notification (step S83), and the distributed memory database management system 2 is set to a search processing request standby state at this point.

[Join processing between back-end servers]
Next, an operation for performing a join process by communicating column data between different back-end servers will be described based on the flowchart of FIG.
Here, it is assumed that the back-end server A is the back-end server 23 and the back-end server B that holds the reference data of the combined key is the back-end server 24 (FIG. 7).

First, the local query execution unit 231 of the back-end server 23 analyzes schema information (step S91).
The schema information management unit 232 confirms that the back-end server 23 has all the data included in the combined key string (here, it is assumed to be the product ID string) (step S92).
Next, the back-end server 23 performs a join process using only the join key string (step S93: execute the join process using only the join key).

  The partitioning rule in the schema information is confirmed, the server (here, the back-end server 24) where the reference source data (table) is located is specified, and the join column data (column data) is joined to the back-end server 24. A command requesting processing (join command) is transmitted (step S94).

  Next, the local query execution unit 241 of the back-end server 24 performs a join process based on the column data and the join command sent (Step S95), and returns join data as a join process result to the back-end server 23 (Step S95). S96: Return combined data)

  Next, the back-end server 23 checks whether or not the combined data has been prepared for all the data included in the combined key string, and completes the process when it is confirmed that the combined data has been prepared without duplication (step S97). ).

In general, a distributed database system uses a semi-join method in which data necessary for joining is transmitted between servers.
However, in the normal semi-join method, a join key string used for joining is extracted from, for example, server A, sent to server B, joined at server B, the result is sent back to server A, and the join is completed at server A. Let

On the other hand, in the present embodiment, at the time of step S94, data necessary for combining is specified from the data included in the combined key string, and only the specified data is transmitted to the server. It is possible to perform the combining process.
In addition, when the target join column corresponds to a partition rule (managed by the schema information management unit), the data (join data) used for further join processing can be further narrowed down (limited). As a result, the amount of communication data can be further reduced.

[Search processing]
Next, the operation (search processing operation) of the distributed memory database management system 2 when there is a data search request (search inquiry) from the database client 1 will be specifically described.

Here, it is assumed that the column data to be searched is partitioned in each back-end server 22, 23, 24 as shown in FIG.
At this time, the following SQL statement is input from the database client 1 to the query execution unit 211, and a search operation in the distributed memory database management system 2 is performed based on the SQL statement.

[SQL sentence]
SELECT Product ID, year, period, cumulative number * price FROM A handling products, repair sales summary handling A handling A products. Product ID = Repair sales summary. Product ID
AND cumulative number * list price> 300,000

First, the query execution unit 211 determines with respect to the schema information management unit 212 which of the back-end servers 22, 23, and 24 the column data specified by the SQL statement is arranged. Inquire.
Here, the schema information management unit 212 specifies that “repair sales summary” and “A handling product” are stored in the back-end servers 23 and 24 based on the schema information, and sends them to the query execution unit 211. Notice.

  The query execution unit 211 issues a search command based on the SQL to the local query execution units 231 and 241 of the back-end servers 23 and 24 based on the notification from the schema information management unit 212.

Hereinafter, each of the back-end servers 23 and 24 performs the same search process according to the search command sent from the query execution unit 211.
Here, the operation content in the back-end server 24 will be described based on the flowchart of FIG. Note that the local query execution unit 241 can refer to the data stored in the database storage memory area 243 via the schema information management unit 242 (FIG. 3).
Here, as in the case described above, the back-end server A is assumed to be the back-end server 23, and the back-end server B that holds the reference data of the combined key is assumed to be the back-end server 24 (FIG. 8).

The local query execution unit 241 performs a join process based on the join condition included in the schema information.
Here, the local query execution unit 241 is based on a join condition that the product ID column in the index column of the repair sales summary table is equal to the product ID column in the index column of the A handling product table (replica column) that is the index column. A combining process is performed (step S103).
As a result, the local query execution unit 241 extracts a set of common values {4, 8}.

  Next, the local query execution unit 241 is a set of corresponding fourth and eighth values of the product ID column, which is an actual data column, based on the set {4, 8} extracted by the join processing in the index column { 10013, 10030}.

  Thus, in this embodiment, the data in each back-end server is not transferred without transferring the index number = 5 (product ID = 1015) of the “repair sales summary table” that is transferred during the joining process in the normal semi-join method. It is possible to perform the combining process by prioritizing and using.

  Here, the local query execution unit 241 does not distribute (store) the “price” data corresponding to the product ID {10013, 10030} as data necessary for the combination in the backend server 24 local (step) S105: When there is not enough data necessary for combining), the corresponding “list price” data is acquired from another back-end server. At this time, the local query execution unit 241 refers to the partitioning rule of the A handling product table via the schema information management unit 242.

The local query execution unit 241 stores the data indicating the list price corresponding to “product ID: 10013” in the back-end memory server 22 based on the partitioning rule of the product ID, and also stores “product ID: 10030”. It is specified that the data indicating the corresponding list price is stored in the back-end server 23.
Here, the local query execution unit 241 transfers the corresponding column data and a processing command for requesting the “list price” data corresponding to the column data to the back-end memory servers 22 and 23 (step S106). .

  Next, in each of the back-end servers 22 and 23, the local query execution units 221 and 231 search for a price value corresponding to the product ID: 10013 or the product ID: 10030 based on the received processing command, and the product ID 10013 or 10030 and a search / acquired list price value are combined (step S107), and the generated combination data is returned to the back-end server 24 (step S108).

  Next, the local query execution unit 241 of the back-end server 24 acquires the combined data sent from each of the back-end servers 22 and 23, and confirms that the combined data that is data for the search request has been collected from the front server 21. The intermediate search result composed of the product ID and the fixed price obtained by combining the combined data sent from the back-end servers 22 and 23 is returned to the query execution unit 211.

  The query execution unit 241 stores the intermediate processing results returned from the back-end servers 23 and 24 in the temporary memory area 213, and merges the intermediate search results when it is confirmed that all the processing results are available. Then, the generated table information is returned to the database client 1 as a final search result (step S109).

  As described above, in this embodiment, the replication data for performing the join process is limited to the data of the join key column, so that each server can perform the join process of data (tables) stored in different databases. Can reduce the amount of memory used.

  In particular, since the upper limit of data is limited in the memory database as compared with the case of the disk, in the memory database in the present embodiment, the server (back-end server) including all the records is limited to the join key column. As a result, the memory area required for data storage on the memory can be reduced.

  Next, in the definition of the join in the schema information, the column data (referred to as “corresponding column data”) specified in the aggregation or search condition specified in the search request corresponding to the item (row) included in the join key column is specified. An operation (search processing operation) in the distributed memory database management system 2 in the case of being performed (in the case of the above modification) will be specifically described.

At this time, as described above, the query execution unit 211 performs not only the join key column but also the corresponding column data specified by the definition of the join when replicating the data to each back-end server 22, 23, 24. Replicate with join key columns.
As a result, the table data in FIGS. 2A to 2C is partitioned (distributed) as column data to the back-end servers 22, 23, and 24 as shown in FIG.

  At this time, a case where the following SQL statement is input from the database client 1 to the query execution unit 211 and a search operation in the distributed memory database management system 2 is performed based on the SQL statement will be described.

[SQL sentence]
SELECT Product ID, year, period, cumulative number * price FROM A handling products, repair sales summary handling A handling A products. Product ID = Repair sales summary. Product ID
AND cumulative number * list price> 300,000

  Here, as described above, the query execution unit 211 sends the schema data management unit 212 the column data specified in the SQL statement to any of the back-end servers 22, 23, and 24. The schema information management unit 212 identifies whether the “repair sales summary” and the “A handling product” are stored in the back-end servers 23 and 24 based on the schema information. The query execution unit 211 is notified.

  The query execution unit 211 issues a search command based on the SQL to the local query execution units 231 and 241 of the back-end servers 23 and 24 based on the notification from the schema information management unit 212.

  Hereinafter, since each back-end server 23 and 24 performs the same search process according to the search command sent from the query execution unit 211, the operation contents in the back-end server 24 are the same as described above. explain.

  Similar to the above (FIG. 8), the local query execution unit 241 performs a join process based on the join condition included in the schema information. Here, the local query execution unit 241 is based on a join condition that the product ID column in the index column of the repair sales summary table is equal to the product ID column in the index column of the A handling product table (replica column) that is the index column. A combination process is performed to extract a set of common values {4, 8}.

Here, after {4, 8} of the index column of the combined key product ID is extracted, the local query execution unit 241 checks the A handling product table (replica table) and stores them in the same row. The “price” index can be similarly determined.
In addition, since the actual data is also stored in the back-end server 24 that is itself, there is no need to acquire data by communicating with other back-end servers in order to acquire “price” data.

  As a result, the back-end server 24 can generate a search result and return it to the query execution unit 211 without performing communication for a combined number with another server (back-end server 23).

  Similarly, in the back-end server 23, the search results can be returned to the query execution unit 211, and the query execution unit 211 merges all the search results returned from each back-end server. It can be returned to the database client 1 as a final search result.

  As described above, in the case of the above modification, it is possible to further reduce the amount of communication between the databases by performing replication by combining the columns necessary for the join, and thus further improve the high speed of the search processing. Can do.

Regarding the above-described embodiment, the main points of the new technical contents are summarized as follows.
In addition, although a part or all of said embodiment is put together as follows as a novel technique, this invention is not necessarily limited to this.

(Appendix 1)
A request data processing device is provided that distributes preset table data as column data to a plurality of different database servers, and obtains a search result based on the search request from each database server in response to a search request from an external terminal. Distributed database management system,
The request data processing device includes:
A schema information replication transfer unit for transferring schema information indicating a distribution destination of the column data to each database server;
When there is item data that is not stored in the combination key column stored in each database server, a combination key data supplement storage unit that complementarily stores the item data as duplicate data in the back-end server; Prepared,
Each of the database servers is
When a search request is sent from the request data processing device, a search required by the search request from the combined key column is performed by combining the column data distributed in the self database server based on the combined key column A data combination extractor for extracting column data;
A server identification unit for coupling that identifies another server to which data for coupling corresponding to the extracted search string data is distributed based on the schema information;
A distributed database management system comprising: a search result combination generation unit that generates the search result by transmitting the search string data to the specified database server and combining the search string data with the combination data.

(Appendix 2)
In the distributed database management system according to attachment 1,
The combination key data complementation storage unit specifies column data corresponding to item data of the combination key column as target column data, which is column data to be searched based on the content of the search request, and the target column data A distributed database management system comprising a target column storage function for storing a target column as the replicated data in the corresponding back-end server.

(Appendix 3)
In the distributed database management system according to attachment 1,
The distributed data base management system, wherein the request data processing device comprises data arrangement management means for redundantly and exclusively normalizing item data included in column data distributed to each database server.

(Appendix 4)
In the distributed database management system according to attachment 1,
Each database server includes index column generation means for generating an index column composed of reference index data corresponding to each item row of column data distributed in the self database server,
The distributed database management system, wherein the data combination extraction unit extracts column data for search required by the search request by performing a combination based on the index column.

(Appendix 5)
A distributed database in which a request data processing device for distributing preset table data as column data to a plurality of different database servers obtains a search result based on the search request from each database server in response to a search request from an external terminal A management method,
The request data processing apparatus transfers schema information indicating the distribution destination of the column data to each database server, and when there is item data that is not stored in the join key column stored in each database server, The item data is stored as duplicate data in a complementary manner to the backend server,
Each of the database servers is
When a search request is sent from the request data processing device, it is required for the search request from the combined key sequence by combining with the data distributed in the self database server based on the combined key sequence. Search column data is extracted, and another server to which the binding data corresponding to the extracted search column data is distributed is specified based on the schema information, and the search column data is specified to the specified database server. A distributed database management method, wherein the search result is generated by transmitting and combining with the data for combination.

  The present invention can be effectively applied to a system that creates data marts by extracting data from a large amount of databases.

1 Database client 2 Distributed memory database management system 4 Internal network 21 Front memory database server (front server)
22, 23, 24 Backend memory database server (backend server)
211 Query execution unit 212, 222, 232, 242 Schema information management unit 213 Temporary storage memory area 221, 231, 241 Local query execution unit 223, 233, 243 Database storage memory area

Claims (5)

A request data processing device is provided that distributes preset table data as column data to a plurality of different database servers, and obtains a search result based on the search request from each database server in response to a search request from an external terminal. Distributed database management system,
The request data processing device includes:
A schema information replication transfer unit for transferring schema information indicating a distribution destination of the column data to each database server;
When there is item data that is not stored in the combination key column stored in each database server, a combination key data supplement storage unit that complementarily stores the item data as duplicate data in the back-end server; Prepared,
Each of the database servers is
When a search request is sent from the request data processing device, a search required by the search request from the combined key column is performed by combining the column data distributed in the self database server based on the combined key column A data combination extractor for extracting column data;
A server identification unit for coupling that identifies another server to which data for coupling corresponding to the extracted search string data is distributed based on the schema information;
A distributed database management system comprising: a search result combination generation unit that generates the search result by transmitting the search string data to the specified database server and combining the search string data with the combination data. The distributed database management system according to claim 1,
The combination key data complementation storage unit specifies column data corresponding to item data of the combination key column as target column data, which is column data to be searched based on the content of the search request, and the target column data A distributed database management system comprising a target column storage function for storing a target column as the replicated data in the corresponding back-end server. The distributed database management system according to claim 1,
The distributed data base management system, wherein the request data processing device comprises data arrangement management means for redundantly and exclusively normalizing item data included in column data distributed to each database server. The distributed database management system according to claim 1,
Each database server includes index column generation means for generating an index column composed of reference index data corresponding to each item row of column data distributed in the self database server,
The distributed database management system, wherein the data combination extraction unit extracts column data for search required by the search request by performing a combination based on the index column. A request data processing device that divides preset table data into column data and places them in a plurality of different database servers obtains search results based on the search requests from the respective database servers in response to search requests from external terminals. A distributed database management method,
The request data processing apparatus transfers schema information indicating the distribution destination of the column data to each database server, and when there is item data that is not stored in the join key column stored in each database server, The item data is stored as duplicate data in a complementary manner to the backend server,
Each of the database servers is
When a search request is sent from the request data processing device, it is required for the search request from the combined key sequence by combining with the data distributed in the self database server based on the combined key sequence. Search column data is extracted, and another server to which the binding data corresponding to the extracted search column data is distributed is specified based on the schema information, and the search column data is specified to the specified database server. A distributed database management method, wherein the search result is generated by transmitting and combining with the data for combination.

Images