JP2012108635A - Distributed memory database system, front database server, data processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a distributed memory database system and the like in which processing for an aggregate function such as a total can be performed at high speed by reducing communication traffic volume and throughput.SOLUTION: A front database server 10 has a data structure conversion section 113 and an inquiry processing section 111. The data structure conversion section 113 divides table data inputted from the outside to produce a plurality of value ID tables and stores the value ID tables while distributing them to data nodes. The inquiry processing section 111 inquires the number of appearance of a specific value ID in the table data of the data nodes on the basis of a query including an aggregate function issued from an external client machine, calculates a value of the aggregate function corresponding to the query from the number of appearance of the specific value ID returned from each of the data nodes in response to the inquiry and redirects the calculated value to the client machine. The data structure conversion section produces a plurality of value ID tables individually for each of a plurality of data items designated beforehand as a sequence that may become a tabulation axis in the table data.

Description

  The present invention relates to a distributed memory database system, a front database server, a data processing method, and a program, and more particularly, to a distributed memory database system and the like capable of quickly executing processing on a set function.

  In a system using a computer device of a certain size or more, such as a web service or a business system, it is indispensable to use a database management system (DBMS) in order to handle a large amount of data. In recent years, so-called cloud computing technology has been established in which a large number of computers connected via a network are linked to perform processing as one huge computer.

  In particular, a DBMS that distributes processing related to a database to a large number of computers connected by a network is called a distributed memory database system. The distributed memory database system is particularly effective for speeding up processing, such as batch applications that process a large amount of data at once, and creation of a data mart that extracts a specific department's needs from a large amount of data handled by a company. Is demonstrated.

  This will be described below. In this specification, in order to simplify the explanation, only a very small number of data and number of items are illustrated, but in practice, the processing as illustrated for the enormous number of data and number of items is performed. It is.

  FIG. 14 is an explanatory diagram showing a configuration of a general distributed memory database system 901. The distributed memory database system 1 includes a front memory database server 910 (hereinafter referred to as a front DB server 910) and a plurality of data nodes 921 to 923 connected to each other via an internal network 930. Although three data nodes 921 to 923 are shown in FIG. 1, of course, this number may be two or more. The front DB server 10 is connected to the client machine 950 via the external network 940.

  The client machine 950 is a computer that issues a query (processing request) to the front DB server 910. Data processing based on this query is performed in cooperation between the front DB server 910 and the data nodes 921 to 923, and the front DB server 910 returns the search result to the client machine 50. At that time, the front DB server 910 divides the query issued from the client machine 950 toward the data nodes 921 to 923 and aggregates the results from the data nodes 921 to 923.

  In the front DB server 910, an inquiry processing unit 911, a data arrangement information management unit 912, and a data structure conversion unit 913 are configured to execute respective functions described later as computer programs.

  The inquiry processing unit 911 receives a query issued by the client machine 950, inquires the data arrangement information management unit 912 about the location of the data item to be processed by this query, and in response to this inquiry, the data arrangement information management unit Based on the answer obtained from 112, the query from the client machine 950 is divided for each data node 921-923, and the divided query is transmitted to each data node 921-923. The answers from the data nodes 921 to 923 for each transmitted query are collected and returned to the client machine 950.

  The data arrangement information management unit 912 returns to the inquiry processing unit 911 which of the data nodes 921 to 923 data of the data item inquired from the inquiry processing unit 911 exists. The data structure conversion unit 913 divides the input table structure data and stores the divided data in the data nodes 921 to 923.

  In each of the data nodes 921 to 923, the inquiry processing unit 914 performs a function such as a search based on the query divided by the inquiry processing unit 911 and returns the result to the front DB server 910 as a computer program. It is configured as follows.

  FIG. 15 is an explanatory diagram showing an example of table data 960 input to the distributed memory database system 901 shown in FIG. The table data 960 shown in FIG. 15 has three items of data: date 960a, store ID 960b, and sales 960c. Of these, the date 960a uses two types of values, “August 10” and “August 11”.

  The data structure conversion unit 913 divides the table data 960 based on the values “A1”, “D3”, and “E1” of the store ID 910b, and generates table data 961 to 963 for each store ID. The data nodes 921 to 923 are sent and stored. FIG. 16 is an explanatory diagram showing table data 961 to 963 for each shop ID generated by the data structure conversion unit 913 dividing the table data 910 shown in FIG.

The inquiry processing unit 911 receives a query issued by the client machine 950 in a state where the table data 961 to 963 for each store ID shown in FIG. 16 are stored in the data nodes 921 to 923, respectively. For example, when the query shown in the following equation 1 is received, the inquiry processing unit 911 receives the total for each store ID from the table data 961 to 963 for each store ID stored in each of the data nodes 921 to 923. The sales amount is calculated, and the total sales amount for each returned shop ID is collectively returned to the client machine 950.

  There are the following technologies related to this. Among them, Patent Document 1 assigns a common global dimension value number to data processed by a plurality of processing modules of a parallel computer, thereby reducing the occurrence of inter-processor communication and realizing data sorting and aggregation. An information processing system is described. Patent Document 2 describes a grouping method that uses a hash value to speed up the process of grouping rows that share one or more column values.

  Patent Document 3 describes a data analysis system that speeds up data analysis on a plurality of computers by dividing data including a plurality of analysis problems into layers. Patent Document 4 describes a database processing method for shortening the end processing time of a system when processing data distributed to a plurality of devices.

  Patent Document 5 describes a data processing system that adds a label code to data to speed up data processing by a plurality of devices. Non-Patent Document 1 describes a “bitmap index” that can speed up the exact match search in an Oracle (registered trademark) database that is widely used as a database.

Re-specialized WO2005 / 041067 JP 2000-187668 A JP 2006-107129 A JP 2010-134583 A Japanese Patent Laid-Open No. 07-182368

Paul Lane, “Using Bitmap Indexes in Data Warehouses”, Oracle Database Data Warehousing Guide 11g Release 1, 2007, Oracle Corporation Japan [retrieved November 8, 2010], Internet <URL: http://otndnld.oracle.co.jp/document/products/oracle11g/111/doc_dvd/server.111/E05763-01/indexes.htm>

  Particularly in the distributed memory database system, it is important to reduce the communication generated between the nodes as much as possible in order to reduce the processing cost and processing time. The example of the distributed memory database system 901 shown in FIGS. 14 to 16 is illustrated with respect to a very small number of data in order to simplify the description, but actually processes a huge amount of data. There is a need.

  In the distributed memory database system 901 shown in FIGS. 14 to 16, the data is divided for each “store ID” and stored in each data node 921 to 923. If so, a set function such as a sum can be calculated by each of the data nodes 921 to 923 alone. Therefore, the processing in the inquiry processing unit 911 simply summarizes numerical values such as the sum received from the data nodes 921 to 923, and can perform the aggregation processing at high speed.

  However, if data is to be tabulated other than the column used as a reference for data division (in this case, “store ID”), data is exchanged between the data nodes 921 to 923 and between them and the front DB server 910. Is required. For this reason, it takes time for the aggregation processing, and communication costs associated with data exchange occur.

In the example of the distributed memory database system 901 shown in FIGS. 14 to 16, when the query shown in the following equation 2 is accepted, calculation of a set function such as a sum is performed only by each of the data nodes 921 to 923. Therefore, all data is re-divided on the basis of “date” (first method) or each data node 921 to 923 holding data for each “store ID” for each “date”. Totaling is performed, and the result is sent to the front DB server 910, and the total for each “date” is calculated (second method).

In the first method, all of the table data 961 to 963 are once sent to the front DB server 910, returned to the original table data 960, and then re-divided on the basis of “date” as the table data. It is necessary to send to the data nodes 921 to 923. For example, there are m data nodes, where 1 byte of data per row is n times per day for one node, s (m, n, s, l are natural numbers, and s = m for simplicity. If the data for the stores are present evenly, in order to re-divide the table data, communication with a data capacity as shown in the following Equation 3 occurs.

  In addition to the amount of communication generated, it is also necessary to read individual data and determine which data node to move each of them, so the amount of processing required for the front DB server 910 and each data node also Increase.

  In the second method, the total data obtained for each “store ID / date” is transmitted from each data node 921 to 923 to the front DB server 910. For this reason, the amount of communication to the front DB server 910 increases. In addition, since the front DB server 910 needs to obtain a total for each “date” in addition to simple replacement (unlike the query shown in Equation 1), the amount of processing here also increases.

  In order to solve this problem, a method of preparing an index (index) and corresponding to aggregation by different aggregation axes is already known. However, even in this case, it is necessary to transmit the calculation target data to another computer for the tabulation operation. For this reason, the effect of reducing the amount of communication generated is small.

  In Patent Documents 1 and 4 to 5, data obtained by assigning IDs (or labels) to the same values in the same column and performing grouping (so-called range partitioning) based on the IDs (or labels) is provided for each data. A technique for storing in a node is described. If this is used, the amount of communication will be reduced somewhat rather than communicating raw data itself. However, since the processing amount on the front DB server 910 side is not reduced, the processing amount is rather increased because the processing involves replacing the ID with an actual value. The remaining Patent Documents 2 to 4 and Non-Patent Document 1 do not describe a technique that can solve this problem.

  An object of the present invention is to provide a distributed memory database system, a front database server, a data processing method, and a program that can reduce the amount of communication and the amount of processing, and can perform processing on a set function such as a sum at high speed. It is in.

  To achieve the above object, a distributed memory database system according to the present invention is a distributed memory database system configured by connecting a front database server and a plurality of data nodes to each other, and the front database server is externally connected. A data structure conversion unit that generates a plurality of value ID tables by dividing the table data input from, and replaces each actual data with a value ID, and distributes and stores them in each data node; and an external client Based on the query including the set function issued from the machine, each data node is inquired about the number of occurrences of the specific value ID in the table data, and the specific value ID returned from each data node in response to this is inquired. Query processing that calculates the value of the set function corresponding to the query from the number of occurrences and returns it to the client machine When the data structure conversion unit generates a plurality of value ID tables, each of the plurality of data items designated in advance as a column that can serve as an aggregation axis in the table data A value ID table is generated.

  In order to achieve the above object, a front database server according to the present invention is a front database server that is connected to a plurality of data nodes to constitute a distributed memory database system, and divides table data input from the outside. A data structure conversion unit that generates a plurality of value ID tables in which each actual data is replaced with a value ID and stores them in each data node, and a set function issued from an external client machine Based on the included query, each data node is inquired about the number of occurrences of the specific value ID in the table data, and in response to this, the query is handled from the number of occurrences of the specific value ID returned from each data node. A query processing unit that calculates the value of the set function and sends it back to the client machine. When generating the value ID table, and generating a plurality of values ID table individually for each of the plurality of data items specified in advance as a column which can be a tabulation axis in the table data.

  In order to achieve the above object, a data processing method according to the present invention is a distributed memory database system in which a front database server and a plurality of data nodes are connected to each other. The data structure conversion unit of the front database server accepts the input table data, and the data structure conversion unit of the front database server individually divides the input table data for each of the data items specified in advance as columns that can be aggregate axes. A plurality of value ID tables are generated by replacing the data with value IDs, and the data structure conversion unit of the front database server stores the generated value ID tables in a distributed manner in each data node. Query the front database server for the query that contains the issued aggregate function The inquiry processing unit of the front database server inquires each data node about the number of occurrences of the specific value ID in the table data based on the received query, and the specific value ID returned from each data node. The query processing unit of the front database server calculates the value of the set function corresponding to the query from the number of appearances and returns it to the client machine.

  In order to achieve the above object, a data processing program according to the present invention is a distributed memory database system in which a front database server and a plurality of data nodes are connected to each other in a computer included in the front database server. A procedure for receiving input of table data from the outside, a plurality of input table data divided into individual data items designated in advance as columns that can be aggregated axes, and replacing actual data with value IDs For generating a value ID table, a procedure for storing a plurality of generated value ID tables in each data node, a procedure for receiving a query including a set function issued from an external client machine, a received query Ask each data node for the number of occurrences of a specific value ID in the table data based on Characterized in that to execute a procedure to return to the client machine instructions cause I, and the number of occurrences of a particular value ID returned from each data node by calculating the value of the set function corresponding to the query.

  As described above, the present invention is configured to divide the table data individually for each data item designated in advance as a column that can be the aggregation axis. It is possible to perform processing related to the set function only within each data node while suppressing the occurrence of communication with the device. As a result, a distributed memory database system, a front database server, a data processing method, and a program having an excellent feature that it is possible to reduce the amount of communication and the amount of processing, and to perform processing on a set function such as a total at high speed. It becomes possible to provide.

It is explanatory drawing which shows the structure of the distributed memory database system which concerns on the 1st Embodiment of this invention. It is explanatory drawing which shows an example of the table data input with respect to the distributed memory database system demonstrated in FIG. It is explanatory drawing which shows the value list and value ID table which a data structure conversion part produces from the table data shown in FIG. FIG. 4 is a flowchart showing a process in which the data structure conversion unit shown in FIG. 1 divides the data shown in FIG. 2 as shown in FIG. It is explanatory drawing shown about the example corresponding to each data shown in FIG. 2 and FIG. 3 of the data arrangement | positioning information shown in FIG. 5 is a flowchart showing processing performed in the distributed memory database system described with reference to FIG. 1 for the query shown in Equation 4. FIG. 7 is an explanatory diagram showing a table of the number of appearances for each “value ID” returned from the data node to the front DB server in the process shown in step S403 (Equation 4) of FIG. 6. It is explanatory drawing shown about result data which shows the sum total of the sales for every date returned as a result of the process shown to step S406 (Formula 5) of FIG. FIG. 7 is an explanatory diagram showing a table of the number of appearances for each “value ID” returned from the data node to the front DB server in the process shown in step S403 (formula 6) of FIG. 6. FIG. 7 is an explanatory diagram showing result data indicating the total sales for each shop ID returned to the client machine as a result of the process shown in step S406 (Equation 7) in FIG. 6; It is explanatory drawing which shows the structure of the distributed memory database system which concerns on the 2nd Embodiment of this invention. It is explanatory drawing which shows an example of the table data input with respect to the distributed memory database system demonstrated in FIG. It is explanatory drawing which shows the example of the value ID table which a data structure conversion part produces from the table data shown in FIG. It is explanatory drawing which shows the structure of a general distributed memory database system. It is explanatory drawing which shows an example of the table data input with respect to the distributed memory database system shown in FIG. It is explanatory drawing which shows the table data according to shop ID which the data structure conversion part divided | segmented and produced | generated the table data shown in FIG.

(First embodiment)
Hereinafter, the structure of the 1st Embodiment of this invention is demonstrated based on attached FIGS. 1-3.
First, the basic content of the present embodiment will be described, and then more specific content will be described.
The distributed memory database system 1 according to the present embodiment is a distributed memory database system configured by connecting a front database server (front DB server 10) and a plurality of data nodes 21 to 23 to each other. The front database server 10 generates a plurality of value ID tables 221 to 222, 231 to 233 in which table data input from the outside is divided and each actual data is replaced with a value ID, and these are stored in each data node. Based on the data structure conversion unit 113 to be stored in a distributed manner and a query including a set function issued from the external client machine 50, each data node is inquired about the number of occurrences of a specific value ID in the table data. In response, a query processing unit 111 that calculates the value of the set function corresponding to the query from the number of occurrences of the specific value ID returned from each data node and returns it to the client machine. Then, when the data structure conversion unit 113 generates the plurality of value ID tables, the data structure conversion unit 113 individually generates a plurality of value ID tables for each of the data items designated in advance as a column that can be an aggregation axis in the table data. Generate.

  In addition, when the data structure conversion unit 113 of the front database server 10 generates a plurality of value ID tables 221 to 222 and 231 to 233, the data structure conversion unit 113 is designated in advance as columns that can be aggregated axes and aggregation targets in the table data. For each of the data items, an actual value is replaced with a value ID, and a value list 211 to 213 indicating the correspondence between the value ID and the actual value is generated. A data arrangement information management unit 112 that stores in which storage unit the value ID table is distributed among the data nodes is provided.

  Furthermore, when the data structure conversion unit 113 of the front database server 10 generates the value ID tables 221 to 222 and 231 to 233, the actual values are sorted in order of magnitude and replaced with the value ID.

With the above configuration, the distributed memory database system 1 can reduce the amount of communication and the amount of processing, and can perform processing on a set function such as a sum at high speed.
Hereinafter, this will be described in more detail.

  FIG. 1 is an explanatory diagram showing a configuration of a distributed memory database system 1 according to the first embodiment of the present invention. The distributed memory database system 1 includes a front database server 10 (hereinafter referred to as a front DB server 10) and a plurality of data nodes 21 to 23 connected to each other via an internal network 30. Although three data nodes 21 to 23 are shown in FIG. 1, of course, this number may be two or more.

  The internal network 30 is connected to a client machine 50 that is an external computer device via an external network 40. Any network system and protocol for the internal network 30 and the external network 40 can be used. The front DB server 10 can accept operations from the client machine 50 through the internal network 30 and the external network 40.

  The client machine 50 issues a query (processing request) to the front DB server 10, and the front DB server 10 and the data nodes 21 to 23 perform data processing based on the query in cooperation with the front DB server 10. The search result is returned to the client machine 50. At that time, the front DB server 10 divides the query issued from the client machine 50 toward the data nodes 21 to 23 and aggregates the results from the data nodes 21 to 23.

  The front DB server 10 is a computer device that includes main calculation control means 101, storage means 102, and communication means 103. The main arithmetic control means 101 is a CPU (Central Processing Unit) that is the main body of the computer program, and the storage means 102 is a main memory such as a RAM (Random Access Memory) that stores data that the main arithmetic control means 101 is working on. It is a storage device. The communication means 103 performs data communication with other computers via the internal network 30 and the external network 40.

  In the main calculation control means 101, the inquiry processing unit 111, the data arrangement information management unit 112, and the data structure conversion unit 113 are configured to execute respective functions described later as computer programs. The storage unit 102 stores data arrangement information 121 to be described later.

  The inquiry processing unit 111 accepts a query issued by the client machine 50, inquires the data arrangement information management unit 112 about the location of the data item to be processed by this query, and in response to this inquiry, the data arrangement information management unit Based on the answer obtained from 112, the query from the client machine 50 is divided for each data node 21-23, and the divided query is transmitted to each data node 21-23. Then, the answers from the data nodes 21 to 23 for each transmitted query are collected and returned to the client machine 50.

  The data arrangement information management unit 112 refers to the data arrangement information 121 to determine which of the data nodes 21 to 23 the data of the data item queried from the inquiry processing unit 111 exists. Reply to 111.

  As will be described later, when the table structure data is input to the present system, the data structure conversion unit 113 converts the data structure into a data structure unique to the present embodiment described later and divides the data structure into each data node 21 to 23. Remember.

  Each of the data nodes 21 to 23 also has a configuration as a general computer device like the front DB server 10, but all have the same configuration as hardware and software, but the contents stored therein are different. Only. Therefore, FIG. 1 shows a detailed configuration only for the data node 21. Similar to the front DB server 10, the data node 21 is a computer device including a main arithmetic control unit 201, a storage unit 202, and a communication unit 203.

  In the main arithmetic control means 201, the inquiry processing unit 204 is configured to execute each function described later as a computer program. The storage unit 202 stores value lists 211 to 213 and value ID tables 221 to 222 and 231 to 233 described later. The inquiry processing unit 204 stores the value list 211 to 213 and the value ID tables 221 to 222 and 231 to 233 that are divided and generated by the front DB server 10 in the storage unit 202. The value list and the value ID table are searched, and the result is returned to the front DB server 10.

  FIG. 2 is an explanatory diagram showing an example of the table data 210 input to the distributed memory database system 1 described in FIG. This data input may be performed from the client machine 50 or directly from the front DB server 10. The data structure conversion unit 113 converts the input table data 210 into a format described below, distributes the data to the plurality of data nodes 21 to 23, and stores them.

  The table data 210 shown in FIG. 2 has three items of data: date 210a, store ID 210b, and sales 210c. Of these, “date 210a” and “store ID 210b” are designated in advance as “columns that can be aggregated axes (hereinafter referred to as reference columns)” and sales 210c are designated in advance as “columns that can be aggregated (hereinafter referred to as target columns)”. Yes. The contents of the table data 210 are the same as the table data 910 in FIG.

  FIG. 3 is an explanatory diagram showing value lists 211 to 213 and value ID tables 221 to 222 and 231 to 233 created by the data structure conversion unit 113 from the table data 210 shown in FIG. Among the items specified as the reference column in the table data 210 shown in FIG. 2, the date 210a has two types “August 10” and “August 11”, and the store ID 210b has “A1” “D3” “E1”. Three types of values are used.

  Therefore, the data structure conversion unit 113 arranges the unique values existing in each column in an ascending order, assigns numbers (value IDs) for specifying the respective values in order from the top, and sets the “date” value list. Three value lists are created: 211, “Store ID” value list 212, and “Sales” value list 213.

  The “date” value list 211 indicates correspondence between “date” value IDs and dates. When “date” value ID = “0”, date = “August 10”, and when “date” value ID = “1”, date = “August 11”.

  The “store ID” value list 212 indicates the correspondence between the “store ID” value ID and the store ID value. When “Store ID” value ID = “0”, Store ID = “A1”. When “Store ID” value ID = “1”, Store ID = “D3”, “Store ID” value ID = “ In the case of “2”, the store ID = “E1”.

  The “sales” value list 213 indicates correspondence between “sales” value IDs and sales values. In the case of “sales” value ID = “0” to “4”, the sales are “800”, “1000”, “1200”, “4800”, and “12000”, respectively.

  Then, the data structure conversion unit 113 replaces the table data 210 with each value ID, further divides the table data 210 for each value of the date 210a and the store ID 210b, and creates the value ID tables 221 to 222, 231 to 233. . The value ID table 221 shows the correspondence between the “store ID” value ID and the “sales” value ID when “date” value ID = “0”. The value ID table 222 shows the correspondence between the “store ID” value ID and the “sales” value ID when “date” value ID = “1”.

  The value ID table 231 shows the correspondence between the “date” value ID and the “sales” value ID in the case of “store ID” value ID = “0”. The value ID table 232 shows the correspondence between the “date” value ID and the “sales” value ID in the case of “store ID” value ID = “1”. The value ID table 233 shows the correspondence between the “date” value ID and the “sales” value ID in the case of “store ID” value ID = “2”.

  The data structure conversion unit 113 distributes and stores the value lists 211 to 213 and the value ID tables 221 to 222 and 231 to 233 created as described above to the data nodes 21 to 23. In the example illustrated in FIG. 3, the data structure conversion unit 113 stores the value ID table 222, the value ID table 231, and the value list 211 in the storage unit 202 of the data node 21. Further, the value ID table 221, the value ID table 232, and the value list 212 are stored in the storage unit 202 of the data node 22. Further, the value ID table 233 and the value list 213 are stored in the storage unit 202 of the data node 23.

  Here, with respect to the processing performed by the data structure conversion unit 113, an arbitrary division method is used as long as the data capacity and the processing amount are not largely biased to one specific data node among the data nodes 21 to 23. be able to. Furthermore, when designing the data structure of the table data 210 shown in FIG. 2, it is assumed that the operator has input in advance the attributes of each column such as the reference column and the target column. The data structure conversion unit 113 creates a value list for all the data columns set to be the “reference column” and the “target column”, and the above-described data for all the data columns set to be the “reference column”. The table data 210 is divided as shown in FIG.

  FIG. 4 is a flowchart showing a process in which the data structure conversion unit 113 shown in FIG. 1 divides the data shown in FIG. 2 as shown in FIG. 3, distributes the data to the data nodes 21 to 23, and stores them. First, an input by the user is accepted for the data structure of the table data 210 (step S301). At this time, an input as to which data item is the “reference column” or “target column” is also performed at the same time.

  Next, the input of the table data 210 is received from the user (step S302), and when this is completed, the value of the data string is first replaced with the value ID for one of the reference columns (steps S303 to 304) (step S305). At the same time, the table is divided for each value of the reference column (step S306). The data structure conversion unit 113 repeats the processing in steps S305 to S306 for all the reference columns, and simultaneously creates value lists 211 to 213 indicating the correspondence between the replaced values and the value IDs.

  When this process is completed for all the reference columns, the completed value list and value ID table are transmitted to and stored in the data nodes 21 to 23 (step S307).

  FIG. 5 is an explanatory diagram showing an example of the data arrangement information 121 shown in FIG. 1 corresponding to each data shown in FIG. 2 and FIG. The data arrangement information 121 includes the data node name 121a indicating the computer name of each data node 21 to 23, the value lists 211 to 213 stored in the computer, and the data types of the value ID tables 221 to 222 and 231 to 233. The correspondence with the stored content 121b shown is shown.

  The data arrangement information 121 is not limited to the form shown in the example shown in FIG. 5, and the data used in the calculation and the actual value corresponding to the value ID are among the data nodes 21 to 23. Any data format may be used as long as it can be specified where the data is stored.

(Sales summary processing by date)
As described above, when the query (SQL sentence) is issued from the client machine 50 to the front DB server 10 for the data distributed and stored in each of the data nodes 21 to 23, the front DB server 10 and each Processing performed in the data nodes 21 to 23 will be described below.

Formula 4 shown below is an example of a query that the client machine 50 issues to the front DB server 10 for the data example having the contents shown in FIGS. This is a query for obtaining the total sales for each date from the table data 210.

  FIG. 6 is a flowchart showing processing performed in the distributed memory database system 1 described with reference to FIG. The query processing unit 111 of the front DB server 10 that has received the query shown in Equation 4 refers to the data allocation information 121 by the data allocation information management unit 112, and a value ID table for each “date” relative to “sales table”. Is present on each of the data nodes 21 to 23, and the node where each of the divided tables exists is identified (step S401).

  Then, the inquiry processing unit 111 of the front DB server 10 sends the “sales” column expressed by “value ID” in the date-specific value ID tables 221 to 222 to the inquiry processing unit 204 of the data nodes 21 to 23. An inquiry is issued so as to count the number of occurrences for each “value ID” (step S402). Since the data node 23 does not store the value ID table for each date, it is excluded here.

  The inquiry processing unit 204 of the data nodes 21 to 22 that has received the inquiry calculates the number of appearances for each “value ID” in the “sales” column for the date-specific value ID tables 221 to 222, and uses this as the front DB. Return to the server 10 (step S403). FIG. 7 is an explanatory diagram showing tables 241 to 242 of the number of appearances for each “value ID” returned from the data nodes 21 to 22 to the front DB server 10 in the process shown in step S403 (expression 4) of FIG. is there.

  Receiving this, the inquiry processing unit 111 of the front DB server 10 acquires the value lists 211 to 213 corresponding to the “value IDs” of “date” and “sales” from the data nodes 21 to 23 (steps). S404-405). Note that the processing in steps S404 to S405 may be such that the value list 211 to 213 itself is not acquired from each data node 21 to 23, but the value corresponding to the value ID is acquired from each data node 21 to 23. .

  Then, the inquiry processing unit 111 of the front DB server 10 applies the actual “sales” numerical value to the number of appearances of each “value ID” in the “sales” column returned in step S403, thereby calculating the actual sales amount. Is calculated as result data 243 and returned to the client machine 50 (step S406).

Equation 5 below shows the calculation actually performed by the query of Equation 4 for the data example having the contents shown in FIGS. 2 to 3 in the process shown in step S406 of FIG. FIG. 8 is an explanatory diagram showing result data 243 indicating the total sales for each date returned to the client machine 50 as a result of the process shown in step S406 (Equation 5) of FIG.

(Sales summary processing by store ID)
For example, even when the client machine 50 issues the query shown in the following Equation 6 by the operation shown in FIG. 6, the processing shown above can be similarly processed. This is a query for calculating the total sales for each store ID from the table data 210.

  In this case, the inquiry processing unit 111 refers to the data arrangement information 121 by the data arrangement information management unit 112, and a value ID table for each “store ID” exists on each data node 21 to 23 with respect to “sales table”. To identify the node in which each divided table exists (step S401).

  Then, the inquiry processing unit 111 of the front DB server 10 sends “sales” expressed by “value ID” in the value ID tables 231 to 233 for each store ID to the inquiry processing unit 204 of the data nodes 21 to 23. An inquiry is issued so as to count the number of occurrences of each column value “value ID” (step S402).

  The inquiry processing unit 204 of the data nodes 21 to 23 that received the inquiry calculates the number of appearances for each “value ID” in the “sales” column for the value ID tables 231 to 233 for each store ID, Return to the DB server 10 (step S403). FIG. 9 is an explanatory diagram showing tables 251 to 253 of the number of appearances for each “value ID” returned from the data nodes 21 to 23 to the front DB server 10 in the process shown in step S403 (formula 6) of FIG. is there.

  In response to this, the inquiry processing unit 111 of the front DB server 10 acquires the value lists 211 to 213 corresponding to the “value IDs” of “store ID” and “sales” from the data nodes 21 to 23 ( Steps S404 to 405). Note that the processing in steps S404 to S405 may be such that the value list 211 to 213 itself is not acquired from each data node 21 to 23, but the value corresponding to the value ID is acquired from each data node 21 to 23. .

  Then, the inquiry processing unit 111 of the front DB server 10 applies the actual “sales” numerical value to the number of appearances of each “value ID” in the “sales” column returned in step S403, thereby calculating the actual sales amount. Is calculated as result data 254, and is calculated and returned to the client machine 50 (step S406).

Equation 7 below shows the calculation actually performed by the query of Equation 6 for the data example shown in FIGS. 2 to 3 in the processing shown in step S406 of FIG. FIG. 10 is an explanatory diagram showing result data 254 indicating the total sales for each store ID returned to the client machine 50 as a result of the process shown in step S406 (Equation 7) of FIG.

(Aggregation processing other than Japanese)
The processing when the query for calculating the sum (SUM) of the target columns is issued from the client machine 50 to the front DB server 10 for the data distributed and stored in the data nodes 21 to 23 has been described above. However, other operations such as obtaining the minimum value (MIN), the maximum value (MAX), the number of appearances (COUNT), and the average value (AVG) can be performed by the same system.

  When a query for obtaining the minimum value (MIN) or the maximum value (MAX) is received, the inquiry processing unit 204 uses the value ID tables 221 to 222 and 231 to 233 stored in the data nodes 21 to 23, respectively. A value having the maximum or minimum value ID is selected and returned to the inquiry processing unit 111 of the front DB server 10. Then, the inquiry processing unit 111 of the front DB server 10 returns values corresponding to the value IDs returned from the value lists 211 to 213 to the client machine 50 in the same manner as in step S406 of FIG.

  Since the value lists 211 to 213 are arranged in order from the smallest value as described above, and value IDs are assigned in that order, the inquiry processing unit 204 of each data node 21 to 23 receives the value list 211. Even if the actual value is not referred to in steps 213 to 213, the value having the maximum or minimum value ID can be determined to be the maximum value or the minimum value.

  When a query for the number of occurrences (COUNT) is received, the number of occurrences for each value ID is stored in the value ID tables 221 to 222 and 231 to 233 in the data nodes 21 to 23. Therefore, the front DB server 10 The inquiry processing unit 111 calculates the number of appearances on each data node 21 to 23 side. Then, the inquiry processing unit 111 of the front DB server 10 receives the number of appearances returned from each of the data nodes 21 to 23, converts the value ID into an actual value in the same manner as in step S406 in FIG. Return to machine 50.

  When a query for obtaining an average value (AVG) is received, the same processing as steps S401 to S406 in FIG. 6 is performed. In step S406, the query processor 111 of the front DB server 10 sets each value ID to an actual value. The only difference from the processing shown in FIG. 6 is that the average value is obtained after conversion and returned to the client machine 50.

(Overall operation of the first embodiment)
Next, the overall operation of the above embodiment will be described. The data processing method according to the present embodiment is a distributed memory database system 1 configured by connecting a front database server (front DB server 10) and a plurality of data nodes 21 to 23 to each other. The data structure conversion unit of the front database server accepts the input of table data (FIG. 4, steps S301 to 302), and the input table data is stored in the front database for each data item designated in advance as a column that can be an aggregation axis. The data structure conversion unit of the server individually generates a plurality of value ID tables in which actual data is replaced with value IDs (FIG. 4, steps S304 to 306), and the generated plurality of value ID tables are converted into a front database. The server's data structure conversion unit is distributed and stored in each data node (FIG. 4, step S). 07), the query processing unit of the front database server accepts a query including a set function issued from an external client machine, and the number of occurrences of a specific value ID in the table data is determined based on the accepted query. The query processing unit queries each data node (FIG. 6, steps S401 to 402), and the query processing unit of the front database server determines the set function corresponding to the query from the number of occurrences of the specific value ID returned from each data node. The value is calculated and returned to the client machine (FIG. 6, steps S404 to S406).

Here, each of the above operation steps may be programmed so as to be executable by a computer, and these may be executed by the front DB server 10 which is a computer that directly executes each of the steps. The program may be recorded on a non-transitory recording medium, such as a DVD, a CD, or a flash memory. In this case, the program is read from the recording medium by a computer and executed.
By this operation, this embodiment has the following effects.

  In the present embodiment, the table data 210 is divided and distributed to the data nodes 21 to 23 for all the reference columns, that is, columns that can be aggregate axes. As a result, even if a query is issued for any reference column, the aggregation processing for the set function is performed by only one of the data nodes 21 to 23 without causing the exchange of data with other devices, Only the total result can be transmitted to the front DB server 10. Furthermore, the processing in the front DB server 10 can be performed at high speed because it is sufficient to simply replace the value ID with the actual value.

  At this time, the data stored in each of the data nodes 21 to 23 is stored after being converted into an expression based on the value list and the value ID, so that the capacity of the data stored in each of the data nodes 21 to 23 can be reduced. it can. In particular, when there are many overlapping values, the effect of reducing the data capacity becomes more remarkable. Furthermore, an increase in data capacity when the table data 210 is divided for a plurality of reference columns can be suppressed to a minimum.

(Second Embodiment)
In the second embodiment of the present invention, the data structure conversion unit of the front database server (front DB server 510) is data designated in advance as a column that cannot be either the aggregation axis or the aggregation target in the table data. Each item is stored in association with any one of the value ID tables for columns that can be aggregate axes.

According to this configuration, the same effect as that of the first embodiment can be obtained for data including data items that cannot be any of the aggregation axis and the aggregation target.
Hereinafter, this will be described in more detail.

  FIG. 11 is an explanatory diagram showing the configuration of the distributed memory database system 501 according to the second embodiment of the present invention. The distributed memory database system 501 is configured by connecting a front DB server 510 and a plurality of data nodes 521 to 523 to each other via the same internal network 30 as in the first embodiment. The external network 40 and the client machine 50 are the same as those in the first embodiment.

  The front DB server 510 and the data nodes 521 to 523 have the same hardware configuration as the front DB server 10 and the data nodes 21 to 23 of the first embodiment. The inquiry processing unit 111 and the data structure conversion unit 113 that operate in the main arithmetic control unit 101 of the front DB server 510 are replaced with an inquiry processing unit 511 and a data structure conversion unit 513, respectively. Further, regarding the data nodes 521 to 523, the value ID tables 621 to 622 and 631 to 632 stored therein are different from those of the first embodiment.

  FIG. 12 is an explanatory diagram showing an example of the table data 610 input to the distributed memory database system 501 described in FIG. The table data 610 has data items such as an item A 610a, an item B 610b, an item C 610c, and an item D 610d. Among these, the item A 610a and the item B 610b are specified as the reference columns, and the item D 610d is specified as the target column. Does not fall into either column or target column. The item A 610a is also designated as a primary key at the same time.

  FIG. 13 is an explanatory diagram showing an example of value ID tables 621 to 622 and 631 to 632 created by the data structure conversion unit 513 from the table data 610 shown in FIG. As described above, when the item C610c that is neither the reference column nor the target column is included in the table data 610, the data structure conversion unit 513 assigns the value ID of the item C610c only to the value ID tables 621 to 622 for each value ID of the item A610a. At the same time, the value ID tables 631 to 632 for each value ID of the item B 610b are not included in the value ID of the item C 610c. The value list is created for all items A to D in the same manner as in the first embodiment, and is distributed and stored in the data nodes 521 to 523.

  For example, when the user wants to know the value or value ID of the corresponding item C610c from the specific value or value ID of the item B610b, first, from the value ID tables 631 to 632 for each value ID of the item B610b, The value ID of the item A 610a corresponding to the specific value ID can be specified, and the value ID of the item C 610c corresponding to this can be specified by referring to the value ID tables 621 to 622 for each value ID of the item A 610a. .

  Since the item C610c is neither the reference column nor the target column, and therefore is not a target of processing by the set function, it is not necessary to maintain the correspondence relationship in all the value ID tables, and the correspondence with the values of other data items is merely It should be understood. Therefore, if the correspondence with the item C610c is known only for any one of the reference columns (not necessarily the primary key), the correspondence with the other values can be traced.

(Extended embodiment)
In the first and second embodiments described above, various extensions can be considered without departing from the spirit of the first and second embodiments.
For example, for the created value ID table and value list, it is not always necessary to store one table in one data node. One value ID table or value list may be divided into a plurality of data nodes and stored in accordance with restrictions such as the storage capacity of each data node. In this case, the data arrangement information management unit 112 grasps which value ID table or value list is divided and stored in which data node and stores it in the data arrangement information 121. That's fine.

  The present invention has been described with reference to the specific embodiments shown in the drawings. However, the present invention is not limited to the embodiments shown in the drawings, and any known hitherto provided that the effects of the present invention are achieved. Even if it is a structure, it is employable.

  About each embodiment mentioned above, it is as follows when the summary of the novel technical content is put together. In addition, although part or all of the said embodiment is summarized as follows as a novel technique, this invention is not necessarily limited to this.

(Supplementary note 1) A distributed memory database system comprising a front database server and a plurality of data nodes connected to each other,
The front database server is
A data structure conversion unit that divides table data input from the outside and generates a plurality of value ID tables in which each actual data is replaced with a value ID, and distributes and stores these in each data node;
Based on a query including an aggregate function issued from an external client machine, the data node is inquired about the number of occurrences of a specific value ID in the table data, and the data node returns a response accordingly. A query processing unit that calculates a value of a set function corresponding to the query from the number of occurrences of the specified value ID and returns the value to the client machine,
When the data structure conversion unit generates the plurality of value ID tables, the plurality of value IDs individually for each of a plurality of data items designated in advance as a column that can be an aggregation axis in the table data. A distributed memory database system characterized by generating a table.

(Supplementary Note 2) When the data structure conversion unit of the front database server generates the plurality of value ID tables, data items designated in advance as columns that can be aggregated axes and aggregation targets in the table data And a function of generating a value list indicating a correspondence between the value ID and the actual value, and replacing an actual value with a value ID for each of
The distributed memory according to claim 1, wherein the front database server includes a data arrangement information management unit that stores in which of the data nodes the value list and the value ID table are distributed. Database system.

(Supplementary note 3) When the data structure conversion unit of the front database server generates the plurality of value ID tables, the actual values are sorted in order of magnitude and then replaced with the value ID. The distributed memory database system according to attachment 2.

(Additional remark 4) The said data structure conversion part of the said front database server is set to the said total axis about each of the data item previously designated as a column which cannot become any of the total axis and the total object in the said table data. The distributed memory database system according to appendix 2, wherein the distributed memory database system is stored in association with any one type of the value ID table for possible columns.

(Supplementary Note 5) A front database server that is connected to a plurality of data nodes to constitute a distributed memory database system,
A data structure conversion unit that divides table data input from the outside and generates a plurality of value ID tables in which each actual data is replaced with a value ID, and distributes and stores these in each data node;
Based on a query including an aggregate function issued from an external client machine, the data node is inquired about the number of occurrences of a specific value ID in the table data, and the data node returns a response accordingly. A query processing unit that calculates a value of a set function corresponding to the query from the number of occurrences of the specified value ID and returns the value to the client machine,
When the data structure conversion unit generates the plurality of value ID tables, the plurality of value IDs individually for each of a plurality of data items designated in advance as a column that can be an aggregation axis in the table data. A front database server characterized by generating a table.

(Supplementary Note 6) In a distributed memory database system configured by connecting a front database server and a plurality of data nodes to each other,
The data structure conversion unit of the front database server accepts input of table data from the outside,
The input table data is divided into a plurality of data items specified in advance as columns that can be aggregated axes, and the data structure conversion unit of the front database server individually divides and replaces actual data with value IDs. Generate a value ID table,
The data structure conversion unit of the front database server distributes and stores the generated plurality of value ID tables in the data nodes,
The query processing unit of the front database server accepts a query including a set function issued from an external client machine,
Based on the accepted query, the query processing unit of the front database server queries each data node for the number of occurrences of the specific value ID in the table data,
Data in which the query processing unit of the front database server calculates the value of the set function corresponding to the query from the number of occurrences of the specific value ID returned from each data node and returns it to the client machine Processing method.

(Supplementary Note 7) In a distributed memory database system configured by connecting a front database server and a plurality of data nodes to each other,
In the computer provided in the front database server,
Procedure to accept external table data input,
A procedure for generating a plurality of value ID tables in which the input data is individually divided for each data item designated in advance as a column that can serve as an aggregation axis, and actual data is replaced with value IDs;
A procedure for distributing and storing the generated plurality of value ID tables in each data node;
A procedure for accepting a query including a set function issued from an external client machine,
A procedure for inquiring each data node about the number of occurrences of a specific value ID in the table data based on the accepted query;
A data processing program for executing a procedure for calculating a value of a set function corresponding to the query from the number of appearances of a specific value ID returned from each data node and returning the value to the client machine.

  The present invention can be widely applied to a computer system using a database, particularly a database system using a distributed memory.

DESCRIPTION OF SYMBOLS 1,501 Distributed memory database system 10,510 Front DB server 21-23, 521-523 Data node 30 Internal network 40 External network 50 Client machine 101, 201 Main operation control means 102, 202 Storage means 103, 203 Communication means 111, 204, 511 Query processing unit 112, 513 Data arrangement information management unit 113 Data structure conversion unit 121 Data arrangement information 210, 610 Table data 211-213 Value list 221-222, 231-33, 621-622, 631-632 Value ID table

Claims (7)

A distributed memory database system comprising a front database server and a plurality of data nodes connected to each other,
The front database server is
A data structure conversion unit that divides table data input from the outside and generates a plurality of value ID tables in which each actual data is replaced with a value ID, and distributes and stores these in each data node;
Based on a query including an aggregate function issued from an external client machine, the data node is inquired about the number of occurrences of a specific value ID in the table data, and the data node returns a response accordingly. A query processing unit that calculates a value of a set function corresponding to the query from the number of occurrences of the specified value ID and returns the value to the client machine,
When the data structure conversion unit generates the plurality of value ID tables, the plurality of value IDs individually for each of a plurality of data items designated in advance as a column that can be an aggregation axis in the table data. A distributed memory database system characterized by generating a table. When the data structure conversion unit of the front database server generates the plurality of value ID tables, each data item designated in advance as an aggregation axis and a column that can be an aggregation target in the table data is actually A value list indicating the correspondence between the value ID and the actual value;
The front database server includes a data arrangement information management unit that stores in advance storage means in which the value list and the value ID table are distributed among the data nodes. The distributed memory database system according to claim 1.   3. The data structure conversion unit of the front database server, when generating the plurality of value lists, sorts the actual values in order of magnitude and then replaces them with the value IDs. The distributed memory database system described.   The column that can be the aggregation axis for each data item that is specified in advance by the data structure conversion unit of the front database server as a column that cannot be either the aggregation axis or the aggregation target in the table data The distributed memory database system according to claim 2, wherein the distributed memory database system is stored in association with any one of the value ID tables. A front database server interconnected with a plurality of data nodes to constitute a distributed memory database system,
A data structure conversion unit that divides table data input from the outside and generates a plurality of value ID tables in which each actual data is replaced with a value ID, and distributes and stores these in each data node;
Based on a query including an aggregate function issued from an external client machine, the data node is inquired about the number of occurrences of a specific value ID in the table data, and the data node returns a response accordingly. A query processing unit that calculates a value of a set function corresponding to the query from the number of occurrences of the specified value ID and returns the value to the client machine,
When the data structure conversion unit generates the plurality of value ID tables, the plurality of value IDs individually for each of a plurality of data items designated in advance as a column that can be an aggregation axis in the table data. A front database server characterized by generating a table. In a distributed memory database system configured by connecting a front database server and a plurality of data nodes to each other,
The data structure conversion unit of the front database server accepts input of table data from the outside,
The input table data is divided into a plurality of data items specified in advance as columns that can be aggregated axes, and the data structure conversion unit of the front database server individually divides and replaces actual data with value IDs. Generate a value ID table,
The data structure conversion unit of the front database server distributes and stores the generated plurality of value ID tables in the data nodes,
The query processing unit of the front database server accepts a query including a set function issued from an external client machine,
Based on the accepted query, the query processing unit of the front database server queries each data node for the number of occurrences of the specific value ID in the table data,
Data in which the query processing unit of the front database server calculates the value of the set function corresponding to the query from the number of occurrences of the specific value ID returned from each data node and returns it to the client machine Processing method. In a distributed memory database system configured by connecting a front database server and a plurality of data nodes to each other,
In the computer provided in the front database server,
Procedure to accept external table data input,
A procedure for generating a plurality of value ID tables in which the input data is individually divided for each data item designated in advance as a column that can serve as an aggregation axis, and actual data is replaced with value IDs;
A procedure for distributing and storing the generated plurality of value ID tables in each data node;
A procedure for accepting a query including a set function issued from an external client machine,
A procedure for inquiring each data node about the number of occurrences of a specific value ID in the table data based on the accepted query;
A data processing program for executing a procedure for calculating a value of a set function corresponding to the query from the number of appearances of a specific value ID returned from each data node and returning the value to the client machine.

Images