JP6480495B2 - Data management apparatus, data management method, and program - Google Patents

Description

  The present invention relates to a data management device, a data management method, and a program.

  Conventionally, a method for detecting names for Chinese, Japanese, and Korean languages is known (see Patent Document 1). This method deals with structured data.

  By the way, the data structure in the database includes a row-oriented data structure and a column-oriented data structure. A row-oriented data structure is a data structure that holds one record as a group of logical structures. On the other hand, a column-oriented data structure is known. The column-oriented data structure is a data structure that holds data corresponding to the same index (name, age, sex, etc. in the case of user attribute data) as a group of logical structures. The logical structure means a key, an LBA (Logical Block Addressing), a label on a logical / physical conversion table, and other logical information used when searching for data. The row-oriented data structure is easy to add or delete data, while the column-oriented data structure is suitable for statistical processing for each index.

JP 2013-109364 A

  Here, a function such as JSON that handles row-oriented data structures can automatically generate a tree structure of data, but the cost is high in terms of network, storage device, and software processing. In particular, the processing time for reading data for statistical processing from a database having a column-oriented data structure is long.

  On the other hand, when data is stored in a column-oriented data structure, it is difficult to manage all indexes that can be employed and to add or delete data. In particular, when data is input in the Stream format, it is assumed that the data is processed for each record, but cannot be directly written in a column-oriented manner from the process for each record. Further, in the column-oriented type, an effective method for performing management at the time of writing failure and deduplication has not been developed.

  The present invention has been made in view of such circumstances, and saves unstructured input data in a form that can be read at high speed for each data item, and easily adds or deletes input data. Another object is to provide a data management device, a data management method, and a program that can be used.

  According to one aspect of the present invention, an interpretation unit that interprets an input data storage request and converts it into an abstract representation, and a column-oriented type in which data that has been converted into an abstract representation by the interpretation unit is not secured in advance. And a conversion unit that stores the data in the storage unit while managing the data structure.

  According to one aspect of the present invention, unstructured input data can be stored in a manner that can be read at high speed for each data item, and input data can be easily added or deleted.

It is a figure which shows an example of the log by a json format. FIG. 2 is a diagram representing the log of FIG. 1 in a tree structure. It is a figure which shows another example of the log by a json format. FIG. 4 is a diagram representing the log of FIG. 3 in a tree structure. It is the figure which expressed the data of the columner file by the tree structure. It is a figure which shows an example of the use environment and structure of the database server 100 which is an example of a data management apparatus. It is a figure for demonstrating the function of the interpretation part. FIG. 6 is a diagram (part 1) for describing a function of a conversion unit. It is a figure (example 1) which shows an example of the schema information 154B. FIG. 6 is a diagram (No. 2) for describing a function of a conversion unit. It is a figure (the 2) which shows an example of the schema information 154B. FIG. 11 is a third diagram for explaining the function of the conversion unit 114; FIG. 13 is a third diagram illustrating an example of schema information 154B. 6 is a flowchart illustrating an example of a flow of processing executed by a conversion unit 114. It is a figure which shows the image of the output data by the data user interface. 4 is a flowchart showing an example of a flow of processing executed by a data user interface 120. FIG. 10 is a diagram for explaining another function by the conversion unit 114. It is a figure which shows an example of the schema information 154B corresponding to the example of FIG.

  Hereinafter, embodiments of a data management device, a data management method, and a program according to the present invention will be described with reference to the drawings. The data management apparatus is an apparatus that stores data received from a client in a storage device, and reads and provides data in response to a request from a data transmission source client or another client from the storage device. The data management apparatus may be referred to as a DBMS (database management system). The client includes an application server (hereinafter referred to as a front-end server) that operates in cooperation with an application program that operates on a terminal device used by an end user, a data user server that uses accumulated data as statistical data, and the like. Is included.

  First, a conceptual aspect of the present invention will be described. In recent years, Hadoop is mainly accessing RDB-like hdfs with "SQL on Hadoop" represented by hive and presto, and directly handling "unstructured large amount of data" files that were said in the past Has become rare. On the other hand, the stored data is mostly an unstructured “log” at the time of acquisition. Therefore, in many cases, data is acquired and processed as “regular unstructured data”. The representative of “regular unstructured data” is json or xml, which can be expressed in “key value format including nest”, which can be viewed as a tree structure. FIG. 1 is a diagram showing an example of a log in json format, and FIG. 2 is a diagram representing the log of FIG. 1 in a tree structure. The representation by tree structure is suitable for the abstraction of "key value format including nesting". FIG. 3 is a diagram showing another example of the log in the json format, and FIG. 4 is a diagram expressing the log of FIG. 3 in a tree structure.

  As shown in Fig. 4, "keyvalue including nest" is (x, z) plane, the dimension can be expanded in the y direction with respect to the array, and the tree structure in multidimensional space includes "nest includes" You can see that it is suitable for expressing "keyvalue format", that is, schema. “Schemaobject” is an object obtained by separating this “tree structure in a multidimensional space” from the data format (json, xml, avro, message pack, etc.) and abstracting it.

  On the other hand, distributed storage represented by Hadoop was originally designed and developed to access a large amount of unstructured data with high throughput and high latency, but in recent years it has achieved high throughput and low latency. For this reason, cases where data is structured and arranged are increasing. When structuring on hdfs, a file format called columnar that persists RDB-like data is common, and typical examples include hive ORC file and apache parquet. If the data of the columnar file is expressed in a tree structure, it can be drawn with a “two-dimensional tree having only a hierarchy directly under the root” as shown in FIG. FIG. 5 is a diagram representing the data of the columner file in a tree structure.

  The advantage of the columnar file format is “cost savings by accessing each column”, and outperforms other unstructured data file formats that are familiar to Hadoop in terms of memory, CPU, and IO. On the other hand, the columnar file has a weakness of "force data structuring". As described above, data is an unstructured “log” at the time of acquisition, and a high-level expression “multi-dimensional tree structure” is impossible even if it is structured.

  The system adopted in the present invention solves this problem. This is a file format that can perpetuate data having a multidimensional spread (in the depth direction in the case of a tree structure). Since the schemaobject described above is used as it is, the data can be retained while maintaining the expressive power of "key value array including nesting".

  In general, the weakness of the columnar format of data is the "data structuring" part, and it has become necessary to implement logic and processing somewhere that compresses multidimensional data into two dimensions. Is a common “schema”. Schema management and modification is costly. In the method of the present invention, since dimension compression processing is not required, data storage frees you from this “schema problem”. In addition, access to a “specific value” of an array or Struct type, which is impossible in the structure of a columner file, has a great advantage in that it can be accessed as a tree search without fully expanding the column.

  Hereinafter, a specific configuration and function will be described. FIG. 6 is a diagram illustrating an example of the usage environment and configuration of the database server 100 which is an example of the data management apparatus. One or more terminal devices 10 used by the end user communicate with the front-end server 20. In the terminal device 10, an application program runs and transmits / receives data necessary for executing the application program to / from the front-end server 20. The front-end server 20 transmits data that needs to be saved among the data acquired from the terminal device 10 to the database server 100 via the proxy server 30 for storage. The front-end server 20 reads data necessary for executing the application program from the database server 100 and transmits the data to the terminal device 10. There are a plurality of such combinations of one or more terminal devices 10 and the front-end server 20. Each front-end server 20 makes a data write request or a data read request to the database server 100 in an arbitrary format such as JSON (JavaScript (registered trademark) Object Notation) or MySQL.

  On the other hand, the data user server 50 acquires, from the database server 100, data that is permitted to be used for statistical processing or the like according to the usage agreement among the data collected from the front end server 20. The distinction between the front end server 20 and the data user server 50 is not necessarily strict, and a part of the front end server 20 may operate as the data user server 50. Further, the data user server 50 may communicate with the database server 100 via the proxy server 30. Each device shown in FIG. 6 is connected to be communicable with each other via a network such as the Internet, a WAN (Wide Area Network), and a LAN (Local Area Network).

  The database server 100 includes a front-end interface 110, a data user interface 120, and a storage unit 150, in addition to a communication interface such as a network interface card (NIC) (not shown). The front-end interface 110 and the data user interface 120 are each realized by a processor (CPU) or the like executing a program (software). One or both of these functional units may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or software. It may be realized by cooperation of hardware.

  The front end interface 110 includes, for example, an interpretation unit 112 and a conversion unit 114. The interpretation unit 112 abstracts data acquired from the front-end server 20. Further, the interpretation unit 112 converts the abstracted data into a format corresponding to the front end server 20 when providing the data to the front end server 20. The conversion unit 114 converts the row-oriented data into column-oriented data and causes the storage unit 150 to store the data in the storage unit 150. The data user interface 120 reads data corresponding to the request acquired from the data user server 50 from the storage unit 150 and transmits the data to the data user server 50. Details of these functions will be described later.

  The storage unit 150 includes, for example, a cache memory 152 and a nonvolatile memory 154. The cache memory 152 is realized by a RAM (Random Access Memory), a register, a flash memory, or the like. The nonvolatile memory 154 is realized by an HDD (Hard Disk Drive), a flash memory, or the like. The non-volatile memory 154 stores client data 154A and schema information 154B. The schema information 154B is an example of reference information. The storage unit 150 may be a NAS (Network Attached Storage) that can be accessed by the database server 100 via a network.

[Front-end interface]
Hereinafter, functions of the front end interface 110 will be described. The interpretation unit 112 of the front end interface 110 converts data having different definitions for each front end server 20 into one common format. FIG. 7 is a diagram for explaining the function of the interpretation unit 112. Here, data indicating that the age (age) of a user having a user name (name) of aaaaaa is 24 years old is shown. Hereinafter, information such as name and age is referred to as an index. In contrast, the lower diagram of FIG. 7 schematically shows abstracted data that can be handled by the database server 100. As illustrated in FIG. 7, the interpretation unit 112 interprets the data storage request acquired from the front end server 20, performs an abstraction process, and passes the data to the conversion unit 114.

  The data abstracted by the front-end interface 110 usually has a row-oriented data structure unless special processing is applied. The conversion unit 114 further converts the abstracted data into a column-oriented data structure, and stores the converted data in the nonvolatile memory 154 of the storage unit 150 as client data 154A.

  FIG. 8 is a diagram (No. 1) for explaining the function of the conversion unit 114. Here, it is assumed that three records 1 to 3 are acquired from the front-end server 20 and abstracted by the interpretation unit 112. The record 1 includes id (identification information), name (user name), and sex (gender) as data items. In the figure, id (1) is written in the meaning of the id of record 1. The same applies to other data items. The record 2 includes id, name, and age (age) as data items, and the record 3 includes id and name as data items. These abstracted records are stored in the cache memory 152.

  When a certain amount of data is stored in the cache memory 152, the conversion unit 114 stores them in the nonvolatile memory 154 while managing them in a column-oriented data structure in which the arrangement is not secured in advance. In the example of FIG. 8, [id (1), id (2), id (3)], [name (1), name (2), name (3)], [sex (1)], and [age Each of (2)] is managed so as to have a group of logical structures.

  Here, the conversion unit 114 stores the schema information 154B in the nonvolatile memory 154 so that the search in the row direction can be performed later (may be temporarily stored in the cache memory 152). FIG. 9 is a diagram (part 1) illustrating an example of the schema information 154B. As shown in the figure, the schema information 154B is information representing a log in a tree structure. Based on this information, for example, when a request for reading the information of the record 2 is acquired from the front-end server 20, it can be accessed as a tree search. The format of the schema information 154B is not limited to that shown in FIG. 9, and an arbitrary format may be adopted.

  When a request for storing another record is acquired, the conversion unit 114 manages data by the following method. The conversion unit 114 may manage data by adding (method 1) to the already managed data structure, or (method 2) managing data every time data is transferred from the cache memory 152 to the nonvolatile memory 154. Data may be segmented.

(Method 1)
FIG. 10 is a diagram (No. 2) for describing the function of the conversion unit 114. Here, it is further assumed that three records 4 to 6 are acquired from the front-end server 20 and abstracted by the interpretation unit 112. The record 4 includes name, age, and job as data items. Further, the record 5 includes id, name, and sex as data items, and the record 6 includes id, name, and job as data items. These abstracted records are stored in the cache memory 152.

  When a certain amount of data is stored in the cache memory 152, the conversion unit 114 stores them in the nonvolatile memory 154 while managing them with a column-oriented data structure. Here, the records 4 to 6 include a data item called job (profession) that was not included in the records 1 to 3. In this case, the conversion unit 114 sets a new column and manages data. In the example of FIG. 10, [id (1), id (2), id (3), id (5), id (6)], [name (1), name (2), name (3), name (4), name (5), name (6)], [sex (1), sex (5)], [age (2), age (4)], and [job (4), job (6) ] Are managed so as to have a group of logical structures. In this case, the schema information 154B has contents as shown in FIG. FIG. 11 is a diagram (part 2) illustrating an example of the schema information 154B.

(Method 2)
FIG. 12 is a diagram (No. 3) for explaining the function of the conversion unit 114. Here, the process for the records 1 to 3 illustrated in FIG. 12 is referred to as session 1, and the process for the subsequently acquired records 4 to 6 is referred to as session 2. In the example of FIG. 12, as session 1, [id (1), id (2), id (3)], [name (1), name (2), name (3)], [sex (1)] , And [age (2)] are managed so as to have a set of logical structures, and as session 2, [id (5), id (6)], [name (4), name ( 5), name (6)], [sex (5)], [age (4)], and [job (4), job (6)] are managed so as to have a single logical structure. The In this case, the schema information 154B has contents as shown in FIG. FIG. 13 is a third diagram illustrating an example of the schema information 154B.

  By managing the data in this way, for example, when a request such as “I want to acquire jobs for all users” is acquired from the data user server 50, the database server 100 allows other data items (id, name, age, etc.) to be acquired. , Sex,...) Can be read without referring to the data item “job”. As a result, it is possible to shorten the time required for reading and quickly respond to the need for data utilization. When the nonvolatile memory 154 is an HDD, it is preferable to hold a group of logical structures in, for example, the same track so that the seek time is shortened. However, the present invention is not limited to this. .

  FIG. 14 is a flowchart illustrating an example of the flow of processing executed by the conversion unit 114. First, the conversion unit 114 waits until the write timing to the nonvolatile memory 154 arrives (S100). As described above, the write timing to the non-volatile memory 154 includes the timing at which a certain amount of data is stored in the cache memory 152, the timing at which the database server 100 is shut down, the timing at which the most recent aggregation processing is requested, etc. Can be arbitrarily defined.

  When the write timing to the nonvolatile memory 154 arrives, the conversion unit 114 selects one record stored in the cache memory 152 (S102), and selects one data item included in the record (S104). Then, the conversion unit 114 determines whether or not the selected data item is an already managed data item (S106).

  When the selected data item is a data item that has already been managed, the conversion unit 114 adds data to the end of the data item (S108). On the other hand, if the selected data item is not a data item that has already been managed, the conversion unit 114 newly sets (defines) a column and writes data in the set column (S110). Further, the conversion unit 114 adds information related to the written data item to the schema information 154B (S112).

  Next, the conversion unit 114 determines whether or not all data items of the selected record have been selected (S114). If all data items of the selected record have not been selected, the process returns to S104. When all the data items of the selected record are selected, the conversion unit 114 determines whether all the records stored in the cache memory 152 have been selected (S116). If all the records stored in the cache memory 152 have not been selected, the process returns to S102. When all the records stored in the cache memory 152 are selected, the processing of one routine of this flowchart is completed.

[Data user interface]
Hereinafter, functions of the data user interface 120 will be described. The data user interface 120 provides tabular data (array data) in response to a request from the data user server 50, for example. The request from the data user server 50 is performed by designating an arbitrary data item. At this time, for the record that does not include the designated data item, the data user interface 120 displays the data corresponding to the data item in an arbitrary form indicating “no corresponding data” such as “null” (or blank). Tabular data may be generated and provided to the data user server 50. The data user interface 120 does not return an error when the specified data item is not in the already managed data items, but instead returns all the data for the data item to “null” (or blank, etc.). The data may be in any form indicating “no corresponding data”) and is provided to the data user server 50. The request from the data user server 50 may be made by designating a predetermined extension, for example.

  For example, when data as shown in FIG. 10 is stored in the nonvolatile memory 154 as the client data 154A, a data request specifying a data item [sex, age, job, hobby] is given. . In this case, an image of output data by the data user interface 120 is as shown in FIG. FIG. 15 is a diagram illustrating an image of output data by the data user interface 120. As shown in the figure, the output data from the data user interface 120 is data that is expressed by arranging data for each record and for each data item regardless of the presence or absence of the data. Thereby, the database server 100 can provide data in a format according to the needs of the data user server 50.

  FIG. 16 is a flowchart showing an example of the flow of processing executed by the data user interface 120. First, the data user interface 120 stands by until a data request is acquired (S200). When the data request is acquired, the data user interface 120 acquires the maximum number of records at the current time from the schema information 154B (S202). Let this maximum number be n. Next, the data user interface 120 defines an array of the number of data items × n included in the data request (S204). This array is the framework for output data.

  Next, the data user interface 120 selects one data item from the data request (S206), and determines whether the selected data item has already been set in the client data 154A (S208). If the selected data item is not already set in the client data 154A, the data user interface 120 sets all data of the data item to null (S210).

  On the other hand, if the selected data item has already been set in the client data 154A, the data user interface 120 reads one data item of the currently selected data item from the client data 154A (S212). Next, the data user interface 120 determines whether there is no data that can be read in S212 (S214). If there is data that can be read in S212, the data user interface 120 determines whether or not the record number has been skipped before the data is read (S216). When the record number is skipped, the data user interface 120 sets the skipped record number data to null (S218). Then, the data user interface 120 includes the data read from the client data 154A in the array set in S204 (S220).

  After performing the process of S210 or obtaining a positive determination in S214, the data user interface 120 determines whether or not all data items have been selected during the repeated S206 (S222). ). If not all data items have been selected, the process returns to S206. On the other hand, when all the data items are selected, data is output (S224). At this stage, the data read from the client data 154A or null should be stored in all the data in the array.

  The client data 154A managed by the database server 100 may include hierarchical data in one data item. For example, there is a data item corresponding to a plurality of data such as hobby. In this case, data as shown in FIG. FIG. 17 is a diagram for explaining another function of the conversion unit 114. In the figure, hobby (1-h) indicates the h-th data of the data item hobby in the record 1. Similarly, the conversion unit 114 manages these with a column-oriented data structure, and leaves the schema information 154B as data included in the same record. 18 is a diagram illustrating an example of schema information 154B corresponding to the example of FIG.

  According to the data management device, data management method, and program of the present invention described above, the interpretation unit 112 that interprets an input write request and converts it into an abstract representation, and the data that is converted into an abstract representation by the interpretation unit 112 Is converted into a column-oriented data structure in which an array is not secured in advance and stored in the storage unit 150, so that unstructured input data is read at high speed for each data item. While saving in a possible manner, it is possible to easily add or delete input data.

  As mentioned above, although the form for implementing this invention was demonstrated using embodiment, this invention is not limited to such embodiment at all, In the range which does not deviate from the summary of this invention, various deformation | transformation and substitution Can be added.

DESCRIPTION OF SYMBOLS 10 Terminal apparatus 20 Front end server 30 Proxy server 50 Data user server 100 Database server 110 Front end interface 112 Interpretation part 114 Conversion part 120 Data user interface 150 Storage part 152 Cache memory 154 Non-volatile memory 154A Client data 154B Schema information

Claims (7)

An object that interprets the input data storage request, abstracts the tree structure in multidimensional space from the data format including json, xml, avro, or message pack, and can expand the dimension to the array An interpreter to convert to
The data converted into the object is converted into client data managed in a column-oriented data structure having a data structure different from that of the converted data, and an array is not secured in advance, and stored in the storage unit. causes stored, and the included in the object, conversion unit for storing reference information for the data managed by the data structure of the column-oriented reading is enabled in the row direction in the memory unit,
A data management device comprising: The column-oriented data structure is a data structure in which data included in a record included in the data storage request is grouped into a logical structure for each data item,
The conversion unit adds a new column when the data item included in the data storage request is a data item not set in the column-oriented data structure;
The data management apparatus according to claim 1. A data user interface that reads and outputs data managed in the column-oriented data structure from the storage unit for each data item included in the input data request;
The data management apparatus according to claim 1 or 2. The data user interface fills the data corresponding to the data item with any form of data indicating that the corresponding data does not exist for a record that does not include the specified data item.
The data management apparatus according to claim 3. When the specified data item is a data item not set in the column-oriented data structure, the data user interface indicates that there is no corresponding data for all the data items. Fill in any form of data,
The data management apparatus according to claim 3 or 4. Computer
An object that interprets the input data storage request, abstracts the tree structure in multidimensional space from the data format including json, xml, avro, or message pack, and can expand the dimension to the array Converted to
The data converted into the object is converted into client data managed in a column-oriented data structure having a data structure different from that of the converted data, and an array is not secured in advance, and stored in the storage unit. causes stored, the included in the object, and stores the reference information for the data managed by the data structure of the column-oriented reading is enabled in the row direction in the memory unit,
Data management method. On the computer,
An object that interprets the input data storage request, abstracts the tree structure in multidimensional space from the data format including json, xml, avro, or message pack, and can expand the dimension to the array Converted to
The data converted into the object is converted into client data managed in a column-oriented data structure having a data structure different from that of the converted data, and an array is not secured in advance, and stored in the storage unit. Storing the reference information for allowing the data managed in the column-oriented data structure included in the object to be read in a row direction, and storing the reference information in the storage unit.
program.

Images