JP6668513B2 - Data management method and computer - Google Patents

Description

  The present invention relates to data processing in a Data Lake system.

  2. Description of the Related Art In the field of big data, a Data Lake method for integrally managing data of various data structures such as RDB data, JSON data, and tabular data is known (for example, see Patent Document 1). Here, Data Lake indicates a repository that stores a wide variety of data without depending on the type of data, the size of data, and the like.

US Patent Application Publication No. 2014/0280333 US Patent Application Publication No. 2014/0108467 JP 2016-14944 A

  The use of the Data Lake system facilitates data management. However, when performing data search processing or the like, it is necessary to use a query in which relevance between data is defined. Therefore, there is a problem that the data search process is complicated.

  As a technique for improving the efficiency of data search processing, a Data Discovery tool that can be analyzed by a simple operation has been attracting attention (for example, see Patent Documents 2 and 3). Patent Literature 2 describes that data can be retrieved with a simple query by extracting the relevance between different data tables. Patent Document 3 discloses a correlation rule analyzer that extracts data dependencies, constraints, and the like of DB columns of a database from data stored in the database. A means for extracting information on the simultaneous occurrence relationship of data between columns as an association rule, a means for aggregating the extracted association rules based on a specific commonality, and a method for extracting data from the appearance frequency and combination of each aggregated association rule It has means for calculating a usefulness index as a dependency or a constraint. "

  In order to achieve both data management and simplification of search processing, it is conceivable to combine Patent Document 1 with Patent Document 2 or Patent Document 3. However, in the technology described in Patent Literature 2, the relevance between data is extracted based on the commonality of vocabulary. Input of ETL (Extract Transform Load) logic is required. The technique described in Patent Document 3 is based on the premise that the database is an RDB (Relational Database). Further, the invention realized by the above-described combination does not describe using information such as an association rule for a query.

  SUMMARY OF THE INVENTION It is an object of the present invention to facilitate integrated management and search processing of data in which the relationship between data is not defined.

  A typical example of the invention disclosed in the present application is as follows. That is, a data management method of at least one computer configuring a data store, wherein the at least one computer has a processor, a memory connected to the processor, and a network interface connected to the processor, In the data store, system information including a plurality of data blocks each including at least one attribute is stored in a format independent of a data structure of the system information. The memory stores identification information of two related system information. Information and a relationship management information including a record for managing an access path defined by a correspondence relationship between attributes associated with the two pieces of system information, wherein the data management method comprises the steps of: A first step of receiving A second system in which the processor stores the new system information in the data store; and a first system in which the processor has a relationship with the new system information from among the system information stored in the data store. A third step of specifying information, wherein the processor is defined by a correspondence relationship between identification information of the new system information and the first system information and an attribute for associating the new system information and the first system information. A fourth step of registering a record including the first access path in the relevance management information, and when the processor receives a query for the data store, based on the relevance management information, To a query to which information for making a reference between the system information is added , Wherein said processor comprises a the steps of the sixth to execute the converted query.

  According to the present invention, in a Data Lake system, a computer specifies a relationship between system information at the time of registration of system information, and converts a query based on the relationship between system information at the time of executing a query. This facilitates integrated management of data and facilitates data search processing. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

2 is a configuration example of a computer system according to the first embodiment. FIG. 3 is a block diagram illustrating a hardware configuration and a software configuration of the data management node according to the first embodiment. FIG. 4 is a diagram illustrating an example of site data according to the first embodiment. FIG. 4 is a diagram illustrating an example of site data according to the first embodiment. FIG. 4 is a diagram illustrating an example of site data according to the first embodiment. FIG. 4 is a diagram illustrating an example of business data according to the first embodiment. FIG. 7 is a diagram illustrating an example of attribute management information according to the first embodiment. FIG. 6 is a diagram illustrating an example of attribute group management information according to the first embodiment. FIG. 6 is a diagram illustrating an example of relevance management information according to the first embodiment. 9 is a flowchart illustrating an example of a process performed when the data management module according to the first embodiment receives a registration request for system information. 9 is a flowchart illustrating an example of an attribute management information update process performed by a relevance analysis module according to the first embodiment. 9 is a flowchart illustrating an example of an attribute group management information update process executed by a relevance analysis module according to the first embodiment. 6 is a flowchart illustrating an example of a process of updating the relevance management information executed by the relevance analysis module according to the first embodiment. 9 is a flowchart illustrating an example of a process performed when the data management module according to the first embodiment receives a query. 9 is a flowchart illustrating an example of a query expansion process executed by the conversion module according to the first embodiment. FIG. 7 is a diagram illustrating an example of output data output by executing a query by the data management module according to the first embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. In the following description, information may be described in terms of “AAA table” or the like, but such information may be represented in a data structure other than a table or the like. Therefore, the "AAA table" or the like may be referred to as "AAA information" in order to show that it does not depend on the data structure. In describing the contents of each information, identification information expressed as “number” and “name” is adopted, but other types of identification information may be used. The “AAA processing” in the following description may be an “AAA program”. In the following description, the description with "processing" as the subject may be a description with the processor as the subject. Part or all of the processing may be realized by dedicated hardware. Various programs may be installed in each computer by a program distribution server or a computer-readable storage medium.

  FIG. 1 is a configuration example of a computer system 10 according to the first embodiment.

  The computer system 10 includes a site system 110, a business system 120, an analysis system 130, and a data management node 100. The site system 110, the business system 120, the analysis system 130, and the data management node 100 are connected to each other via a network switch 140.

  Note that the present invention is not limited to the type of network connecting the devices and is not limited to the type of connection between the devices. Further, two or more field systems 110, business systems 120, analysis systems 130, and data management nodes 100 may exist.

  In this embodiment, Data Lake is realized using the storage area of the data management node 100. When Data Lake is implemented using a plurality of data management nodes 100, the plurality of data management nodes 100 have a distributed cluster configuration, and data is arranged in a distributed manner. Here, Data Lake indicates a repository that stores a wide variety of data without depending on the type of data, the size of data, and the like.

  The data management node 100 includes a control program group 101 and a management information group 102. Details of the control program group 101 and the management information group 102 will be described with reference to FIG.

  The field system 110 is a system operated to achieve any purpose, and includes one or more computers. The computer executes a program for realizing the data communication module 111, and holds field data 112. The computer included in the site system 110 may hold structural information for managing the data structure of data handled by the system.

  The data communication module 111 transmits the site data 112. The site data 112 is data indicating the operation state of the site system 110 and the like. The site data 112 may be, for example, data in a table format including a plurality of records including columns of a plurality of attributes (fields). Note that the present invention is not limited to the data structure of the site data 112. For example, a data structure in a message format such as a CSV format, a list format, and sensor information may be used. Details of the site data 112 will be described with reference to FIGS. 3A, 3B, and 3C.

  The business system 120 is a system that executes business processing using data and the like of the site system 110, and includes one or more computers. The computer executes a program for realizing the data communication module 121 and holds business data 122. Note that a computer included in the business system 120 may hold structural information or the like for managing the data structure of data handled by the system.

  The data communication module 121 transmits the business data 122. Note that the data communication module 121 may transmit log data that is a result of business processing. The business data 122 is data used for business processing. The business data 122 may have a data structure of a table format, a CSV format, a list format, and a message format, like the site data 112. Details of the business data 122 will be described with reference to FIG.

  The analysis system 130 is a system that executes a predetermined analysis process using the site data 112 and the business data 122, and includes one or more computers. The computer executes a program that implements the data communication module 131, and holds the analysis data 132.

  The data communication module 131 transmits analysis data including a result of the analysis processing. The analysis data 132 is data used for the analysis processing. As in the case of the site data 112, the analysis data 132 may have a data structure of a table format, a CSV format, a list format, and a message format.

  In the data lake of the present embodiment, it is assumed that site data 112, business data 122, and analysis data 132 are stored.

  In the following description, when the information of each system stored in Data Lake is not distinguished, it is also described as system information. In this embodiment, the site data 112, the business data 122, and the analysis data 132 correspond to the system information. In the following description, data of a management unit of information stored in Data Lake is also described as a data block. For example, in the case of the site data 112 corresponding to the RDB, one record corresponds to one data block. In the following description, “attribute” indicates identification information of a column (field) that constitutes one data block. In the following description, “attribute value” indicates a specific value of “attribute”.

  Here, a hardware configuration and a software configuration of the data management node 100 will be described. FIG. 2 is a block diagram illustrating a hardware configuration and a software configuration of the data management node 100 according to the first embodiment.

  The data management node 100 has a CPU (Central Processing Unit) 201, a memory 202, a network interface 203, and a storage device 204 as a hardware configuration. The CPU 201, the memory 202, the network interface 203, and the storage device 204 are connected to each other by an internal bus or the like.

  The CPU 201 is an arithmetic unit that executes various processes of the data management node 100, and executes a program stored in the memory 202. The CPU 201 operates as a module that realizes a specific function by executing processing according to a program stored in the memory 202. In the following description, when a process is described using a module as a subject, it indicates that the CPU 201 is executing a program for realizing the module.

  The memory 202 is a main storage device of the data management node 100, and stores programs executed by the CPU 201 and various data. In this embodiment, the memory 202 stores the control program group 101 and the management information group 102. The memory 202 includes a work area used by the CPU 201 for processing.

  The network interface 203 is an interface for connecting to another device via a network. In this embodiment, the data management node 100 connects to the network switch 140 via the network interface 203.

  The storage device 204 is a secondary storage device of the data management node 100 and stores a large amount of data. The storage device 204 is, for example, an FC (Fibre Channel) disk, a SCSI (Small Computer System Interface) disk, a SATA disk, an ATA (AT Attachment) disk, a SAS (Serial Attached SCSI) disk, or the like.

  Here, the control program group 101 and the management information group 102 stored in the memory 202 will be described.

  The management information group 102 includes attribute management information 221, attribute group management information 222, relevancy management information 223, and a data store 224.

  The attribute management information 221 is information for managing the attributes of the data block included in the system information. For example, when a table including a record composed of a column group corresponding to a plurality of attributes is stored in the data store 224, information on each column is stored in the attribute management information 221. Details of the attribute management information 221 will be described with reference to FIG.

  The attribute group management information 222 is information for managing a combination of attributes used when two pieces of system information are combined. Details of the attribute group management information 222 will be described with reference to FIG.

  The association management information 223 is information for managing the association between the system information. Details of the relevance management information 223 will be described with reference to FIG.

  The data store 224 is a storage for realizing Data Lake, and stores various data handled by the site system 110, the business system 120, and the analysis system 130. Specifically, the data store 224 stores a system information group 225 including the site data 112, the business data 122, the analysis data 132, and the like. The data store 224 stores data in a data format that does not depend on the data structure of the system information. For example, a key value store may be adopted as the data store 224.

  The data store 224 of this embodiment is configured using a storage area of the memory 202 of the data management node 100. Note that the data store 224 may be configured using the storage area of the storage device 204 of the data management node 100. When there are a plurality of data management nodes 100, the data store 224 is realized by using the storage area of the memory 202 or the storage device 204 of each data management node 100.

  The control program group 101 includes programs for realizing the data management module 211, the association analysis module 212, and the conversion module 213. Although a plurality of modules are shown for simplicity of description, a plurality of modules may be combined into one module. Further, the functions may be divided into an arbitrary number of modules.

  When there are a plurality of data management nodes 100, the same program may not be stored in all data management nodes 100. For example, one data management node 100 may store a program that implements the data management module 211. The same applies to the association analysis module 212 and the conversion module 213.

  The data management module 211 manages the data store 224. Further, the data management module 211 controls a data write process, a data read process, and the like for the data store 224.

  The data management module 211 receives a registration request including the site data 112 from the site system 110, a registration request including the business data 122 from the business system 120, and a registration request including the analysis data 132 from the analysis system 130. And store it in the data store 224. Further, the data management module 211 calls the association analysis module 212.

  When receiving a search request from the business system 120, the data management module 211 calls the conversion module 213.

  The association analysis module 212 analyzes the association between the system information. The relevance analysis module 212 generates or updates the attribute management information 221, the attribute group management information 222, and the relevancy management information 223 based on the analysis result.

  The conversion module 213 converts a query for the data store 224 based on the relevance management information 223 and the like.

  Next, information included in the computer system 10 will be described.

  FIGS. 3A, 3B, and 3C are diagrams illustrating an example of the site data 112 according to the first embodiment.

  FIG. 3A shows sensor information 300 which is one of the site data 112. The sensor information 300 includes a record including a device ID 301, a sensor ID 302, a sensor type ID 303, a time 304, and a value 305.

  In the first record of the sensor information 300, the device whose device ID 301 is “EQ001” has the sensor whose sensor ID 302 is “T001” and the sensor type ID 303 is “1”, and the sensor is “2016/06 / 15:24:23 "indicates that the value measured is" 42 ".

  FIG. 3B is a diagram illustrating an example of the dictionary information 310 which is one of the site data 112. The dictionary information 310 includes a record including a sensor type ID 311 and a sensor type 312.

  The first record of the dictionary information 310 indicates that the sensor whose sensor type ID 311 is “1” is a temperature sensor that measures “temperature”.

  FIG. 3C is a diagram illustrating an example of the setting information 320 that is one of the site data 112. The setting information 320 includes a record including a sensor type 321, setting information 322, and a setting value 323.

  The first record of the setting information 320 indicates that the threshold value set for the temperature sensor is “40”.

  FIG. 4 is a diagram illustrating an example of the business data 122 according to the first embodiment.

  The device information 400 is included in the business data 122. The device information 400 includes a record including a base 401, a device ID 402, and a sensor ID 403.

  The first record of the device information 400 indicates that a device whose device ID is “EQ001” is installed at the base “Tokyo”, and that the device has a sensor whose sensor ID is “T001”. Show.

  FIG. 5 is a diagram illustrating an example of the attribute management information 221 according to the first embodiment.

  The attribute management information 221 includes a record including an ID 501, a data source name 502, an attribute name 503, a primary key 504, and a composite key 505. One record corresponds to one attribute that constitutes one data block included in the system information.

  The ID 501 is identification information for uniquely identifying a record included in the attribute management information 221. The data source name 502 is identification information of system information. The attribute name 503 is identification information of an attribute constituting a data block included in the system information.

  The primary key 504 is a value indicating whether or not the attribute is a primary key for uniquely identifying a data block included in the system information. A predetermined symbol is stored in the primary key 504 of the attribute serving as the primary key. Note that the number of attributes serving as the primary key need not be one.

  The composite key 505 is identification information of an attribute group serving as a primary key, that is, an attribute constituting the composite key. The composite key 505 stores identification information of attributes other than the attribute corresponding to the record among the attribute group including the attribute corresponding to the record. For example, if each of “C1” and “C2”, “C1” and “C3”, “C1” and “C4”, and “C1” and “C5” are composite keys, a record with ID 501 “C1” In the composite key 505, “C2”, “C3”, “C4”, and “C5” are stored, and records with IDs 501 of “C2”, “C3”, “C4”, and “C5” include: “C1” is stored.

  In the following description, when the primary key and the composite key are not distinguished, they are also described as search keys.

  FIG. 6 is a diagram illustrating an example of the attribute group management information 222 according to the first embodiment.

  The attribute group management information 222 includes a record including an attribute group ID 601 and a member 602.

  The attribute group ID 601 is identification information for uniquely identifying a record included in the attribute group management information 222.

  The member 602 is information on a combination of attributes used when combining two pieces of system information. In this embodiment, the combination of the IDs 501 of the attribute management information 221 is stored in the member 602.

  The first record of the attribute group management information 222 shown in FIG. 6 indicates that data blocks having the same value of “C3” and the value of “C6” are to be combined. The third record indicates that data blocks having the same value of “(C1, C2)” and the value of “(C12, 13)” are combined.

  FIG. 7 is a diagram illustrating an example of the relevance management information 223 according to the first embodiment.

  The association management information 223 includes a record including a first data source name 701, a second data source name 702, and an access path 703.

  The first data source name 701 and the second data source name 702 are identification information of relevant system information. The first data source name 701 is identification information of system information that can be accessed based on the foreign key included in the system information corresponding to the second data source name 702. The second data source name 702 is identification information of system information that can be accessed based on a foreign key included in the system information corresponding to the first data source name 701. The same value as the value set in the data source name 502 is set in the first data source name 701 and the second data source name 702.

  The access path 703 is path information defined by the correspondence between attributes that associate two pieces of system information. In this embodiment, there are three types of paths: an attribute matching path, an attribute group matching path, and a springboard matching path.

  The attribute matching path is a path defined by the correspondence between one attribute included in one system information and one attribute included in the other system information. In the case of the attribute matching path, the access path 703 stores information indicating the correspondence between one attribute included in one system information and one attribute included in the other system information. An attribute included in the reference system information is used as a foreign key.

  The first and second records in FIG. 7 are records corresponding to the attribute matching path. For example, the first record indicates that a data block in which the value of C6 matches the foreign key can be searched using the value of C3 as the foreign key.

  The attribute group matching path is a path defined by a correspondence between a plurality of attributes included in one system information and a plurality of attributes included in the other system information. In the case of the attribute group matching path, the access path 703 stores information indicating the correspondence between the attribute group included in one system information and the attribute group included in the other system information.

  The third record in FIG. 7 is a record corresponding to the attribute group matching path. The third record indicates that a data block in which the combination of the values of C12 and C13 matches the foreign key can be searched using the combination of the values of C1 and C2 as the foreign key.

  The springboard matching path is a correspondence between the attribute included in one system information and the attribute included in the relay system information, and the correspondence between the attribute included in the relay system information and the attribute included in the other system information. A path defined by a relationship. In the case of a springboard-matching path, two or more paths are connected. In the present embodiment, connections of different paths are represented using semicolons. The path connected using a semicolon may be either an attribute matching path or an attribute group matching path. Note that the attribute connected via the semicolon is an attribute included in the same system information.

  The fourth record in FIG. 7 is a record corresponding to the springboard matching path. The fourth record uses the value of C3 as the first foreign key, searches for the first data block in which the value of C6 matches the first foreign key, and retrieves the value of C7 in the first data block. Used as a second foreign key, indicates that a second data block whose C8 value matches the second foreign key can be retrieved. The attributes C6 and C7 connected via the semicolon are attributes included in the same system information (dictionary information 310).

  FIG. 8 is a flowchart illustrating an example of a process performed when the data management module 211 according to the first embodiment receives a registration request for system information.

  The data management module 211 receives a registration request for system information from an arbitrary system (step S801). In the following description, system information to be registered based on a registration request is also described as target system information.

  The data management module 211 stores the target system information included in the registration request in the data store 224 (Step S802). Note that the registration request may include a data block. Since the method of registering data in the data store 224 is a known technique, a detailed description is omitted.

  The data management module 211 determines whether a record related to the target system information is registered in the relevance management information 223 (Step S803).

  Specifically, the data management module 211 determines whether there is a record in which the identification information of the target system information is set in at least one of the first data source name 701 and the second data source name 702. If the above-mentioned record does not exist, the data management module 211 determines that the record related to the target system information is not registered in the relevance management information 223.

  When it is determined that the record related to the target system information is registered in the relevance management information 223, the data management module 211 ends the processing.

  When it is determined that the record related to the target system information is not registered in the relevance management information 223, the data management module 211 instructs the relevancy analysis module 212 to update the attribute management information 221 (step S804). Details of the process of updating the attribute management information 221 will be described with reference to FIG.

  When the data management module 211 receives a response indicating the completion of the process from the relevance analysis module 212, it instructs the relevance analysis module 212 to update the attribute group management information 222 (step S805). Details of the process of updating the attribute group management information 222 will be described with reference to FIG.

  When the data management module 211 receives a response indicating the completion of the process from the relevance analysis module 212, it instructs the relevancy analysis module 212 to update the relevancy management information 223 (step S806). Details of the relationship management information 223 will be described with reference to FIG.

  When the data management module 211 receives a response indicating the completion of the process from the association analysis module 212, the process ends.

  If no system information is registered in the data store 224, the data management module 211 generates initialized attribute management information 221 in step S804, and initializes the initialized attribute group management information 222 in step S805. Is generated, and in step S806, the initialized relationship management information 223 is generated.

  FIG. 9 is a flowchart illustrating an example of an update process of the attribute management information 221 performed by the association analysis module 212 according to the first embodiment.

  The relevance analysis module 212 specifies an attribute included in the target system information (Step S901).

  Specifically, the association analysis module 212 analyzes the data structure of the data body part of the target system information, and specifies the attributes included in the data body part of the target system information. Further, the association analysis module 212 analyzes the metadata of the target system information, and acquires the identification information of the target system information from the metadata of the target system information. At this time, the association analysis module 212 may also specify the attribute of the metadata.

  For example, a method of analyzing the data structure of the target system information based on the schema definition information of the target system information can be considered. The present embodiment does not depend on a method of analyzing the data structure of the target system information.

  Next, the association analysis module 212 adds the specified attribute to the attribute management information 221 (step S902).

  Specifically, the association analysis module 212 adds records by the number of attributes specified in the attribute management information 221 and sets the record identification information in the ID 501 of the added record. The association analysis module 212 sets the identification information of the target system information in the data source name 502 of the added record. The relevance analysis module 212 sets the identification information of the specified attribute in the attribute name 503 of each added record.

  Next, the association analysis module 212 determines whether or not a primary key exists in the target system information (step S903).

  Specifically, the association analysis module 212 refers to the attribute value of each attribute included in the target system information, and specifies an attribute that can uniquely identify the data block. For example, the relevance analysis module 212 specifies, as a primary key, an attribute having no duplicate attribute value.

  For example, when the setting information 320 is the target system information, since the attribute values of the sensor type 321 and the setting value 323 do not overlap, each of the sensor type 321 and the setting value 323 is specified as the primary key.

  If it is determined that the primary key exists in the target system information, the relevance analysis module 212 sets a value indicating the primary key in the primary key 504 of the record of the attribute to be the primary key (step S904). Thereafter, the association analysis module 212 outputs a response indicating the completion of the process to the data management module 211, and ends the process.

  When it is determined that the primary key does not exist in the target system information, the association analysis module 212 specifies the composite key (step S905).

  Specifically, the association analysis module 212 generates a combination of attributes and specifies a combination of attributes that can uniquely identify the data block. For example, the association analysis module 212 specifies, as a composite key, an attribute combination in which no duplicate attribute value combination exists.

  For example, when the sensor information 300 is the target system information, a record can be uniquely identified by a combination of the device ID 301 and another attribute, so that the combination of the device ID 301 and another attribute is specified as a composite key. In the present embodiment, it is assumed that a composite key that minimizes the number of attributes included in the combination of attributes is specified. Therefore, the device ID 301 and the sensor ID 302, the device ID 301 and the sensor type ID 303, the device ID 301 and the time 304, and the device ID 301 and the value 305 are specified as the composite keys.

  Next, the relevance analysis module 212 sets a value to the composite key 505 of the record of the attribute constituting the composite key (step S906). Thereafter, the association analysis module 212 outputs a response indicating the completion of the process to the data management module 211, and ends the process.

  Specifically, the relevance analysis module 212 sets the identification information of the attribute other than the attribute corresponding to the record in the composite key 505 of the attribute record forming the composite key. By the above processing, attribute management information 221 as shown in FIG. 5 is generated.

  It should be noted that the system information registration request may include information indicating an attribute serving as a primary key or a composite key. In this case, the association analysis module 212 updates the attribute management information 221 based on the information.

  FIG. 10 is a flowchart illustrating an example of an update process of the attribute group management information 222 performed by the association analysis module 212 according to the first embodiment.

  The relevance analysis module 212 selects one system information to be compared with the target system information from among the system information registered in the data store 224 (step S1001). In the following description, the system information to be compared with the target system information is also described as sub-target system information.

  For example, the association analysis module 212 selects sub-target system information based on the data source name 502 of the attribute management information 221.

  Next, the association analysis module 212 compares the search keys of the target system information and the sub-target system information (step S1002). Specifically, the following processing is executed.

  The relevance analysis module 212 determines whether each search key of the target system information and the sub-target system information is the same type of search key. That is, when the search key of the target system information is the primary key, the association analysis module 212 determines whether the search key of the sub-target system information is the primary key.

  When it is determined that the search keys of the target system information and the sub-target system information are not the same type of search key, it is determined that the search keys of the respective system information do not match. In this case, the relevance analysis module 212 proceeds to step S1006 without executing the processing from step S1003 to step S1005.

  When it is determined that the respective search keys of the target system information and the sub-target system information are the same type of search key, the association analysis module 212 sets the name or attribute value of the attribute corresponding to the search key of each system information. Compare.

  Next, the relevance analysis module 212 determines whether the search key of each system information is the same based on the comparison result (step S1003).

  For example, a method of comparing attribute names or a method of comparing types of attribute values can be considered. When the names of the attribute values match or the types of the attribute values match, the association analysis module 212 determines that the search key of each system information is the same.

  When it is determined that the search key of each system information is not the same, the relevance analysis module 212 performs similarity determination of the attribute name or the attribute value using a thesaurus and the like, and the search key of each system information is similar. It is determined whether the key is a search key (step S1004). Note that the similar dictionary or the like may be held in advance by the data management node 100 or may be input from outside.

  If it is determined that the search keys of the respective system information are not similar, the relevance analysis module 212 proceeds to step S1006.

  When it is determined that the search key of each system information is the same, or when it is determined that the search key of each system information is similar, the association analysis module 212 adds a record to the attribute group management information 222 ( Step S1005). Thereafter, the relevance analysis module 212 proceeds to step S1006.

  Specifically, the relevance analysis module 212 adds a record to the attribute group management information 222 and sets identification information to the attribute group ID 601 of the added record. The relevance analysis module 212 sets a search key of each system information in the member 602 of the added record. That is, a combination of attributes that can be identified is set to the member 602. If the search key is a primary key, a value associated with the primary key of each system information is set in the member 602. When the search key is a composite key, a value corresponding to the composite key of each system information is set in the member 602.

  In step S1006, the relevance analysis module 212 determines whether the processing has been completed for all the system information registered in the data store 224 (step S1006).

  If it is determined that the processing has not been completed for all the system information registered in the data store 224, the relevance analysis module 212 returns to step S1001 and executes the same processing.

  When it is determined that the processing has been completed for all the system information registered in the data store 224, the association analysis module 212 outputs a response indicating the completion of the processing to the data management module 211, and ends the processing.

  FIG. 11 is a flowchart illustrating an example of an update process of the relevance management information 223 performed by the relevance analysis module 212 according to the first embodiment.

  The association analysis module 212 selects one piece of sub-target system information from the system information registered in the data store 224 (step S1101). The processing in step S1101 is the same as the processing in step S1001.

  Next, the relevance analysis module 212 refers to the attribute management information 221 and the attribute group management information 222 to search for records related to the target system information and the sub-target system information (step S1102).

  Specifically, the relevance analysis module 212 obtains, from the ID 501 of the attribute management information 221, identification information assigned to an attribute included in each system information. The relevance analysis module 212 refers to the member 602 of the attribute group management information 222 and stores a record in which the identification information of the attribute included in the target system information and the identification information of the attribute included in the sub-target system information are set. Search for.

  Next, the relevance analysis module 212 determines whether or not a record related to the target system information and the sub-target system information exists in the attribute group management information 222 based on the search result (step S1103).

  If it is determined that the record related to the target system information and the sub-target system information does not exist in the attribute group management information 222, the process proceeds to step S1105.

  If it is determined that a record related to the target system information and the sub-target system information exists in the attribute group management information 222, the relationship analysis module 212 adds a record to the relationship management information 223 (step S1104). Specifically, the following processing is executed.

  The relevance analysis module 212 adds a record to the relevance management information 223, sets the identification information of the target system information in the first data source name 701 of the added record, and sets the second data of the added record. The identification information of the sub target system information is set in the source name 702. Further, the relevance analysis module 212 sets a value in the access path 703 of the added record based on the value set in the member 602 of the record retrieved from the attribute group management information 222.

  Here, a specific example of the process of step S1104 will be described. Here, it is assumed that the target system information is the sensor information 300 and the sub target system information is the dictionary information 310.

  The identification information of the attributes included in the sensor information 300 is C1, C2, C3, C4, and C5, and the identification information of the attributes included in the dictionary information 310 is C6 and C7. In this case, the attribute group management information 222 includes a record in which C3 and C6 are set. Therefore, the relevance analysis module 212 sets a record in which “sensor information” is set as the first data source name 701, “dictionary information” is set as the second data source name 702, and a path in which C3 and C6 are associated with the access path 703. Is added to the relevance management information 223.

  The data management node 100 combines the records in which the attribute corresponding to C3 of the sensor information 300 and the attribute corresponding to C6 of the dictionary information 310 match based on the record added by the above-described processing. Two system information can be combined. The above is the description of the process in step S1104.

  If the decision result in the step S1103 is NO, or after the process in the step S1104 is completed, the relevance analysis module 212 decides whether or not the process has been completed for all the system information registered in the data store 224. (Step S1105).

  If it is determined that the processing has not been completed for all the system information registered in the data store 224, the relevance analysis module 212 returns to step S1101 and executes the same processing.

  If it is determined that the processing has been completed for all the system information registered in the data store 224, the relevance analysis module 212 starts the processing for adding the record corresponding to the springboard matching path.

  First, the relationship analysis module 212 selects one target record from records newly added to the relationship management information 223 (step S1106). For example, a method of selecting the records in order from the upper record can be considered.

  The relevance analysis module 212 specifies system information that can be accessed via one of the system information set in the target record (step S1107). Specifically, the following processing is executed.

  The relevance analysis module 212 acquires the identification information of the system information set in the first data source name 701 and the second data source name 702 of the target record, sets one system information as the starting system information, and sets the other system information as the starting system information. Set the system information of the relay system information.

  The relevance analysis module 212 searches for a record in which the identification information of the relay system information is stored in one of the first data source name 701 and the second data source name 702. The relevance analysis module 212 performs the same processing by exchanging the origin system information and the relay system information. However, only records in which a path of the same type as the type of the path set in the access path 703 of the target record is set as a search target. System information that is different from the relay system information of the retrieved record is used as the platform system information.

  By the above-described processing, the pedestal matching path can be specified. In other words, a path in which the starting point system information and the ladder system information are associated with each other via the relay system information is a ladder platform matching path. If there is no corresponding record, the relevance analysis module 212 proceeds to step S1109 without executing step S1108. The above is the description of the process in step S1107.

  The relevance analysis module 212 adds a step ladder matching path record to the relevance management information 223 based on the processing result of step S1107 (step S1108).

  Specifically, the relevance analysis module 212 adds entries as many as the number of the identified ladder matching paths. The association analysis module 212 sets the identification information of the starting system information and the stepping system information in the first data source name 701 and the second data source name 702 of each of the added entries. Further, the association analysis module 212 sets a springboard matching path in the access path 703 of each of the added entries.

  Note that the springboard matching path can be generated by connecting the attribute of the relay system information included in the access path 703 of the target record and the attribute of the relay system information included in the access path 703 of the searched record with a semicolon.

  Next, the relationship analysis module 212 determines whether or not the processing has been completed for all records newly added to the relationship management information 223 (step S1109).

  If it is determined that the processing has not been completed for all records newly added to the relevance management information 223, the relevance analysis module 212 returns to step S1106 and executes the same processing.

  When it is determined that the processing has been completed for all records newly added to the relevance management information 223, the relevance analysis module 212 outputs a response indicating the completion of the processing to the data management module 211, and ends the processing. I do.

  After performing the processing in step S1107, the relevance analysis module 212 may set the stepping system information to new relay system information and execute the same processing. Specifically, the processing may be changed as follows.

  The relevance analysis module 212 sets the first data source name 701 and the second data source name 702 to the identification information of the starting system information and the ladder system information, and sets the access path 703 to the pedestal matching path via the relay system information. Generates a temporary record with. Further, the association analysis module 212 sets the stepping system information as new relay system information, and executes the same processing as step S1107.

  If there is no corresponding record, the relationship analysis module 212 proceeds to step S1108, and adds the temporary record to the relationship management information 223 as it is. If the corresponding record exists, the relevance analysis module 212 copies the temporary records by the number of the searched records, and replaces the identification information of the system information of the searched record in each record with the identification information of the new platform system information. Register as Further, the relevance analysis module 212 updates the access path 703 of the temporary record by connecting the access path 703 of the searched record and the access path 703 of the temporary record.

  Thereafter, the same loop processing is repeatedly executed. After all the loop processing is completed, the relationship analysis module 212 adds the generated temporary record to the relationship management information 223.

  By the above-described processing, a ladder matching path for relaying a plurality of pieces of system information can be set. Note that an upper limit may be set for the number of relayed system information.

  When there are a plurality of paths between the target system information and the ladder system information, all the paths may be set as the access path 703, or the shortest path may be set as the access path 703. Good.

  Here, the processing from step S1106 to step S1109 will be described using a specific example. In the following description, it is assumed that the sensor information 300 is target system information, and only the second record is registered in the relevance management information 223. In this case, the first record and the third record are registered in the relevance management information 223 by the processing from step S1101 to step S1105.

  In step S1106, the relevance analysis module 212 selects the first record as the target record.

  In step S1107, the association analysis module 212 searches for a record using the identification information of the sensor information 300 and the identification information of the dictionary information 310 as keys. When the identification information of the dictionary information 310 is used as a key, a second record is searched. Therefore, the association analysis module 212 specifies the setting information 320 as the platform system information.

  Since the target record is a record corresponding to the attribute matching path, the third record corresponding to the attribute group matching path is not a search target.

  In step S1108, the association analysis module 212 converts the attribute C6 of the dictionary information 310 included in the access path 703 of the first record and the attribute C7 of the dictionary information 310 included in the access path 703 of the second record into a semicolon. A stepboard matching path is generated by connecting with. The relevance analysis module 212 adds a record, sets the identification information of the sensor information 300 to the first data source name 701 of the added record, and sets the identification information of the setting information 320 to the second data source name 702 of the added record. Set identification information. Further, the association analysis module 212 sets the generated springboard matching path in the access path 703 of the added record.

  In step S1109, the relevance analysis module 212 determines that there is step platform system information that has not been processed, and returns to step S1106. When the third record is set as the target record, no record satisfies the condition. By the above processing, the relevance management information 223 is in a state as shown in FIG.

  FIG. 12 is a flowchart illustrating an example of a process performed when the data management module 211 according to the first embodiment receives a query.

  The data management module 211 receives a query from any system (step S1201).

  The data management module 211 instructs the conversion module 213 to execute the query expansion processing (Step S1202). Details of the query expansion process will be described with reference to FIG.

  When receiving the response indicating the completion of the processing from the conversion module 213, the data management module 211 executes the converted query (step S1203). The method of executing the query is a known technique, and thus the description is omitted.

  The data management module 211 transmits the execution result of the query as output data to any system (step S1204).

  FIG. 13 is a flowchart illustrating an example of a query expansion process performed by the conversion module 213 according to the first embodiment. FIG. 14 is a diagram illustrating an example of output data output when the data management module 211 of the first embodiment executes a query.

  The conversion module 213 analyzes the query and specifies a combination of system information that needs to be referred to (Step S1301). If there is no combination of system information that needs to be referred to, the conversion module 213 outputs a response indicating completion of the processing to the data management module 211, and ends the processing. In this case, the query is not transformed.

  The conversion module 213 selects a target combination from combinations of system information that need to be referred to (step S1302).

  The conversion module 213 refers to the relevance management information 223 and determines whether or not reference between the system information corresponding to the target combination is possible (step S1303).

  Specifically, the conversion module 213 determines whether there is a record in which the combination of the identification information set in the first data source name 701 and the second data source name 702 matches the target combination. When there is a record that satisfies the above-described condition, the conversion module 213 determines that it is possible to refer to the system information corresponding to the target combination.

  If it is determined that it is not possible to refer to the system information corresponding to the target combination, the conversion module 213 outputs a response indicating an error to the data management module 211 (Step S1304). After that, the conversion module 213 ends the processing. At this time, the converted query is discarded. When receiving the response indicating the error, the data management module 211 transmits an error message to the transmission source of the query.

  If it is determined that the system information corresponding to the target combination can be referred to, the conversion module 213 determines whether the specified path is a step-coincidence path (step S1305).

  Specifically, the conversion module 213 refers to the access path 703 of the record searched in step S1303, and determines whether the specified path is a springboard-matching path. For example, the conversion module 213 determines whether the value of the access path 703 includes a semicolon.

  When it is determined that the specified path is not the platform matching path, the conversion module 213 converts the received query into a query in which a foreign key for directly referring to the target combination is set (step S1306). Thereafter, the conversion module 213 proceeds to step S1308. For example, it is converted into a query to which an ON clause or a WHERE clause relating to a foreign key has been added.

  Specifically, the conversion module 213 specifies the identification information of the attribute included in the two pieces of system information based on the specified path, and generates a query using the identification information of the specified attribute as a foreign key.

  When it is determined that the specified path is a springboard-matching path, the conversion module 213 converts the received query into a query in which a foreign key is set to refer to the target combination via the relay system information ( Step S1307). Thereafter, the conversion module 213 proceeds to step S1308. Specifically, the following processing is executed.

  The conversion module 213 specifies the identification information of the attribute included in the target combination and the identification information of the attribute included in the relay system information based on the specified path. The conversion module 213 refers to the attribute management information 221 based on the attribute identification information included in the relay system information, and specifies the identification information of the relay system information.

  The conversion module 213 generates a query that combines a first query that makes a reference between one system information and the relay system information and a two query that makes a reference between another system information and the relay system information.

  The first query includes, as a foreign key, the identification information of the attribute included in each of the one system information and the relay system information, and the second query includes the identification information of the attribute included in each of the other system information and the relay system information. Contains identification information as a foreign key. The above is the description of the processing in step S1307.

  In step S1308, the conversion module 213 determines whether the processing has been completed for all the specified combinations (step S1308).

  If it is determined that the processing has not been completed for all the specified combinations, the conversion module 213 returns to step S1302 and executes the same processing.

  If it is determined that the processing has been completed for all the specified combinations, the conversion module 213 outputs a response including the converted query to the data management module 211 (step S1309), and ends the processing.

  Here, the query expansion processing will be described using a specific example. In the following description, it is assumed that a query including Join (sensor information, setting information) has been received. It is assumed that the relevance management information 223 is in the state shown in FIG.

  In step S1301, the conversion module 213 specifies a combination of the sensor information 300 and the setting information 320. In step S1302, the conversion module 213 selects a combination of the sensor information 300 and the setting information 320 as a target combination.

  In step S1303, the conversion module 213 searches for a record in which the combination of the identification information set in the first data source name 701 and the second data source name 702 matches the target combination. Here, the fourth record is searched. Therefore, the determination result of step S1303 is YES. In addition, the access path 703 of the fourth record is set to the step-coincidence path, so that the determination result in step S1305 is YES.

  In step S1307, the conversion module 213 specifies that the dictionary information 310 is the platform system information by referring to the attribute management information 221 based on the attribute identification information of the platform system information. Therefore, Join (sensor information, setting information) is converted into Join (Join (sensor information, dictionary information), Join (dictionary information, setting information)). At this time, C3 is specified as an external key in Join (sensor information, dictionary information), and C7 is specified as an external key in Join (dictionary information, setting information).

  The data management module 211 transmits the output data as illustrated in FIG. 14 to the system that transmitted the query by executing the query converted as described above.

  As described above, even a query input by the user without knowing the dictionary information 310 is automatically converted to a query that complements the relevance between system information. This facilitates search processing in the Data Lake system.

  Note that the present invention is not limited to the above-described embodiment, and includes various modifications. Further, for example, in the above-described embodiment, the configuration has been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to the configuration including all the described configurations. Further, a part of the configuration of each embodiment can be added, deleted, or replaced with another configuration.

  In addition, each of the above-described configurations, functions, processing units, processing means, and the like may be partially or entirely realized by hardware, for example, by designing an integrated circuit. The present invention can also be realized by software program codes that realize the functions of the embodiments. In this case, a storage medium storing the program code is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the program code itself and the storage medium storing the program code constitute the present invention. As a storage medium for supplying such a program code, for example, a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, a magnetic tape, A non-volatile memory card, ROM, or the like is used.

  Further, the program code for realizing the functions described in the present embodiment can be implemented by a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, and Java.

  Further, by distributing the program code of the software for realizing the functions of the embodiment via a network, the program code is stored in a storage means such as a hard disk or a memory of a computer or a storage medium such as a CD-RW or a CD-R. Alternatively, a processor included in a computer may read and execute a program code stored in the storage unit or the storage medium.

  In the above-described embodiment, the control lines and the information lines are considered to be necessary for the description, and do not necessarily indicate all the control lines and the information lines on the product. All components may be interconnected.

Claims (10)

A data management method for at least one computer constituting a data store,
The at least one computer has a processor, a memory connected to the processor, and a network interface connected to the processor,
In the data store, system information including a plurality of data blocks including at least one attribute is stored in a format independent of a data structure of the system information,
The memory stores identification information of two system information having a relationship, and relationship management information including a record for managing an access path defined by a correspondence relationship between attributes to be associated with the two system information,
The data management method includes:
A first step in which the processor receives a registration request for new system information;
A second step in which the processor stores the new system information in the data store;
A third step in which the processor specifies, from among the system information stored in the data store, first system information that is relevant to the new system information;
A record including identification information of the new system information and the first system information, and a first access path defined by a correspondence relationship of an attribute for associating the new system information with the first system information; A fourth step of registering in the relevance management information;
A fifth step of, when the processor receives the query for the data store, converting the query into a query to which information for performing a reference between the system information is added based on the relevance management information; When,
A sixth step in which the processor executes the converted query. The data management method according to claim 1, wherein
The third step is
A seventh step in which the processor compares the attribute included in the new system information with the attribute included in the first system information to determine whether or not the attribute includes an attribute that can be identified;
An eighth step of, when the processor is determined to include the attribute that can be identified, identifying the first system information as system information having relevance to the new system information;
A ninth step in which the processor generates the first access path based on the attribute that can be identified. The data management method according to claim 2, wherein
The memory holds attribute management information for managing attributes included in a data block of the system information stored in the data store,
The second step is
Wherein the processor specifies an attribute included in a data block of the new system information;
Setting an attribute included in a data block of the new system information in the attribute management information,
The seventh step includes a step in which the processor compares the attribute included in the new system information and the attribute included in the first system information, based on the attribute management information. Management method. The data management method according to claim 2, wherein
The third step is
The processor referring to the association management information to identify second system information having an association with the first system information;
Acquiring a second access path defined by a correspondence relationship of an attribute that associates the first system information and the second system information;
The processor is defined by a correspondence relationship of an attribute that associates the new system information via the first system information with the second system information based on the first access path and the second access path. Generating a third access path,
The processor setting, in the relevance management information, a record including identification information of the new system information and the second system information, and a record including the third access path. Management method. The data management method according to claim 4, wherein
The fifth step is:
The processor identifying combinations of relevant system information;
The processor refers to the association management information, and searches for a record corresponding to the specified combination of system information;
Converting the query based on an access path included in the searched record by the processor. A computer constituting a data store,
A processor, a memory connected to the processor, and a network interface connected to the processor,
In the data store, system information including a plurality of data blocks including at least one attribute is stored in a format independent of a data structure of the system information,
The memory stores identification information of two pieces of system information having relevance, and relevance management information including a record for managing an access path defined by a correspondence between attributes to be associated with the two pieces of system information,
A data management unit that manages the data store;
By analyzing the association between the system information stored in the data store, an association analysis unit that updates the association management information,
A conversion unit that converts the query received by the data management unit into a query to which information for performing a reference between the system information is added, based on the association management information,
The data management unit includes:
If a registration request for new system information is received, the new system information is stored in the data store, and the relevance analysis unit is instructed to update the relevance management information,
When receiving a query for the data store, instruct the conversion unit to convert the query, execute the query converted by the conversion unit,
The relevance analyzer,
From the system information stored in the data store, first system information having a relationship with the new system information is specified,
A record including the identification information of the new system information and the first system information and a first access path defined by a correspondence relationship of an attribute for associating the new system information and the first system information is stored in the association table. A computer characterized by registering in management information. The computer according to claim 6, wherein
The relevance analyzer,
By comparing an attribute included in the new system information with an attribute included in the first system information, it is determined whether or not an attribute that can be identified is included,
If it is determined to include the attribute that can be identified, the first system information is specified as system information having a relationship with the new system information,
A computer that generates the first access path based on the attribute that can be identified. The computer according to claim 7, wherein
The memory holds attribute management information for managing attributes included in a data block of the system information stored in the data store,
The relevance analyzer,
Identify the attributes included in the data block of the new system information,
Setting an attribute included in the data block of the new system information in the attribute management information,
A computer that compares an attribute included in the new system information acquired from the attribute management information with an attribute included in the first system information. The computer according to claim 7, wherein
The relevance analyzer,
With reference to the relevance management information, specifying second system information having relevance to the first system information,
Acquiring a second access path defined by a correspondence relationship of an attribute that associates the first system information and the second system information;
A third system defined by a correspondence relationship of an attribute for associating the new system information with the second system information via the first system information based on the first access path and the second access path; Generate an access path,
A computer, wherein a record including identification information of the new system information and the second system information and the third access path is set in the association management information. The computer according to claim 9,
The conversion unit,
Identify relevant combinations of system information,
With reference to the relationship management information, search for a record corresponding to the specified combination of system information,
A computer that converts the query based on an access path included in the searched record.

Images