JP2019121181A - Data management system and data management method - Google Patents

Abstract

To flexibly re-use a data mart in response to a request from a user and to effectively concentrate analyzed results regarding the data mart.SOLUTION: A data catalog management server 100 of a data management system 10 is arranged so as to have: a storage apparatus 101 for holding data store information 125 and data store application connection information 126; and an arithmetic apparatus 104 for, based on a refining request of analyzed results received from a terminal 001 and the data store information 125, if a target data mart indicated by the reduction request does not include a predetermined meta information, selecting a data mart including the meta information from data marts on a data path including the target data mart, generating a new data path including a filter and the like applied to the data mart, and generating a predetermined analysis result corresponding to the refining request in accordance with the new data path.SELECTED DRAWING: Figure 1B

Description

  The present invention relates to a data management system and a data management method, and more specifically, flexibly reuses a data mart in response to a user's request, and efficiently narrows down analysis results regarding the data mart. Related to technology.

  The IoT platform has been introduced to collect and use predetermined data from sensors and machine tools installed in the factory for analysis of production efficiency in the factory and failure prediction of machine tools. In general, in the IoT platform, there are four operating data lake, data mart, ETL (Extract Transform Load), and analysis application.

  In the IoT platform on which such programs operate, the data saved in the data mart is properly reused to avoid the wasteful development process of the analysis application and ETL described above. When reusing data, the ETL and analysis application repeat the process of reading and processing data from the data mart and separately storing it in the data mart. As a result, the data mart and ETL and analysis applications are logically connected in multiple stages.

  As described above, as a conventional technique related to reuse of data stored in a data mart, for example, data including data required by a developer or user of an analysis application using a predefined ETL template A technique (see Patent Document 1) for automatically generating a mart has been proposed.

US9519695 B2

  However, the following problems remain in the prior art. In the prior art, a defined ETL template is used to automatically generate a data mart according to the requirements (designated by the developer or user of the analysis application) expressed as input parameters. In practice, however, there are many cases where no predefined ETL template suitable for such requirements is prepared.

  Therefore, the developer or user of the analysis application that specified the above-mentioned requirements checks the connection status between the huge and diverse data stored in the data mart and the existing analysis application one by one, and carries out the ETL template itself. It must be defined. This is a heavy burden on the developer or user of the analysis application, and it may be impossible in some cases depending on the skill and knowledge of the person concerned.

  In addition, since complicated analysis processing can not be executed only by ETL, it is also difficult to narrow down the analysis results by predetermined meta information (eg, data collection conditions and analysis conditions). Specifically, if the meta information value specified by the analysis application developer as the condition for narrowing is not fixed but can change for each narrowing request, data in the data lake or data mart can not be reused. to cause.

  For example, consider a case where failure rate data of a machine tool producing a specific type of part is taken as an input from a data lake and reused as an analysis target. An analysis application that analyzes the above-mentioned failure rate data generally adds information of the machine tool as meta information to the analysis result, and generally stores the information in a predetermined data mart. However, it is not common to add meta information on parts produced by the machine tool. Then, even if the data mart is reused, the data mart produces another type of component if the other type of component is specified as a condition for narrowing down the analysis. Since the data on the machine tool is held, it can not be used as an analysis target, and the analysis itself can not be executed.

  Therefore, an object of the present invention is to provide a technology for flexibly reusing a data mart in response to a user's request and efficiently narrowing down the analysis result on the data mart.

  A data management system according to the present invention for solving the above problems comprises: data store information storing data rake storing predetermined data; and data store information respectively storing data analysis results storing predetermined analysis results for the predetermined data; Storage device for storing data path / application connection information storing data path information indicating a series of relationships of the data mart and an algorithm responsible for analysis processing for the data rake or the data mart Based on the analysis narrowing request received from the analysis result and the data store information, it is determined whether the target data mart indicated by the narrowing down request includes predetermined meta information, and if the meta information is not included, the data store Information and said data store application In the data path including the target data mart, among the data mart from the target data mart to the data lake, one including the meta information is selected based on the serial information, and the data mart is applied to the data mart A new data path including a filter related to the meta information, which is a type of algorithm, a new data mart obtained by applying the filter, and a predetermined algorithm to be applied to the new data mart is generated according to a predetermined rule, and the new data is An arithmetic device generating a predetermined analysis result corresponding to the narrowing request by executing application of the filter to the selected data mart and application of a predetermined algorithm to the new data mart according to a path It is characterized by

Further, according to the data management method of the present invention, there are provided data store information storing information on each of a data lake storing predetermined data and a data mart storing a predetermined analysis result on the predetermined data, the data lake, and the data mart An information processing system comprising: a storage device for storing data path / application connection information storing information on a data path indicating a series of relationships of an analysis process for the data lake or the data mart or the data mart; Based on the narrowing request of the analysis result received from the predetermined terminal and the data store information, it is determined whether the target data mart indicated by the narrowing request includes predetermined meta information, and when the meta information is not included, The data store information and the data store application Data path including the target data mart, among the data mart from the target data mart to the data lake, which includes the meta information, is selected based on the connection information and is applied to the data mart A new data path including a filter related to the meta information, which is a type of the algorithm, a new data mart obtained by applying the filter, and a predetermined algorithm applied to the new data mart is generated according to a predetermined rule, and the new data path is generated. A predetermined analysis result corresponding to the narrowing request is generated by executing application of the filter to the selected data mart and application of a predetermined algorithm to the new data mart according to a data path. Do.

  According to the present invention, it is possible to flexibly reuse the data mart in response to the user's request and efficiently narrow down the analysis result on the data mart.

It is a figure showing an example of network composition including a data management system of this embodiment. It is a figure showing the example of hardware constitutions of the data management system of this embodiment. It is a figure showing an example of composition of data store information of this embodiment. It is a figure which shows the structural example of data store * application connection information of this embodiment. It is a figure which shows the structural example of the application information of this embodiment. It is a figure which shows the structural example of the data lake in this embodiment. It is a figure which shows the structural example of the 1st data mart in this embodiment. It is a figure which shows the structural example of the 2nd data mart in this embodiment. It is a figure which shows an example of the logical configuration of the data store of this embodiment, and an analysis application. It is a figure which shows an example of the data store information after the new data path creation of this embodiment. It is a figure which shows an example of the data store * application connection information after the new data path creation of this embodiment. It is a figure which shows an example of the application information after the novel data path creation of this embodiment. It is a figure which shows an example of the content of 1st 'data mart after novel data path creation of this embodiment. It is a figure which shows an example of the content of 2nd 'data mart after novel data path creation of this embodiment. It is a figure which shows an example of the content of the new data path of this embodiment. It is a flowchart which shows the process example in the analysis application developer terminal of this embodiment. It is a figure which shows the example 1 of a flow of the data management method in this embodiment. It is a figure which shows the example 2 of a flow of the data management method in this embodiment. It is a figure which shows the example 3 of a flow of the data management method in this embodiment. It is a figure which shows the example 4 of a flow of the data management method in this embodiment. It is a figure which shows the example 5 of a flow of the data management method in this embodiment. It is a figure which shows the example 6 of a flow of the data management method in this embodiment.

--- System configuration ---
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1A is a network configuration diagram including a data management system 10 of the present embodiment. The data management system 10 illustrated in FIG. 1A is a computer system that flexibly reuses a data mart in response to a user's request and efficiently narrows down the analysis result on the data mart.

  Further, as an example, such data management system 10 can be assumed to be configured by a data catalog management server 100 and an IoT-based execution server 200. However, as a minimum configuration of the data management system 10 according to the present embodiment, a mode may be assumed in which functions required for a single device such as only the data catalog management server 100 are implemented.

As an organization that operates the data management system 10 of the present embodiment, for example, a business entity that analyzes production efficiency in a certain factory or predicts failure of a machine tool can be assumed. At a factory managed by this business entity, a mechanism is established and operated in which factory equipment 002 such as sensors and machine tools installed in the factory sends monitoring data at regular intervals and the IoT platform execution server 200 collects it. It shall be.

  Therefore, as described above, the IoT-based execution server 200 appropriately processes and analyzes the data lake 225 for accumulating collected data obtained from the factory equipment 002, the data lake 225, and other data mart 226 according to the purpose. ETL (Extract Transform Load) that performs data extraction and conversion on the acquired data mart 226, data lake 225 and data mart 226, and analysis that performs analysis processing on the data mart 226 prepared by this ETL The four applications will work.

  Among them, the data lake 225 is a database for storing data collected from the factory equipment 002 such as the above-described sensor or machine tool without processing such as processing. The data mart 226 is a database for storing data processed by the ETL and the analysis application. The data lake 225 and the data mart 226 are generically called a data store.

  In addition, ETL aims to prepare the data required by the above-mentioned analysis application, and it is software that extracts the data from Data Lake 225 or Data Mart or 226 and stores the converted result into Data Mart 226 as appropriate. It is.

  The analysis application is software that reads data from the data mart 226 according to the purpose and stores the analysis result in the data mart 226. Such an analysis application may also include a function of displaying data or analysis results read out from the data mart 226 to the user via a predetermined interface or an external terminal.

  The IoT-based execution server 200 in which such a program group operates reuses data stored in the data mart 226 by the analysis application or the ETL in order to avoid redevelopment of the above-described analysis application or ETL. In order to reuse this data, the ETL or analysis application processes the data read from the data mart 226 and repeats the process of storing the data in the separate data mart 226, so that the data mart 226 and ETL or analysis application are logical. It will be connected in multiple stages.

  Such a data store (data lake 225, data mart 226), analysis application, and a structure in which ETLs are connected are called data paths. Also, in this data path, the direction approaching the data lake 225 is referred to as the upstream, and the direction away from the data lake 225 is defined as the downstream.

  Further, as shown in FIG. 1A, the data management system 10 according to the present embodiment is communicably connected to the above-mentioned factory equipment 002 and the analysis application developer / user terminal 001 via an appropriate network 003. It shall be.

  Among them, the data catalog management server 100 includes functional units of an information search unit 110, a data path creation unit 111, a filter creation unit 112, a data store information creation unit 113, and a data store / application connection information creation unit 114. Have. The data catalog management server 100 holds data store information 125, data store / application connection information 126, and application information 127 in an appropriate storage device.

  On the other hand, the IoT-based execution server 200 has each function unit of the application execution unit 210 and the data collection unit 212. Further, the IoT-based execution server 200 stores the data stored in the above-mentioned data store (data lake 225, data mart 226) in the data storage unit 212 implemented in an appropriate storage device. Such data store entities may be databases or key-value stores or tables of databases.

  The analysis application developer / user terminal 001 is a terminal operated by the developer or user of the analysis application in the above-described IoT-based execution server 200. Such a developer or user operates the analysis application developer / user terminal 001 to obtain a data store ID which is an ID uniquely indicating the above-mentioned data store, a meta information name (eg, car type), and the meta A narrowing request consisting of information values (example: A) is sent to the data catalog management server 100. This narrowing-down request is an instruction to narrow the data of the data mart 226 on the condition of desired meta information (eg, “vehicle type” = “A”).

  On the other hand, in response to the above-described narrowing request, the data catalog management server 100 describes the narrowing request among the data stores located upstream in the data path from the data store corresponding to the data store ID included in the narrowing request. Identify a data store that contains data for the meta information name of

  In addition, the data catalog management server 100, in the data path including the data store identified as described above, an analysis application, ETL, etc., located between the data store and the data store (target data mart) described in the narrowing request. Create a new data path 506 consisting of each ID of the data store.

  Further, the data catalog management server 100 uses the meta information name and the meta information value described in the above-described narrowing request to specify the data store identified as described above (a data mart 226 that holds the analysis result by the analysis application, A filter is created as an ETL that narrows down the stored data of the data lake 225) holding collected data obtained from the factory equipment 002. The data catalog management server 100 adds the filter created here to the uppermost stream of the new data path 506 described above (that is, the boundary between the existing data store and the new data path), and then describes the filter in the new data path 506. The IoT-based execution server 200 is instructed as a new data path creation request as to new placement and execution of the filter, data mart, ETL, and analysis application.

  The IoT platform execution server 200 collects monitoring data etc. from machines and sensors included in the factory equipment 002 and stores it in the data lake 225 of the data recording unit 211, and results of executing the ETL and analysis application as a data mart Save to 226.

Further, when the IoT-based execution server 200 receives a creation request of a new data path from the data catalog management server 100 described above, the filter, data mart, ETL, and according to the new data path 506 included in the creation request. Deploy and run an analysis application. In addition, the IoT-based execution server 200 transmits, to the data catalog management server 100, information on the analysis application being executed, that is, an update instruction of the application information 127. In addition, the IoT-based execution server 200 updates each piece of information of the ETL and data store which are appropriately managed therein.
The internal operations of the data catalog management server 100 and the IoT-based execution server 200 having the above-described functions and the like will be described below.

Among them, when the information search unit 110 of the data catalog management server 100 receives a narrowing request from the analysis application developer / user terminal 001 (B1), the information search unit 110 is recorded in the data store information 125 and the data store / application connection information 126. The respective information is acquired (B2, B3, B4, B5), and the above-mentioned new data path 506 is created. In addition, the data catalog management server 100 transmits the new data path 506 created as described above and the above-mentioned narrowing-down request received from the analysis application developer / user terminal 001 to the data path creation unit 111 (B6 ).

  At this time, the data path creation unit 111 of the data catalog management server 100 receives the new data path 506 and the narrowing request from the information search unit 110, and transmits the new data path 506 and the narrowing request to the data store information generation unit 113. To do (B7). Further, the data path creation unit 111 transmits the new data path 506 and the narrowing-down request to the filter creation unit 112 (B9).

  In addition, the data store information creation unit 113 of the data catalog management server 100 receives the meta information name included in the data store indicated by the data store ID 1201 and the data store ID from the data recording unit 211 of the IoT-based execution server 200 at any time. The data 1202, the data name 1203 and the connection destination information 1204 are acquired (A 1), and these are recorded in the data store information 125 (A 2).

  Further, when the data store information creating unit 113 receives the new data path 506 and the narrowing request from the data path creating unit 111 (B7), the meta information name included in the narrowing request, the value of the meta information name, and the new data The narrowed meta information 1205 and the copy destination data store ID 1206 corresponding to the data store ID are calculated from the post-replication object ID 2033 and the copy source object ID 2032 included in the path 506, and the calculated narrowed meta information 1205 and the copy destination data store The ID 1206 is recorded in the data store information 125 (B8).

  Further, the filter creation unit 112 of the data catalog management server 100 creates a filter for narrowing down according to the above-mentioned new data path 506 and narrowing-down request, and records the ID of the filter and the execution method in the application information 127 (B9) . At the same time, the filter creation unit 112 adds the created filter ID to the new data path 506, and transmits the filter and the new data path 506 to the data path creation unit 111 (B10).

  On the other hand, the data path creation unit 111 receives the filter and the new data path 506 from the above-mentioned filter creation unit 112, and transmits the filter and the new data path 506 to the application execution unit 210 in the IoT-based execution server 200. Then, the creation of the above-described filter, analysis application, ETL, and data store is instructed (B12). In addition, the data path creation unit 111 transmits a data store ID, which is the top element of the new data path 506, to the information search unit 110 (B15).

  On the other hand, when the application execution unit 210 of the IoT-based execution server 200 receives an element of the new data path 506 from the data path generation unit 111 (B12), the method of executing the analysis application when this element points to ETL or analysis application Are acquired from the application information 127 (B13) and executed.

  If the element of the new data path 506 received from the data path creation unit 111 points to a data store, the application execution unit 210 creates a new data store in the data storage unit 211 (B14).

Note that the application execution unit 210 executes an analysis application or ETL regardless of the presence or absence of an input from the data path generation unit 111, or creates a new data store, one of the analysis application, ETL, or data store. The data store / application connection information 126 is called with the analysis application located upstream and one downstream, or the ID of the ETL or data store as an argument (D1).

  In addition, when the information search unit 110 of the data catalog management server 100 receives the data store ID from the data path creation unit 111, the information search unit 110 acquires connection information of the data store corresponding to the data store ID from the data store information 125 Data store connection information is returned to the analysis application developer / user terminal (B16).

  In addition, the data store / application connection information creation unit 114 of the data catalog management server 100 is an analysis application, an ETL, or a data store ID, and one upstream thereof from the application execution unit 210 of the IoT-based execution server 200. When the downstream analysis application or ETL or data store ID is accepted (D1), it is recorded in the data store / application connection information 126 (D2).

--- Hardware configuration ---
The hardware configuration of the data catalog management server 100 mainly constituting the data management system 10 of this embodiment is as shown in FIG. 1A.

  The data catalog management server 100 according to the present embodiment includes a storage device 101 configured of an appropriate non-volatile storage element such as a solid state drive (SSD) or a hard disk drive, a memory 103 configured of a volatile storage element such as a RAM, The program 102 held in the unit 101 is read out to the memory 103 or the like to execute the overall control of the apparatus itself and to execute various determinations, calculations and control processing as well as to an arithmetic unit 104 such as a CPU and a network 003 to connect with other apparatuses. A communication device 105 responsible for communication processing is provided.

  Note that, in addition to the program 102 for implementing the functions necessary as the data catalog management server 100 constituting the data management system 10 of the present embodiment, the data store information 125, data store and application connection in the storage device 101. Information 126 and application information 127 are at least stored. Details of these pieces of information will be described later.

  In addition, when the above-mentioned arithmetic unit 104 executes the program 102, the information search unit 110, the data path creation unit 111, the filter creation unit 112, the data store information creation unit 113, and the data store / application connection information creation unit 114. Is implemented. The details of the functions of these functional units will also be described later.

--- Data structure example ---
Next, data used by the above-described data catalog management server 100 and the IoT-based execution server 200, which constitute the data management system 10 of this embodiment, will be described.

FIG. 2A is a view showing a configuration example of the data store information 125 in the present embodiment. This data store information 125 is a set of records including values of data store ID 1201, meta information name 1202, data name 1203, connection destination information 1204, narrowed meta information 1205, and copy destination data store ID 1206. ing.
Among these, the data store ID 1201 is an ID used to uniquely indicate the instance of the corresponding data store, and is a numerical value or a character string.
Also, the meta information name 1202 is a list of meta information names included in the data store indicated by the data store ID 1201.
Also, the data name 1203 is a list of data names included in the data store indicated by the data store ID 1201.
Also, connection destination information 1204 is information of a connection destination that is required to access the data store indicated by the data store ID 1201 and acquire data.
In addition, in the case of a data store narrowed down by the value of a specific meta information, the narrowed meta information 1205 is information indicating the meta information used for the narrowing.

  The copy destination data store ID 1206 is information indicating the copied data store when the data store indicated by the data store ID 1201 is narrowed down by a specific meta information value and another data store is copied.

  The values of the above-described narrowed meta information 1205 and the copy destination data store ID 1206 are recorded by the data catalog management server 100 when the data store is copied.

  Next, FIG. 2B is a view showing a configuration example of the data store / application connection information 126 in the present embodiment. The data store / application connection information 126 is a collection of records including the values of the upstream object ID 1211 and the downstream object ID 1212.

  Among them, the upstream object ID 1211 is an ID of an object located one upstream side in the data path with respect to the object (data lake, data mart, analysis application, ETL, filter) indicated by the downstream object ID 1212.

  On the other hand, the downstream object ID 1212 is an ID of an object positioned one downstream in the data path with respect to the object indicated by the upstream object ID 1211 described above.

Further, FIG. 2C is a view showing a configuration example of the application information 127 in the present embodiment. The application information 127 is a collection of records including each value of the application ID 1221 and the instance creation method 1222.
Among these, the application ID 1221 is an ID uniquely indicating an instance of an analysis application or ETL, and is a numerical value or a character string.

  On the other hand, the instance creation method 1222 is information in which a list of commands and arguments required to execute the analysis application or ETL indicated by the application ID 1221 described above is described in the application execution unit 210 of the IoT-based execution server 200.

Moreover, FIG. 3A is a figure which shows the data structural example of the data lake 225 in this embodiment. The data lake 225 is a collection of records including the values of time 1231, device ID 1232, product type 1233, and rotation duration 1234.
Among these, time 1231 is meta information representing a collection target time of the value of the rotation continuation time 1234.
The device ID 1232 is meta information uniquely indicating a device used for producing a product in the factory equipment 002, and is a numerical value or a character string.
The product type 1233 is meta information indicating the type of product that the device indicated by the device ID 1232 described above has produced at time 1231.

Further, the rotation continuation time 1234 is a value of the time required from the power supply of the device indicated by the device ID 1232 described above being shut off at time 1231 to the stop of the predetermined rotation axis in the device.

Moreover, FIG. 3B is a figure which shows the structural example of the 1st data mart 2261 which is one of the data mart 226 in this embodiment. The first data mart 2261 is a set of records including the values of time 1241, device ID 1242, product type 1243, and failure determination 1244.
Among these, time 1241 is meta information indicating the analysis target time of the failure determination 1244.
The device ID 1242 is meta information uniquely indicating a device used for producing a product in the factory facility 002, and is a numerical value or a character string.
The product type 1243 is meta information indicating the type of product that the device indicated by the device ID 1242 has produced at time 1241.

  In addition, the failure determination 1244 is an analysis result on the failure state of the device obtained by analysis performed by acquiring from the data lake 225 the rotation continuation time 1234 of the time 1231 corresponding to the above-mentioned time 1241.

Further, FIG. 3C is a view showing a configuration example of a second data mart 2262 which is one of the data mart 226 in the present embodiment. The second data mart 2262 is a collection of records including the time 1251, the failure rate 1252, and each value.
Among these, time 1251 is meta information representing analysis target time of failure rate 1252.
The failure rate 1252 is information representing a statistical value of failure determination calculated from the failure determination 1244 stored in the first data mart 2261 described above.

---Specific example of data path---
FIG. 4 shows an example of the logical structure of the existing data path 505 already executed by the IoT-based execution server 200 in this embodiment and the new data path 506 added by executing the data management method of this embodiment. FIG.

  In this case, it is assumed that the data collection unit 212 of the IoT-based execution server 200 in the present embodiment collects monitoring data of equipment and the like from the factory equipment 002 and stores the monitoring data in the data lake 225 as collection data.

  Here, the application execution unit 210 of the IoT-based execution server 200 acquires the rotation collection time 1234 from the collected data of the data lake 225 described above, and uses this as an input to the first analysis application 401 (the IoT-based execution server 200 obtains in advance Hold).

  The first analysis application 401 stores the value of the failure determination 1244 in the first data mart 2261 as a result of analyzing whether or not a predetermined device in the plant facility 002 is broken from the value of the rotation collection time 1234.

  Subsequently, the application execution unit 210 described above acquires the value of the failure determination 1244 from the first data mart 2261 and uses this as an input to execute a second analysis application 402 (previously held by the IoT-based execution server 200).

  The second analysis application 402 tabulates the value of the above-mentioned failure determination 1244 as a failure rate in the corresponding device by a predetermined statistical method, and stores the result, that is, the analysis result in the second data mart 2262.

When the IoT execution base server 200 is in the above state, the data catalog management server 1
It is assumed that the analysis application developer / user terminal 001 sends a request for narrowing down the analysis results to 00. It is assumed that this narrowing-down request is a request to narrow the target data mart to the above-mentioned second data mart 2262 and the analysis result, which is the stored information, for the meta information “product type” for “A”.

  In that case, the data catalog management server 100 searches the data store / application connection information 126 for a data mart including the above meta information “product type” has a value of “A”, and the most downstream data store viewed from the data lake 225 It is assumed that the first data mart 2261 is identified.

  Therefore, the data catalog management server 100 transmits a request for creating a new data path 506 branched from the first data mart 2261 to the IoT-based execution server.

  On the other hand, the application execution unit 210 of the IoT-based execution server 200 creates the filter 410 based on the creation request of the new data path 506 received from the data catalog management server 100 and applies this to the first data mart 2261 to Generate the 1 'data mart 2263 (FIG. 6A).

  In addition, the application execution unit 210 executes the command described in the instance creation method 1222 acquired from the application information 127 regarding the second analysis application 402, and inputs the second 'analysis application 403 and, to this, the first 1' data mart. To generate the second 2 'data mart 2264 (FIG. 6B) given as

  FIGS. 6A and 6B are explanatory diagrams showing an example of contents stored in the 1 'data mart 2263 and the 2' data mart 2264 disposed in the new data path 506. FIG. The first 1 'data mart 2263 in FIG. 6A is a data mart obtained by narrowing down the first data mart 2261 with the value "A" of the meta information name "product type", so the "product type" as meta information is "A". Only the data of are stored.

  Further, although the 2 'data mart 2264 of FIG. 6B does not have a column of "product type" which is meta information, the input information of the 2' 'analysis application 403 for generating the 2' data mart 2264 is the above-mentioned first Since the 'data mart, ie,' product type 'is only data relating to' A ', the value of failure rate 2022 that is the analysis result is also only for' product type 'as' A'. Therefore, the processing of the existing analysis application can be used as it is, and processing results narrowed down by the value of specific meta information can be obtained.

  The application execution unit 210 logically connects the filter 410, the first ′ data mart 2263, the second ′ analysis application 403, and the second ′ data mart 2264 thus obtained, and forms a new data path 506. The application execution unit 210 stores corresponding new records in the data store information 125, the data store / application connection information 126, and the application information 127 in accordance with the new data path 506 described above.

  5A to 5C show configuration examples of the data store information 125, the data store / application connection information 126, and the application information 127 in the case where a record is added with respect to the new data path 506 described above.

In the example of the data store information 125 in FIG. 5A, each record of the copy destination data store 1 ′ and the data store 2 ′ with the data store 1 and the data store 2 as copy sources is added. Therefore, in the corresponding record, copy destination data store ID 3
The values of 016 are "1 '" and "2'", respectively. It should be noted that the information of the corresponding data mart, that is, the duplicated 1 'data mart' and 2 'data mart will be added to these records.

  The first data mart and the second 2 'data mart are the meta data name "product type" value "A", which narrows down the first data mart and the second data mart respectively, so the narrowed meta information The value 3016 is “{meta information name: product type, value of meta information: A}”. In FIG. 5B, as described in FIG. 4, since the first data mart 2261 is replicated to create a new data path, the lower three lines indicating the newly created data path are added. In FIG. 5C, since the 2 'application is duplicated and the filter 410 is placed, the final line showing the newly created 2' application and the filter 410 is added.

  The application execution unit 210 that has obtained the new data path 506 in this way executes the filter 410 with the data and meta information stored in the first data mart 2261 as arguments.

  The filter 410 described above narrows down to only the data values when the value of the product type 1243 which is meta information is “A” among the values of the data and meta information received from the first data mart 2261. Is a filter to be stored in the first 1 'data mart 2263.

  Further, the second 'analysis application 403 acquires the value of the failure judgment 1264 from the first' data mart 2263, and calculates the failure rate 1272 obtained by totaling the failure judgments 1264 in the same process as the second analysis application 402 as the second 'data. Save to Mart 2264

  When the connection destination information of the 2 'data mart 2264 is added to the data store information, the data catalog server 100 transmits the connection destination information of the 2' data mart 2 264 to the analysis application developer / user terminal 001. As a result, the developer or user of the analysis application can obtain data narrowed down by the value of the meta information by connecting to the second 'data mart 2264.

  A specific example of the new data path 506 generated as described above is shown in FIG. The new data path 506 is stored on the memory of the data catalog management server 100 or the IoT-based execution server 200 each time the user makes a data mart acquisition request to the data catalog management server 100 from the analysis application developer / user terminal 001. It is generated.

  The elements of the new data path 506 are generated for the copy source object ID 2033 without duplication. The list element number 2031 is information uniquely indicating an element of the data path. The copy source object ID 2032 is an ID indicating a copy source object. The after-duplication object ID 2033 is an ID indicating a after-duplication object. The upstream object ID 2034 is an ID indicating an object one upstream of the copied object. The downstream object ID 2035 is an ID indicating an object one downstream of the copied object.

--- Example of flow ---
Hereinafter, the actual procedure of the data management method in the present embodiment will be described based on the drawings. Various operations corresponding to the data management method described below are realized by a program that the data catalog management server 100 and the IoT-based execution server 200 that configure the data management system 10 read out to a memory or the like and execute. The program is composed of code for performing various operations described below.

  FIG. 8 is a diagram showing a flow example 1 of the data management method in the present embodiment, and more specifically, is a flowchart showing the operation of the analysis application developer / user terminal 001 in the present embodiment.

  Suppose that the developer or user of the analysis application needs analysis results narrowed down to specific values of specific meta information. In this case, the developer or user of the analysis application uses the ID (for example, 1) of the data store (target data mart) assumed to include the analysis result and the meta information name (for example, product) used for narrowing down. A narrowing request consisting of the type) and the value of meta information to be narrowed (example: A) is input to the analysis application developer / user terminal 001.

  On the other hand, the analyst application developer / user terminal 001 receives the narrowing-down request described above (S101). In addition, the analysis application developer / user terminal 001 calls the information search unit 110 of the data catalog management server 100 with the narrowing-down request accepted in S101 as an argument (S102), and ends the process.

  FIG. 9A and FIG. 9B are diagrams showing a second flow example of the data management method in the present embodiment, and more specifically, a flowchart showing the operation of the information search unit 110 in the data catalog management server 100. Among these, FIG. 9A shows an operation started when the information search unit 110 receives a narrowing request from the analysis application developer / user terminal 001.

  In this case, the information search unit 110 receives a narrowing-down request from the analysis application developer / user terminal 001 (S201), and substitutes the ID (example: 1) of the data store into a variable i (S202).

  Next, the information search unit 110 specifies a record related to the ID of the data store obtained in S202 in the data store information 125, and the meta information name 3012, the narrowed meta information 3015, and the like in each record. The respective values of the copy destination data store ID 3016 are acquired and stored in the memory (S203).

  Thereafter, the information search unit 110 generates a record in which the value of the variable i described above is, for example, the copied object ID 2033, and adds the record as a configuration record of a new new data path 506 (S204). Here, (a part of) a new data path 506 is generated.

  Further, the information search unit 110 checks whether the value of the variable i is a data store ID or whether there is a record including the corresponding value as the data store ID in the data store information 125 (S205).

  If the value of the variable i is not a data store ID as a result of the above check (S205: NO), the information search unit 110 stores an object ID one upstream from the object ID indicated by the variable i in the data store / application The object ID is acquired from the connection information (S209), this object ID is substituted for the variable i (S210), and the process returns to S203.

  On the other hand, when the value of the variable i is the data store ID as a result of the above check (S205: YES), the information search unit 110 determines whether the value of the copy destination data store ID acquired in S203 is empty ( S206).

  If the value of the copy destination data store ID is empty as a result of the above determination (S206: YES), the information search unit 110 subsequently determines whether the value of the narrowed meta information acquired in S203 is empty (S206). S207).

  As a result of the above determination, when the value of the copy destination data store ID is empty (S207: YES), the information search unit 110 determines that the meta information name (e.g., product type) included in the above narrowing request is S203. It is determined whether it is included in the meta information name acquired in step S208.

  Also, as a result of the above determination, if the meta information name included in the narrowing-down request is not included in the meta information name acquired in S203 (S208: NO), the information search unit 110 returns the process to S209. The processes after S209 will be repeated.

  On the other hand, as a result of the above determination, when the meta information name included in the narrowing request is included in the meta information name acquired in S203 (S208: YES), the information search unit 110 determines that the narrowing request is The data path creation unit 111 is called with the new data path 506 generated in S204 as an argument (S220), and the process is ended.

  If the value of the copy destination data store ID is not empty as a result of the determination in S206 described above (S206: NO), the information search unit 110 uses the value of the copy destination data store ID acquired in S203 to perform data. The store information 125 is searched, each value of the meta information name 3012 and the narrowed meta information 3015 is acquired from the record including the value of the data store ID as the data store ID 3011, and is overwritten on the memory (S211).

  Subsequently, the information search unit 110 checks whether the value of the narrowed meta information 3015 acquired in S211 is empty (S212). If it is empty (S212: YES), the process proceeds to S207 and the process is continued Do.

  On the other hand, as a result of S212, when the value of the narrowed meta information 3015 is not empty (S212: NO), the information search unit 110 uses the value of the above copy destination data store ID to generate the new data path generated in S204. The corresponding element in the record of 506, that is, the value of the copy source object ID 2032 is replaced (S213).

  In addition, the information search unit 110 determines whether the value of the narrowed meta information 3015 acquired in S211 matches the meta information name included in the narrowing request and the value of the meta information (S214).

  As a result of the above-described determination, when the value of the narrowed meta information 3015 acquired in S211 does not match the meta information name included in the narrowing request and the value of the meta information (S214: NO), the information search unit 110 A notable error is returned to the analysis application developer / user terminal 001 (S215), and the process is ended.

  On the other hand, as a result of the above determination, if the value of the narrowed meta information 3015 acquired in S211 matches the meta information name included in the narrowing request and the value of the meta information (S214: YES), the information search unit 110 It is determined whether the number of elements of the current new data path 506 is one (S216).

As a result of the above determination, if the number of elements of the new data path 506 is 1, that is, there is one record (S216: YES), the information searching unit 110 determines that the after-duplication object ID 2033 in the record of the new data path 506 is The connection destination information of the indicated data store is acquired in the data store information 125 (S217).
In addition, the information search unit 110 responds to the analysis application developer / user terminal with the connection destination information acquired in the above-described S217 (S218), and ends the processing.

  On the other hand, as a result of the above determination, if the number of elements of the new data path 506 is not 1 (S216: NO), the information search unit 110 deletes the meta information name and meta information value included in the above narrowing request. Then, the data path creation unit is called with the narrowing request and the new data path 506 as arguments (S220), and the process is ended.

  FIG. 9B is a flowchart showing an operation started when the above-mentioned information search unit 110 receives a response request of connection destination information from the analysis application developer / user terminal 001.

  In this case, the information search unit 110 receives a response request for connection destination information from the data path creation unit 111 together with the ID of the data store, for example, "a '" (S301). The connection destination information of a ′ ′ ′ is acquired (S302).

  Further, the information search unit 110 returns the connection destination information of the data store “a ′” acquired in S302 to the analysis application developer / user terminal 001 (S303), and ends the processing.

  FIG. 10 is a flowchart showing the operation of the data path creation unit 111 of this embodiment. In this case, the data path creation unit 111 receives the narrowing request and the new data path from the information search unit 110 described above (S 401), and calls the data store information creation unit 113 with the new data path and the narrowing request as arguments ( S402).

  In addition, the data path creation unit 111 calls the filter creation unit 112 with the narrowing request described above and the last data store ID of the new data path 506 as arguments (S403), and ends the processing.

  On the other hand, the filter creation unit 112 executes the operation shown in the flowchart of FIG. In this case, the filter creation unit 112 receives from the data path creation unit 111 the narrowing request and the data store ID that are the above arguments (S501), and checks whether the meta information name and the meta information of the narrowing request are empty ( S502).

  If the meta information name and the meta information value of the narrowing-down request are not empty in the above-described S502 (S502: NO), the filter creation unit 112 recognizes data of a data store that corresponds to the data store ID obtained in S501. Create a filter that narrows down only to the value when the value of the meta information name indicated by the above narrowing request (example: "product type") becomes the value of the meta information indicated by the narrowing request (example: "A") (S503). For the creation of this filter, for example, with respect to the data extraction algorithm for the data store prepared in the IoT-based execution server 200, the value of the meta information name indicated by the above-mentioned narrowing request (example: "product type") It is possible to set an extraction condition that is a value of meta information (for example, “A”) indicated by the narrowing-down request, a process, etc. (in this case, a filter to be created has a data extraction algorithm with an extraction condition set and Become).

  Next, the filter creation unit 112 creates a filter ID uniquely indicating the filter created in S503 according to a predetermined rule (S504), and creates an instance creation method of the filter ID and the above-mentioned filter (example: the above-mentioned) As described above, the description for setting extraction conditions in the data extraction algorithm is recorded in the application information 127 (S505).

Finally, the filter creation unit 112 calls the data path creation unit 111 with the above-described filter and filter ID as arguments (S506), and ends the process. Note that, as a result of the above-described S502, if the meta information name and the meta information value of the narrowing-down request are empty (S502: YES),
The filter creation unit 112 creates an empty filter that does not narrow down the data (S507), and proceeds to S504 to continue the processing.

  On the other hand, the data path creation unit 111 called by the above-described filter creation unit 112 executes the operation shown in the flowchart of FIG. The data path creation unit 111 in this case receives the filter and the filter ID from the above-described filter creation unit 112 (S 451), and checks whether the filter is empty (S 452).

  As a result of the above-described S452, when the filter and the filter ID are not empty (S452: NO), the data path generation unit 111 generates a record in which the received filter ID is set in the copied object ID, and generates a new data It adds to the end of the path 506 (S 453).

  Next, the data path creation unit 111 calls the application execution unit 210 with the last element of the new data path 506 as an argument (S455), and deletes the last element of the new data path 506 from the new data path 506 (S455) ).

  If the number of elements of the new data path 506 is not 1 at S455 described above (S456: NO), the data path creation unit 111 returns the process to S454 and repeats the subsequent steps (S456).

On the other hand, when the number of elements of the new data path 506 becomes 1 by the above-described S456 (S456: YES), the data path creation unit 111 uses the last element of the new data path 506 as an argument. The application execution unit 210 is called (S457).
Further, the data path generation unit 111 calls the information search unit 110 with the top element of the new data path 506 as an argument (S458), and ends the process.
On the other hand, as a result of the above-described S452, when the filter is empty (S452: YES), the data path creation unit 111 shifts the process to S455 and continues the process.
On the other hand, the application execution unit 210 called from the data path creation unit 111 in S457 described above executes the operation shown in the flowchart of FIG.

The application execution unit 210 in this case receives, from the data path creation unit 111 of the data catalog management server 100 described above, the application ID of the analysis application indicated by the end record of the new data path 506, for example, an execution request for application k (S601) The method for creating an instance of the application k is acquired from the application information 127 (S602).
Next, the application execution unit 210 executes the above-described method for creating an instance of the application k obtained in the above-described S602 (S603), and ends the processing.

  As mentioned above, although the best mode etc. for carrying out the present invention were concretely explained, the present invention is not limited to this, and can be variously changed in the range which does not deviate from the gist.

  According to the present embodiment, it is possible to flexibly reuse the data mart in response to the user's request, and to narrow down the analysis result on the data mart efficiently.

At least the following matters will be made clear by the description of the present specification. That is, in the data management system of the present embodiment, when the arithmetic device selects the data mart including the meta information among the data mart from the target data mart to the data lake, the data mart including the meta information The data mart closest to the target data mart may be selected from the above.

  According to this, generation of a new data mart included in the new data path and used when obtaining an analysis result corresponding to the narrowing request becomes efficient without waste. As a result, it is possible to flexibly reuse the data mart in response to the user's request, and to narrow down the analysis result on the data mart more efficiently.

  Further, in the data management system according to the present embodiment, the arithmetic device narrows down the predetermined attribute of the meta information designated by the narrowing request for the new data mart obtained by the application of the filter or the application of the predetermined algorithm. When the new data mart is to be applied with the target data mart or filter or algorithm due to a new narrowing request after the recording, the data is recorded as the finished meta information in the data store information. If the meta information indicated by the narrowing request is different from the narrowed meta information recorded regarding the new data mart, the process corresponding to the new narrowing request may be avoided.

  According to this, for example, a new data mart obtained by applying a filter or applying an algorithm (analysis application) for a specific event, that is, a narrowed data mart, has a value for an event different from the above-mentioned event Based on the fact that it does not include, it becomes possible to avoid processing in response to the narrowing request for such data mart. As a result, it is possible to flexibly reuse the data mart in response to the user's request, and to narrow down the analysis result on the data mart more efficiently.

  Further, in the data management system according to the present embodiment, the arithmetic unit associates, with the data store information, information associating the target data mart with a new data mart obtained by applying the filter or applying the predetermined algorithm. When the new data mart becomes an application target of the target data mart or filter or algorithm due to a new narrowing request after recording and recording, meta information indicated by the new narrowing request is Reuse the new data mart as the target of the target data mart or filter or algorithm in response to the new narrowing request if it is identical to the narrowed meta information recorded for the new data mart It may be a thing.

  According to this, for example, a new data mart obtained by applying a filter or applying an algorithm (analytical application) for a specific event, that is, a narrowed data mart, generates a request for narrowing the same event as the above-mentioned event. It can be diverted as it is. As a result, it is possible to flexibly reuse the data mart in response to the user's request, and to narrow down the analysis result on the data mart more efficiently.

  Further, in the data management system of the present embodiment, the arithmetic device collects values corresponding to respective items of the data store information from the data lake and the data mart provided in a predetermined device, and the data lake and the data A process of generating a record corresponding to each of the mart and storing the record in the data store information, and an application event of the algorithm to the data lake or the data mart by monitoring each item of the data store / application connection information Collect corresponding values, generate records corresponding to a series of relationships of the data lake, the data mart, and an algorithm applied to the data lake or the data mart, and store the records in the data store / application connection information Further processing In it, it may be used as the.

According to this, each time the data management method of this embodiment is executed, the data store information and the data store / application connection information are stored, and can be used for data management thereafter. As a result, it is possible to flexibly reuse the data mart in response to the user's request, and to narrow down the analysis result on the data mart more efficiently.

  Further, in the data management method of the present embodiment, when the information processing system selects the data mart including the meta information among the data mart from the target data mart to the data lake, the data including the meta information The data mart closest to the target data mart may be selected from the mart.

  Further, in the data management method of the present embodiment, for the new data mart obtained by the application of the filter or the application of the predetermined algorithm, the information processing system performs a predetermined attribute of meta information designated by the narrowing request. When the new data mart is subject to application of the target data mart or filter or algorithm due to a new narrowing request after recording in the data store information as narrowed meta information If the meta information indicated by the new narrowing request is different from the narrowed meta information recorded regarding the new data mart, the process corresponding to the new narrowing request may be avoided.

  Further, in the data management method according to the present embodiment, the information processing system associates the data store information with the target data mart and a new data mart obtained by applying the filter or applying the predetermined algorithm. If the new data mart is subject to application of the target data mart or filter or algorithm due to a new narrowing request after the recording, the meta information indicated by the new narrowing request is , Reuse the new data mart as an application target of the target data mart or filter or algorithm in response to the new narrowing request if it is the same as the narrowed meta information recorded regarding the new data mart. You may do it.

  Further, in the data management method according to the present embodiment, the information processing system collects values corresponding to respective items of the data store information from the data lake and the data mart provided in a predetermined device, and A process of generating a record corresponding to each of the data mart and storing it in the data store information, and monitoring an application event of the algorithm to the data lake or the data mart, and each item of the data store / application connection information Collecting values corresponding to the data store, generating a record corresponding to a series of relationships of the data lake, the data mart, and an algorithm applied to the data lake or the data mart to the data store / application connection information Execute further processing to store That may be.

001 Analysis Application Developer / User Terminal 002 Factory Equipment 003 Network 10 Data Management System 100 Data Catalog Management Server 101 Storage Device 102 Program 103 Memory 104 Arithmetic Unit 105 Communication Device 110 Information Retrieval Unit 111 Data Path Creation Unit 112 Filter Creation Unit 113 Data store information creation unit 114 Data store / application connection information creation unit 125 Data store information 126 Data store / application connection information 127 application information 200 IoT infrastructure execution server 210 application execution unit 211 data recording unit 212 data collection unit 225 data lake 226 data Mart 2261 1st data mart 2262 2nd data mart 2263 1st data mart 2264 2nd data mer

Claims (10)

Data store information storing information on each of a data lake storing predetermined data and a data mart storing a predetermined analysis result on the predetermined data, the data lake, the data mart, and the data lake or the data mart A storage / application connection information storing data path information indicating a series of relationships of an algorithm responsible for analysis processing for the
Based on the narrowing request of the analysis result received from the predetermined terminal and the data store information, it is determined whether the target data mart indicated by the narrowing request includes predetermined meta information, and when the meta information is not included, the above In the data path including the target data mart based on the data store information and the data store / application connection information, the data mart from the target data mart to the data lake including the meta information is selected in the data path including the target data mart, A new data path including a filter related to the meta information, which is a type of the algorithm applied to the data mart, a new data mart obtained by applying the filter, and a predetermined algorithm applied to the new data mart Generate according to a predetermined rule and follow the new data path By executing the application of said predetermined algorithm to application and the new data mart filter to the selected data marts, an arithmetic unit for generating a predetermined analysis result corresponding to the narrowing request,
A data management system comprising: The arithmetic device is
In the data mart from the target data mart to the data lake, when selecting the data mart including the meta information, the data mart closest to the target data mart is selected from the data mart including the meta information. is there,
The data management system according to claim 1, characterized in that: The arithmetic device is
For a new data mart obtained by the application of the filter or the application of the predetermined algorithm, the predetermined attribute of the meta information designated by the narrowing request is recorded as the narrowed meta information in the data store information, and When the new data mart is subject to application of the target data mart or filter or algorithm due to a new narrowing request, the meta information indicated by the new narrowing request is recorded for the new data mart. If it is different from the narrowed down meta information, the process corresponding to the new narrowing request is avoided.
The data management system according to claim 1, characterized in that: The arithmetic device is
Information linking the target data mart and a new data mart obtained by application of the filter or application of the predetermined algorithm is recorded in the data store information, and after the recording, due to a new narrowing request If the new data mart is to be applied to the target data mart or filter or algorithm, the meta information indicated by the new narrowing request is the same as the narrowed meta information recorded for the new data mart Then, in response to the new narrowing request, the new data mart is reused as an application target of the target data mart or filter or algorithm.
The data management system according to claim 3, characterized in that: The arithmetic device is
Values corresponding to the respective items of the data store information are collected from the data lake and the data mart provided in a predetermined device, records corresponding to the data lake and the data mart are respectively generated, and the data store information is generated. Monitoring the process of storing and application of the algorithm to the data lake or the data mart to collect values corresponding to respective items of the data store / application connection information; and the data lake, the data mart, and Generating a record corresponding to a series of relationships of the algorithm applied to the data lake or the data mart, and storing the record in the data store / application connection information.
The data management system according to claim 1, characterized in that: Data store information storing information on each of a data lake storing predetermined data and a data mart storing a predetermined analysis result on the predetermined data, the data lake, the data mart, and the data lake or the data mart An information processing system comprising: a storage device holding data store application connection information storing information of a data path indicating a series of relationships of an algorithm responsible for analysis processing for
Based on the narrowing request of the analysis result received from the predetermined terminal and the data store information, it is determined whether the target data mart indicated by the narrowing request includes predetermined meta information, and when the meta information is not included, the above In the data path including the target data mart based on the data store information and the data store / application connection information, the data mart from the target data mart to the data lake including the meta information is selected in the data path including the target data mart, A new data path including a filter related to the meta information, which is a type of the algorithm applied to the data mart, a new data mart obtained by applying the filter, and a predetermined algorithm applied to the new data mart Generate according to a predetermined rule and follow the new data path By executing the application of predetermined algorithms of application and the to a new data mart of the filter to the selected data mart, generates a predetermined analysis result corresponding to the narrowing request,
Data management method characterized by The information processing system
When selecting data mart including the meta information among the data mart from the target data mart to the data lake, select the data mart closest to the target data mart from the data mart including the meta information.
7. The data management method according to claim 6, wherein: The information processing system
For a new data mart obtained by the application of the filter or the application of the predetermined algorithm, the predetermined attribute of the meta information designated by the narrowing request is recorded as the narrowed meta information in the data store information, and When the new data mart is subject to application of the target data mart or filter or algorithm due to a new narrowing request, the meta information indicated by the new narrowing request is recorded for the new data mart. If it is different from the filtered down meta information, the process corresponding to the new narrowing request is avoided.
7. The data management method according to claim 6, wherein: The information processing system
Information linking the target data mart and a new data mart obtained by application of the filter or application of the predetermined algorithm is recorded in the data store information, and after the recording, due to a new narrowing request If the new data mart is to be applied to the target data mart or filter or algorithm, the meta information indicated by the new narrowing request is the same as the narrowed meta information recorded for the new data mart Then, in response to the new narrowing request, the new data mart is reused as an application target of the target data mart or filter or algorithm.
The data management method according to claim 8, characterized in that: The information processing system
Values corresponding to the respective items of the data store information are collected from the data lake and the data mart provided in a predetermined device, records corresponding to the data lake and the data mart are respectively generated, and the data store information is generated. Monitoring the process of storing and application of the algorithm to the data lake or the data mart to collect values corresponding to respective items of the data store / application connection information; and the data lake, the data mart, and Generating a record corresponding to a series of relationships of the algorithm applied to the data lake or the data mart, and storing the record in the data store / application connection information.
7. The data management method according to claim 6, wherein:

Images