JP7010632B2 - Intermediate data management system and intermediate data management method - Google Patents

Description

The present invention relates to an intermediate data management system and an intermediate data management method. Specifically, the present invention aims at effective management of intermediate data and, by extension, improves the efficiency of analysis processing using the intermediate data. Regarding technology.

Various analytical techniques have appeared that analyze various data such as so-called big data and obtain knowledge from an unprecedented viewpoint. When performing such data analysis, it is often the case that an analysis system is constructed in which a plurality of analysis resources are shared, the processing result of one analysis resource is targeted for analysis by another analysis resource, and the analysis resources cooperate with each other.

In that case, a lot of intermediate data is generated at each analysis opportunity from the data that became the analysis starting point to the analysis result, and a considerable amount of labor and resources are required for its management.

Therefore, as a conventional technique for managing such intermediate data, for example, a data analysis system that analyzes raw data with a computer equipped with a processor and a storage device and outputs the analysis result, and stores the raw data. The raw data storage unit, the analysis unit that reads the raw data and performs analysis, generates intermediate data in the process of the analysis and outputs the analysis result, and the intermediate data that stores the intermediate data generated by the analysis unit. The storage unit is provided with an evaluation reception unit that receives an evaluation value for the analysis result output by the analysis unit, and the analysis unit is an intermediate data that can be used among the intermediate data of the intermediate data storage unit during the analysis. With reference to the data, the evaluation receiving unit distributes the evaluation value to the intermediate data corresponding to the evaluation value, and when the distributed evaluation value satisfies a predetermined condition, the evaluation value. A data analysis system (see Patent Document 1), which is characterized by deleting the intermediate data corresponding to the above, has been proposed.

Japanese Unexamined Patent Publication No. 2011-2911

In the prior art, it is difficult to effectively reduce the used capacity of the data storage even if the intermediate data is deleted. In addition, the intermediate data once deleted cannot be referred to even if it is needed later. These problems can also adversely affect the processing efficiency itself in the analytical system.
Therefore, an object of the present invention is to provide a technique for effectively managing intermediate data and, by extension, improving the efficiency of analysis processing using the intermediate data.

The intermediate data management system of the present invention that solves the above problems includes analysis data, metadata showing various attributes of the analysis data, and data for generating generated data generated based on other analysis data among the analysis data. From the start point to the end point in a series of data processing flows starting from the analysis data and ending from the analysis result of the analysis data based on the storage device for storing the data and the metadata and the generation data. A computing device that selects predetermined intermediate data from the generated data by a predetermined algorithm and executes deletion processing of the intermediate data in each of the sequence of data directed to the sequence and the sequence of data traced from the end point to the start point. , It is characterized by providing.

Further, the intermediate data management method of the present invention stores analysis data, metadata showing various attributes of the analysis data, and data for generating generated data generated based on other analysis data among the analysis data. From the starting point in a series of data processing flows starting from the analysis data and ending at the analysis result of the analysis data based on the metadata and the generation data. In each of the sequence of data toward the end point and the sequence of data tracing from the end point to the start point, a predetermined intermediate data among the generated data is selected by a predetermined algorithm, and the deletion process of the intermediate data is executed. , Characterized by that.

According to the present invention, the intermediate data can be effectively managed, and the efficiency of the analysis process using the intermediate data can be improved.

It is a figure which shows the analysis concept example from the analysis data to the analysis result in this embodiment. It is a figure which shows the configuration example of the intermediate data management system in this embodiment. It is a figure which shows the hardware configuration example of various servers in this embodiment. It is a figure which shows the configuration example of the data receiving server in this embodiment. It is a figure which shows the configuration example of the analysis data display server in this embodiment. It is a figure which shows the configuration example of the data analysis server in this embodiment. It is a figure which shows the configuration example of the intermediate data deletion server in this embodiment. It is a figure which shows the composition example of the metadata in this embodiment. It is a figure which shows the structural example of the generation data in this embodiment. It is a figure which shows the configuration example of the deletion policy in this embodiment. It is a figure which shows the sequence example 1 of the intermediate data management method in this embodiment. It is a figure which shows the sequence example 2 of the intermediate data management method in this embodiment. It is a figure which shows the sequence example 3 of the intermediate data management method in this embodiment. It is a figure which shows the sequence example 4 of the intermediate data management method in this embodiment. It is a figure which shows the flow example 1 of the intermediate data management method in this embodiment. It is a figure which shows the flow example 2 of the intermediate data management method in this embodiment. It is a figure which shows the flow example 3 of the intermediate data management method in this embodiment. It is a figure which shows the flow example 4 of the intermediate data management method in this embodiment.

－－－ About the conceptual example of data analysis －－－
FIG. 1 is a diagram conceptually showing the execution process of data analysis assumed in the present embodiment, and is an analysis concept from the analysis data which is the starting point of the analysis target to the analysis result finally presented to the analyst. This is an example.

In the data analysis system assumed in the present embodiment, a cycle is executed in which a predetermined analysis process is executed by using the analysis data that is the starting point of the analysis as an input, and a different analysis process is executed by using the output obtained by the analysis process as an input. It is an information processing system that operates by repeating as necessary and finally obtaining the analysis result desired by the analyst.

In such an analysis process, the analysis data 102 and the analysis data 112 are the analysis data that are the starting points of the analysis. The analysis data 102 and the analysis data 112 can be assumed to be acquired as electronic files from a data source terminal 201 (FIG. 2) such as a business system or a sensor. On the other hand, the intermediate data 122 is obtained by executing the data processing 161 with the above-mentioned analysis data 102 as an input. This data processing 161 assumes a process called "conversion 1" as an example.

The association 181 exemplified in FIG. 1 is a link indicating that the input of the above-mentioned data processing 161 is the analysis data 102, and the association 182 indicates that the output of the data processing 161 is the intermediate data 122. It's a link.

Similarly, the intermediate data 132 is obtained as the execution result of the data processing 162 with the analysis data 112 as the input data. Here, the data processing 162 assumes a processing called "conversion 2".

Further, the intermediate data 142 is obtained as the execution result of the data processing 163 by using the above-mentioned intermediate data 122 and the intermediate data 132 as input data. Here, the data processing 163 assumes a processing to which the "analysis algorithm A" is applied.

Further, the analysis result 152 is obtained as the execution result of the data processing 164 using the above-mentioned intermediate data 142 as input data. Here, the data processing 164 assumes a processing to which the "analysis algorithm B" is applied.

The analysis result 152 corresponding to the end of such a series of data analysis processes is useful information for the analyst. Valid information is the analysis result intended by the analyst. For example, it is possible to assume analysis data showing the correlation / causal relationship between the flow data at the store and the sales data in the retail company that sells the product. be. Such analytical data is expected to lead to an increase in corporate value at the company.

The above-mentioned analysis result 152 is distributed to the data analysis display client 207 (FIG. 2) used by the analyst as necessary, and is subject to judgment by the analyst. The logic for determining whether the analysis result is intended by the analyst corresponds to, for example, verifying that the correlation coefficient between the target events in the above-mentioned correlation / causal relationship is large.

In the present embodiment, for each of the above-mentioned analysis data, intermediate data, and analysis result, metadata showing the characteristics thereof and generation data which is information necessary for generating the corresponding data are managed. For each of the analysis data, the intermediate data, and the analysis result, the corresponding data and the corresponding metadata and generation data are referred to as data nodes for convenience.
In the example shown in FIG. 1, the data node 101 is composed of analysis data 102 and metadata / generation data 103 (a set of metadata and generation data).

Similarly, the data node 111 is the analysis data 112 and the metadata / generation data 113, the data node 121 is the intermediate data 122 and the metadata / generation data 123, and the data node 131 is the intermediate data 132 and the metadata. In the generation data 133, the data node 141 is composed of the intermediate data 142 and the metadata / generation data 143, and the data node 151 is composed of the analysis result 152 and the metadata / generation data 153, respectively.

The association 171 in FIG. 1 is a link indicating that the input when the intermediate data 122 is generated is the analysis data 102 in the data node 101. The information that the input when the intermediate data 122 is generated is the analysis data 102 in the data node 101 is recorded in the metadata / generation data 123. Further, it is also recorded in the metadata / generation data 123 that the data processing 161 is “conversion 1”.

Similarly, the association 172 is a link indicating that the input when the intermediate data 132 is generated is the analysis data 112 at the data node 111. The information that the input when the intermediate data 132 is generated is the analysis data 112 in the data node 111 and the information that the data processing 162 is "conversion 2" are recorded in the metadata / generation data 133. ing.

Further, in the relation 173, one of the inputs when the intermediate data 142 is generated is the intermediate data 122 in the data node 121, and in the relation 174, one of the inputs when the intermediate data 142 is generated is the data node 131. It is a link indicating that it is the intermediate data 132 in. The information that the input when the intermediate data 142 is generated is the intermediate data 122 in the data node 121 and the intermediate data 132 in the data node 131, and the information that the data processing 163 is the "analysis algorithm A" are metadata. It is recorded in the generation data 143.

Further, the relation 175 is a link indicating that the input when the analysis result 152 is generated is the intermediate data 142 in the data node 141. The information that the input when the analysis result 152 is generated is the intermediate data 142 in the data node 141 and the information that the data processing 164 is the "analysis algorithm B" are recorded in the metadata / generation data 153. Has been done.

In the present embodiment, by managing the metadata and the generation data for each of the intermediate data as described above, even if the intermediate data in the corresponding data node is deleted, the corresponding intermediate data is regenerated thereafter. Is possible.

For example, the intermediate data 122 of the data node 121, which is the data node of the intermediate data (hereinafter referred to as the intermediate data node), refers to the metadata / generation data 123, and the data node 101 is input in the generation thereof. It can be seen from the analysis data 102 of the above that the algorithm used for the generation is the "conversion 1" of the data processing 161. Therefore, by inputting the analysis data 102 of the data node 101 and giving the information obtained from the metadata / generation data 123 as a parameter to the "conversion 1" of the data processing 161, the intermediate data 122 of the data node 121 is regenerated. can.

Similarly, the intermediate data 142 of the data node 141, which is an intermediate data node, refers to the metadata / generation data 143, and the input in the generation is the intermediate data 122 of the data node 121 and the data node 131. It can be seen that the intermediate data 132 of the above, and the algorithm used for the generation is the “analysis algorithm A” of the data processing 163. Therefore, by inputting the intermediate data 122 of the data node 121 and the intermediate data 132 of the data node 131 and giving the information obtained from the metadata / generation data 143 as parameters to the "analysis algorithm A" of the data processing 163, the data can be obtained. The intermediate data 142 of node 141 can be regenerated.

The details of the above-mentioned metadata and generation data will be described later based on FIGS. 8A and 8B. The details of the intermediate data deletion process will be described later with reference to FIGS. 13 to 17.

--- System configuration example ---
FIG. 2 is a diagram showing a configuration example of the intermediate data management system 200 according to the present embodiment. The intermediate data management system 200 includes a data receiving server 202, a data storage server 203, a data analysis server 204, an intermediate data deletion server 205, a data analysis display server 206, and a data analysis display client 207.

Such an intermediate data management system 200 is communicably connected to the data source terminal 201 via an appropriate network 208 such as the Internet. Similarly, the intermediate data management system 200 is communicably connected to the data analysis display client 207 via an appropriate network 209 such as a LAN. Of course, at least one of the above-mentioned data source terminal 201 and data analysis display client 207 may be included in the intermediate data management system 200.

As described above, the network 208 can be assumed to be a wireless network or a wired network provided by a communication carrier such as the Internet. The network 208 may include a network owned by an individual company or the like as a part thereof, or may be a network through which a plurality of types of protocols are passed.

Further, among the above-mentioned network configurations, the data source terminal 201 has a function of collecting data that is an observation result of an observation target by a predetermined sensor or the like from the sensor or the like, and an intermediate data management system 200 that collects the collected data via the network 208. It is a device having a function of transmitting to the data receiving server 202 of the above. The data source terminal 201 is a device that distributes the analysis data 102 and the analysis data 112 to the intermediate data management system 200 in the example of FIG.

Further, the data receiving server 202 constituting the intermediate data management system 200 receives data transmitted from the data source terminal 201 via the network 208, and requests the data storage server 203 to store this data as analysis data. Is. The detailed configuration of the data receiving server 202 will be described later.

Further, the data storage server 203 is a server device that stores data used by other servers constituting the intermediate data management system 200. If the data is a file, it is a shared file server, if it is stored as a record, it is a structural database server, and if it is stored in a format such as json, it is a server that stores data such as an unstructured database such as a key value store. Applies to.

Such a data storage server 203 has various storage units for storing the analysis data, the intermediate data, the analysis result, the metadata, and the generation data in the example of FIG. Details of the storage unit included in the data storage server 203 will be described later with reference to FIGS. 8 and 9.

Further, the data analysis server 204 receives a data analysis execution request from the data analysis display server 206, and acquires and acquires analysis data and intermediate data to be analyzed from the data storage server 203 according to the content of the data analysis execution request. It is a server device that executes data analysis on the generated data by a predetermined analysis program. The data analysis server 204 is a server having a function of storing the analysis result obtained in this way in the data storage server 203 and returning the analysis result to the data analysis display server 206. Such a data analysis server 204 executes the data processing 161 to 164 in the example of FIG. 1 by the above-mentioned analysis program. Details of the data analysis server 204 will be described later with reference to FIG.

Further, the intermediate data deletion server 205 detects the intermediate data deletion timing that arrives at predetermined intervals by, for example, a built-in timer, or receives an intermediate data deletion request from an appropriate terminal such as the data analysis display client 207. Then, the deletion policy, metadata, and generation data of the intermediate data are acquired from the data storage server 203, the intermediate data of the deletion candidate is selected according to the deletion policy, and the selected intermediate data to be deleted is deleted. It is a server device having a function of requesting a request to the data storage server 203.
In the example of FIG. 1, the intermediate data deletion server 205 has intermediate data 122, 132, 14
A process of determining whether 2 is a deletion target will be performed. The example of FIG. 1 shows an example in which the intermediate data 122 and the intermediate data 142 are determined to be deleted (“x” is assigned in the figure, and the object is a broken line). Details of such an intermediate data deletion server 205 will be described later in FIG.

Further, the data analysis display server 206 receives a processing request such as a data analysis request from the data analysis display client 207, and based on the received processing request, makes a processing request to the data analysis server 204 and the data storage server 203, and processes the data. It is a server device that executes a process of returning the result to the data analysis client 207. Details of this data analysis display server 206 will be described later with reference to FIG.

Further, the data analysis display client 207 is a terminal used by an analyst, and is a terminal that generates an execution request or the like for data analysis processing and transmits the request to the data analysis display server 206. The data analysis display client 207 is an information processing device provided with a communication device corresponding to the communication protocol of the network 209, such as a personal computer or a server in a company or a factory, or a smartphone or a tablet terminal.

Here, an example is shown in which the above-mentioned servers are different physical machines, but these servers may be different virtual machines of the same physical machine, or they may be different virtual machines of different physical machines. You may. Alternatively, it may be mixed as a physical machine and a virtual machine.

--- Hardware configuration example ---
FIG. 3 is a diagram showing a common physical configuration in the data receiving server 202, the data storage server 203, the data analysis server 204, the intermediate data deletion server 205, and the data analysis display server 206 shown in FIG. ..

These servers are composed of a computer having a CPU 301, a memory 302, an auxiliary storage device 303, and a communication interface 304, which are arithmetic units.
Of these, the CPU 301 is an arithmetic unit that reads the program 3031 from the auxiliary storage device 303 into the memory 302 and executes it to implement necessary functions.

Further, the memory 302 includes a ROM which is a non-volatile storage element and a RAM which is a volatile storage element. The ROM stores an invariant program (for example, BIOS) and the like. The RAM is a high-speed and volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores data used when the program 3031 and the program 3031 executed by the CPU 301 are executed.

Further, the auxiliary storage device 303 is, for example, a large-capacity non-volatile storage device such as a magnetic storage device (HDD) or a flash memory (SSD), and is used when the program 3031 and the program 3031 executed by the CPU 301 are executed. Store data. That is, the program 3031 is read from the auxiliary storage device 3, loaded into the memory 302, and executed by the CPU 301.
Further, the communication interface 304 is a network interface device that controls communication with other devices according to the protocol of the network to be connected.

The servers in this embodiment may have an input interface 305 and an output interface 308. The input interface 305 is an interface to which a keyboard 306, a mouse 307, or the like is connected and receives input from an operator. The output interface 308 is an interface to which a display device 309, a printer, or the like is connected, and outputs a program execution result in a format that can be visually recognized by an operator.

The program 3031 executed by the CPU 301 is provided to the server via a removable medium (CD-ROM, flash memory, etc.) or a network, and is stored in the non-volatile storage device 303, which is a non-temporary storage medium. For this reason, the server may have an interface to read data from removable media.

FIG. 4 is a diagram showing functional blocks of the data receiving server 202 and the data storage server 203 in the present embodiment. The data reception server 202 includes a data reception processing unit 401 and a data acquisition / storage / deletion request unit 402. On the other hand, the data storage server 203 includes a data acquisition / storage / deletion reception unit 403, an analysis data storage unit 411, and a metadata storage unit 412.

Of these, the data reception processing unit 401 of the data reception server 202 receives the analysis data transmitted from the data source terminal 201, and confirms the validity of the data contents of the analysis data by an appropriate algorithm (example: data item and its). An algorithm that determines whether the data format conforms to a predetermined data format. The same applies below. If the data content is appropriate, the meta related to the received analysis data (including intermediate data and analysis results; the same applies below). The data is generated, and the data acquisition / storage / deletion request unit 402 is requested to store the received analysis data and the generated metadata. Further, the data reception processing unit 401 receives the storage result returned from the data acquisition / storage / deletion request unit 402 in response to the above request, and transmits this to the data source terminal 201.

On the other hand, the above-mentioned data acquisition / storage / deletion request unit 402 receives the request from the data reception processing unit 401 and requests the data acquisition / storage / deletion reception unit 403 of the data storage server 203 to store the data. In this case, the data acquisition / storage / deletion request unit 402 returns the storage result returned from the data acquisition / storage / deletion reception unit 403 of the data storage server 203 to the data reception processing unit 401 of the data reception server 202.

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 receives the storage request of the analysis data and its metadata from the data acquisition / storage / deletion request unit 402 of the data reception server 202, and analyzes the analysis data. It is stored in the data storage unit 411, the metadata is stored in the metadata storage unit 412, and the analysis data and the storage result of the metadata are transmitted to the data acquisition / storage / request unit 402 of the data receiving server 202.

Further, the analysis data storage unit 411 of the data storage server 203 stores the analysis data received from the data source terminal 201, the intermediate data by the data analysis server 204, and the analysis result. Further, the metadata storage unit 412 of the data storage server 203 stores the analysis data received from the data source terminal 201, the intermediate data by the data analysis server 204, and the metadata which is the attribute information of the analysis result. Details of the metadata storage unit 412 will be described later in FIG.

The details of the operation sequence in which the data receiving server 202 receives the analysis data transmitted from the data source terminal 201 and stores the corresponding analysis data and the metadata in the data storage server 203 will be described later with reference to FIG.
FIG. 5 is a diagram showing functional blocks of the analysis data display server 206 and the data storage server 203 in the present embodiment.

The analysis data display server 206 of the present embodiment includes a display request reception unit 501, a display information generation unit 502, a data analysis execution request unit 503, and a data acquisition / storage / deletion request unit 504. ..

On the other hand, the data storage server 203 further includes an analysis data storage unit 411 and a metadata storage unit 412 already shown in FIG. 4, a generation data storage unit 511, and an analysis program information storage unit 512 (FIG. 4). Although it differs in the configuration of 4 and the storage unit, only the configuration necessary for the explanation here is picked up and shown).

The display request receiving unit 501 of the analysis data display server 206 described above receives a display information acquisition request from the data analysis display client 207, confirms the validity of the content of the acquisition request received here, and if the content is valid. For example, the display information generation unit 502 is requested to generate display information according to the content of the acquisition request. Further, the display request receiving unit 501 transmits the display information returned from the display information generation unit 502 to the data analysis display client 207.

Further, the display information generation unit 502 of the analysis data display server 206 receives a request for acquisition of display information from the display request reception unit 501, and requests the data analysis execution request unit 503 to execute data analysis according to the request content. Alternatively, the data acquisition request is notified to the data acquisition / storage / deletion request unit 504, respectively, and display information is generated from the reply results of the data analysis execution request unit 503 and the data acquisition / storage / deletion request unit 504, and the generated display information is generated. Is replied to the display request reception unit 501.

Further, the data analysis execution request unit 503 of the analysis data display server 206 receives a data analysis execution request from the display information generation unit 502, and according to the content of the execution request, the data analysis execution reception unit 505 of the data analysis server 204. Send a data analysis execution request to. Further, the data analysis execution request unit 503 of the analysis data display server 206 returns the execution result transmitted from the data analysis execution reception unit 505 of the data analysis server 204 to the display information generation unit 502.

Further, the data acquisition / storage / deletion request unit 504 of the analysis data display server 206 receives a data acquisition request from the display information generation unit 502, and the data acquisition / storage / deletion reception unit of the data storage server 203 according to the request content. A data acquisition request is sent to 403, and the data acquired from this data acquisition / storage / deletion reception unit 403 is returned to the display information generation unit 502.

Further, the generation data storage unit 511 of the data storage server 203 stores input information required when the analysis program 602 (described later) of the data analysis server 204 generates intermediate data and analysis results. The input information required for the analysis program 602 to generate the intermediate data and the analysis result is the analysis data to be analyzed, the list of the intermediate data, the type of the analysis program used for the data generation, and the data generation. At that time, the parameters applied to the analysis program can be assumed. The details of the generation data storage unit 511 will be described later in FIG.

Further, the analysis program information storage unit 512 of the data storage server 203 stores information such as the type of the analysis program 602 of the data analysis server 204 and the parameters that need to be set in the analysis program 602.

The analysis data display server 206 receives a display information acquisition request from the data analysis display client 207, acquires necessary data from the data storage server 203 according to the request content, and uses the acquired data as display information. The details of the operation sequence of generating and transmitting the display information to the data analysis display client 207 will be described later with reference to FIG.

Further, the data analysis server 204 receives an analysis execution request from the data analysis display client 207 via the analysis data display server 206, and acquires and acquires the necessary data from the data storage server 203 according to the content of the request. An operation of executing the analysis program 602 using the generated data as an input, storing the execution result of the analysis in the data storage server 203, generating display information related to the execution result, and transmitting the generated display information to the data analysis display client 207. Details of the sequence will be described later in FIG.

FIG. 6 is a diagram showing a functional block of the data analysis server 204 in the present embodiment. The data analysis server 204 includes a data analysis execution reception unit 505, an analysis execution management unit 601, an analysis program 602, and a data acquisition / storage / deletion request unit 602.

Of these, the data analysis execution reception unit 505 receives the data analysis execution request from the data analysis execution request unit 503 of the data analysis display server 206, confirms the validity of the content of the execution request, and the content of the execution request is When appropriate, a data analysis execution request is transmitted to the analysis execution management unit 601. Further, the data analysis execution reception unit 505 returns the data analysis execution result returned from the analysis execution management unit 601 to the data analysis execution request unit 503 of the data analysis display server 206.

Further, the analysis execution management unit 601 of the data analysis server 204 receives the data analysis execution request transmitted from the data analysis execution reception unit 505 described above, and executes the analysis program 602 based on the content of the execution request.

Further, the analysis execution management unit 601 creates metadata showing the characteristics of the intermediate data or the analysis result, which is the analysis execution result returned from the analysis program 602, and input information to the analysis program 602 at the time of analysis execution. Create some data for generation and execute.

Further, the analysis execution management unit 601 stores the intermediate data or the analysis result output from the analysis program 602, the metadata created as described above, and the generation data together with the storage request, and the data acquisition / storage / deletion request unit 602. Also, the intermediate data or the analysis result is returned to the data analysis execution reception unit 505.

Further, the analysis program 602 of the data analysis server 204 receives an execution request for data analysis transmitted from the analysis execution management unit 601 and executes data analysis based on the input information of the execution request, and intermediate data which is the execution result. Alternatively, the analysis result is returned to the analysis execution management unit 601.

As described above, in a data analysis system, analysis is generally performed by repeating a plurality of data processes. Therefore, there are a plurality of types of analysis programs 602. In the example of FIG. 1, the "conversion 1" of the data processing 163, the "conversion 2" of the data processing 162, the "analysis algorithm A" of the data processing 163, and the "analysis algorithm B" of the data processing 164 are the analysis programs 602, respectively. Corresponds to the type.

Further, the data acquisition / storage / deletion request unit 603 of the data analysis server 204 receives the data storage request from the analysis execution management unit 601 and transmits the data storage request to the data acquisition / storage / deletion reception unit 403 of the data storage server 203. do. Further, the data acquisition / storage / deletion request unit 603 returns the storage request result returned from the data acquisition / storage / deletion reception unit 403 of the data storage server 203 to the analysis execution management unit 601.

FIG. 7 is a diagram showing functional blocks of the intermediate data deletion server 205 and the data storage server 203 in the present embodiment. The intermediate data deletion server 205 of the present embodiment includes an intermediate data deletion periodic request unit 701, an intermediate data deletion management unit 702, and a data acquisition / storage / deletion request unit 703. On the other hand, the data storage server 203 includes the deletion policy storage unit 711 in addition to the configuration already described above.

Of these, the intermediate data deletion periodic request unit 701 of the intermediate data deletion server 205 detects the arrival of the intermediate data deletion time by a timer or the like in the server (or receives an instruction from the user), and the intermediate data deletion management unit 702 Request to delete the intermediate data.

Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 receives the intermediate data deletion request from the intermediate data deletion periodic request unit 701, and requests the data acquisition / storage / deletion request unit 703 for the deletion policy, metadata, and deletion policy. , The generation data, the deletion policy of the intermediate data, the metadata, and the generation data are acquired, the intermediate data to be deleted is selected based on these, and the selected intermediate data is deleted. The request and the update request of the corresponding metadata are transmitted to the data acquisition / storage / deletion request unit 703. Details of the processing flow in the intermediate data deletion management unit 702 will be described later with reference to FIGS. 14 to 17.

Further, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 appropriately receives any one of the data acquisition, storage, and deletion requests from the above-mentioned intermediate data deletion management unit 702, and makes the request. Data acquisition / storage / deletion reception unit 403 of the data storage server 203 is requested to acquire, store, or delete the corresponding data according to the contents of the data acquisition / storage / deletion reception unit 403. The acquisition request result, the storage request result, and the deletion request result returned from are returned to the intermediate data deletion management unit 702.

On the other hand, the deletion policy storage unit 711 of the data storage server 203 stores the deletion policy, which is information that defines the rules for selecting the intermediate data to be deleted. The details of the deletion policy storage unit 711 will be described later with reference to FIG.

The intermediate data deletion management unit 702 of the intermediate data deletion server 205 receives an intermediate data deletion request from the intermediate data deletion periodic request unit 701, acquires a deletion policy from the data storage server 203, and deletes the data based on the data deletion policy. The details of the operation sequence in which the target intermediate data is selected, a deletion request is made to the data storage server 203, and the metadata of the intermediate data is updated by the data storage server 203 will be described later with reference to FIG.

--- Data structure example ---
Subsequently, the tables used by the intermediate data management system 200 of the present embodiment will be described. FIG. 8A is a diagram showing an example of a table configuration of the metadata storage unit 412 held in the data storage server 203 of the present embodiment.
The metadata storage unit 412 of the present embodiment stores data indicating the characteristics of the analysis data, the intermediate data, and the analysis result, that is, the metadata.

The table of the metadata storage unit 412 exemplified in FIG. 8A has a data node ID column 1601 for uniquely identifying a data node and target data (any of analysis data, intermediate data, and analysis results; the same applies hereinafter). The data size column 1602 that stores the data size of the data, the data storage path column 1603 that stores the location where the target data is stored, the final access time column 1604 that stores the final time when the target data was accessed by the analyst, and the target. Data processing by back-converting the data generation time column 1605 that stores the time when the data was generated, the data generation time column 1606 that stores the time required to generate the target data by data processing, and the target data. A data inverse conversion generation estimated time column 1607 that stores the estimated time required to regenerate the previous target data, and a generation data ID column 1608 that stores the external key of the generation data related to the corresponding data node. Consists of.
Further, FIG. 8B is a diagram showing a table configuration example of the generation data storage unit 511 held in the data storage server 203 of the present embodiment.

The generation data storage unit 511 of the present embodiment stores data indicating what kind of input and data processing the target data generated in the analysis process was generated, that is, generation data. The table of the generation data storage unit 511 exemplified in FIG. 8B includes a generation data ID column 1609 for uniquely specifying the generation data, and a data node ID of the original data input when the target data is generated. The original data list column 1610 for storing the target data, the processing algorithm column 1611 for storing the type of data processing used to generate the target data, and the value used as the data processing parameter when generating the target data. It is configured to include a forward data generation parameter column 1612 to be stored and a reverse data generation parameter column 1613 to store the value of the parameter required when the original data is generated from the target data by the reverse conversion data processing. To.
Further, FIG. 9 is a diagram showing an example of a table configuration of the deletion policy storage unit 711 held in the data storage server 203 of the present embodiment.

The table of the deletion policy storage unit 711 exemplified in FIG. 9 has a policy ID column 1701 for uniquely specifying the deletion policy of the intermediate data and a data size lower limit column representing the minimum value of the data size of the intermediate data to be deleted. 1702, the final access time lower limit column 1703 that represents the minimum access idle time by the analyst of the intermediate data to be deleted, and the generation elapsed time lower limit that represents the minimum elapsed time from the generation time of the intermediate data to be deleted. It includes a column 1704 and a regeneration time upper limit column 1705 that represents the maximum value of the reproduction time of the intermediate data to be deleted.

－－－－ Example of procedure for intermediate data management method －－－－
Hereinafter, the actual procedure of the intermediate data management method in the present embodiment will be described with reference to the drawings. Various operations corresponding to the intermediate data management method described below are realized by a program read into a memory or the like by each server constituting the intermediate data management system 200 and executed. The program is composed of codes for performing various operations described below.

FIG. 10 is a diagram showing a sequence example 1 of the intermediate data management method in the present embodiment. Specifically, the data source terminal 201 transmits analysis data to the data receiving server 202, and the data receiving server 202 performs the analysis. It is a figure which shows the example of a series of operation sequences which generate the metadata about data, and the data storage server 203 stores the analysis data and the said-mentioned metadata.
In this case, first, the data source terminal 201 transmits the analysis data to be analyzed to the data reception processing unit 401 of the data reception server 202 (S801).

Here, the analysis data transmitted by the data source terminal 201 is data that is the starting point of data analysis. This analysis data corresponds to the analysis data 102 and the analysis data 112 in FIG.

On the other hand, the data reception processing unit 401 of the data reception server 202 receives the analysis data transmitted by the data source terminal 201, performs a predetermined confirmation as to whether the content is appropriate, and as a result of the confirmation, the content is If it is determined to be appropriate, metadata showing the characteristics of the analysis data is generated. The generation of this metadata shall be executed by an algorithm predetermined for each table item of the metadata storage unit 412, such as a process of reading the data size from the property information of the analysis data (the method is not limited). ).

For example, the metadata generated by the data receiving server 202 when the analysis data 102 is received corresponds to the metadata 103 in the data node 101 in FIG. The value of the data node ID in the table of the metadata storage unit 412 of FIG. 8A described above corresponds to the record of the data node 101.

Similarly, the metadata generated by the data receiving server 202 when the analysis data 102 is received corresponds to the metadata 113 in the data node 111 in FIG. The value of the data node ID in the table of the metadata storage unit 412 of FIG. 8A described above corresponds to the record of the data node 111.

Subsequently, the data acquisition / storage / deletion request unit 402 of the data reception server 202 transmits an information storage request to the data acquisition / storage / deletion reception unit 403 of the data storage server 203 (S802).

The information storage request transmitted by the data acquisition / storage / deletion request unit 402 of the data reception server 202 includes the analysis data received from the data source terminal 201 and the corresponding analysis generated by the data reception processing unit 401 of the data reception server 202. Includes data metadata.

On the other hand, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 analyzes the analysis data based on the information storage request transmitted by the data acquisition / storage / deletion request unit 402 of the data reception server 202 described above. It is stored in the storage unit 411 (S803).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 uses the metadata based on the information storage request transmitted by the data acquisition / storage / deletion request unit 402 of the data reception server 202 described above. It is stored in the storage unit 412 (S804).

On the other hand, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 transmits the storage result of the analysis data and the metadata to the data acquisition / storage / deletion request unit 402 of the data reception server 202 (S805).

Further, the data reception processing unit 401 of the data reception server 202 receives the reception of the above-mentioned analysis data and the storage result of the metadata, and transmits the reception result of the analysis data to the data source terminal 201 (S806). End the process.

In FIG. 11, the data analysis display client 207 of the present embodiment sends a display information acquisition request to the data analysis display server 206, and the data analysis server 206 stores data required according to the request contents in the data storage server 203. This is an example of an operation sequence in which the acquired data is generated in the display information and the display information is transmitted to the data analysis display client 207.
In this case, the data analysis display client 207 transmits a display information request to the display request reception unit 501 of the data analysis display server 206 (S901).

Here, the display information request transmitted by the data analysis display client 207 includes a type of display information and an input parameter of the display information. Further, as the type of display information, a data information display screen, a data analysis result display screen, an analysis program selection screen, a data analysis execution screen, and the like can be assumed as the user interface provided by the analyst.

Subsequently, the data acquisition / storage / deletion request unit 504 of the data analysis display server 206 receives data acquisition / storage / deletion reception unit of the data storage server 203 based on the display information request transmitted from the above-mentioned data analysis display client 207. An information acquisition request for analysis data, metadata, and generation data is transmitted to 403 (S902).

On the other hand, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires analysis data from the analysis data storage unit 411 in accordance with the above-mentioned information acquisition request (S903).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires metadata from the metadata storage unit 412 in accordance with the above-mentioned information acquisition request (S904).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires the generation data from the generation data storage unit 511 in accordance with the above-mentioned information acquisition request (S905).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 sends the analysis data, the metadata, and the generation data to the data acquisition / storage / deletion request unit 504 of the data analysis display server 206. Send (S906).

On the other hand, the data acquisition / storage / deletion request unit 504 of the data analysis display server 206 requests the data acquisition / storage / deletion reception unit 403 of the data storage server 203 based on the display information request transmitted from the data analysis display client 207. , Send an information acquisition request for analysis program information (S907).
Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires the analysis program information from the analysis program information storage unit 512 (S908).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 transmits the analysis program information to the data acquisition / storage / deletion request unit 504 of the data analysis display server 206 (S909).

Further, the data acquisition / storage / deletion request unit 504 of the data analysis display server 206 transmits an information update request to the data acquisition / storage / deletion reception unit 403 of the data storage server 203 (S910).

The above-mentioned information update request transmitted by the data acquisition / storage / deletion request unit 504 of the data analysis display server 206 includes a request to update the last access time column 1604 in the metadata of the data node to be displayed at the current time. .. For example, when the analyst executes the display request of the intermediate data 122 of FIG. 1 at the time “2016/10/11 10:03”, the data node ID column 1601 of the table of the metadata storage unit 412 illustrated in FIG. 8 In the record whose value is "data node 121", the update request is made with the value of the last access time column 1604 as "20161001-1003".

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 is based on the information update request transmitted by the data acquisition / storage / deletion request unit 504 of the data analysis display server 206, and the meta of the metadata storage unit 412. Update the data (S911).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 transmits the metadata update result to the data acquisition / storage / deletion request unit 504 of the data analysis display server 206 (S912).

Subsequently, the display information generation unit 502 of the data analysis display server 206 includes the display information request transmitted from the data analysis display client 207 and various information acquired from the data acquisition / storage / deletion reception unit 403 of the data storage server 203. Display information is generated based on (S913).

Further, the display request receiving unit 501 of the data analysis display server 206 transmits the display information generated by the display information generation unit 502 of the data analysis display server 206 to the data analysis display client 207 (S914), and ends the process.

In FIG. 12, the data analysis display client 207 of the present embodiment sends an analysis execution request to the data analysis server 204, and the data analysis server 204 acquires data required by the data analysis server 204 according to the request content from the data storage server 203. The data analysis server 204 executes the analysis program 602 with the acquired data as an input, the data storage server 203 stores the analysis execution result, and the data analysis display server 206 generates display information related to the analysis execution result. It is a figure which shows the example of the series of operation sequence in which the data analysis display server 206 transmits the display information to the data analysis display client 207. It should be noted that the same sequence is used when regenerating the deleted intermediate data.

In this case, the data analysis display client 207 transmits a data analysis execution request to the display request receiving unit 501 of the data analysis display server 206 (S1001).

Here, it is assumed that the data analysis execution request transmitted by the data analysis display client 207 includes parameters and the like input by the analyst via the user interface of the data analysis display client 207.

On the other hand, the display request receiving unit 501 of the data analysis display server 206 receives the data analysis execution request transmitted from the above-mentioned data analysis display client 207, and confirms the validity of this content. If the content is appropriate, the data analysis execution request unit 503 of the data analysis display server 206 transmits the data analysis execution request to the data analysis execution reception unit 505 of the data analysis server 204 (S1002).

On the other hand, the data acquisition / storage / deletion request unit 602 of the data analysis server 204 receives the above-mentioned data analysis execution request, and receives analysis data and metadata for the data acquisition / storage / deletion reception unit 403 of the data storage server 203. , And the information acquisition request of the generation data (S1003).
In this case, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires the analysis data from the analysis data storage unit 411 (S1004).
Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires metadata from the metadata storage unit 412 (S1005).
Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires the generation data from the generation data storage unit 511 (S1006).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 responds to the data acquisition / storage / deletion request unit 602 of the data analysis server 204 with the analysis data and metadata obtained in S1004 to S1006 described above. , And data for generation, (S1007).

If the data for generation does not exist, the analysis execution management unit 60 of the data analysis server 204
1 generates data for generation based on the content of the data analysis execution request transmitted from the data analysis execution request unit 503 of the data analysis display server 206.

For example, in the case of the intermediate data 122 in FIG. 1, the input is the analysis data 111 of the data node 101, the data processing 161 is "conversion 1", and the input parameters of the data processing are "P_11 = 8, P12". = 2 ”is used. In that case, the value of the generation data ID column 1609 in the table of the generation data storage unit 511 of FIG. 8B described above is the "generation data 01", and the value of the original data list column 1610 is the "data node 101". A record is generated in which the value of the algorithm column 1611 is "conversion 1" and the value of the forward data generation parameter column 1612 is "P_11 = 8, P12 = 2".

Subsequently, the analysis program 603 of the data analysis server 204 receives the data analysis execution request transmitted from the data analysis display client 207, and the analysis data and metadata acquired from the data acquisition / storage / deletion reception unit 403 of the data storage server 203. , And the data for generation, and the data analysis is performed (S1008). By this process, intermediate data will be generated.

Further, the data acquisition / storage / deletion request unit 602 of the data analysis server 204 transmits an information storage request and an information update request to the data acquisition / storage / deletion reception unit 403 of the data storage server 203 (S1009).

The above-mentioned information storage request transmitted by the data acquisition / storage / deletion request unit 602 of the data analysis server 204 includes the intermediate data or the analysis result which is the execution result of the data analysis execution process (S1008) and the data analysis server 204. It includes the corresponding intermediate data generated by the analysis execution management unit 601 or the data for generating the analysis result.

Further, the information update request transmitted by the data acquisition / storage / deletion request unit 602 of the data analysis server 204 includes the metadata generated by the analysis execution management unit 601 of the data analysis server 204. For example, the intermediate data 122 in FIG. 1 was generated at the time "2016/9/12 03:42" requiring the data generation time "33 [min]", and the data size was "20 [Gbyte]". In this case, in the record in which the value of the data node ID column 1601 of the table in the metadata storage unit 412 of FIG. 8 is "data node 121", the value of the data size column 1602 is set to "20" and the data generation time column. The update request is made with the value of 1605 set to "20160912-0342" and the value of the data generation time column 1606 set to "33".

Subsequently, the data acquisition / storage / deletion reception unit 402 of the data storage server 203 analyzes the intermediate data or the analysis result based on the information storage request transmitted by the data acquisition / storage / deletion request unit 602 of the data analysis server 204. It is stored in the data storage unit 411 (S1010).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 is based on the information update request transmitted by the data acquisition / storage / deletion request unit 602 of the data analysis server 204, and the metadata of the metadata storage unit 412. Is updated (S1011).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 sends the data acquisition / storage / deletion request unit 602 of the data analysis server 204 the storage result of the intermediate data or the analysis result and the data for generation thereof. The update result of the metadata is transmitted (S1013).
When there are a plurality of analysis targets, the above-mentioned processes from process S1003 to process S1013 are repeatedly executed by the number of target data.

Subsequently, the data analysis execution reception unit 505 of the data analysis server 204 transmits the data analysis result to the data analysis execution request unit 503 of the data analysis display server 206 (S1014).

Further, the display information generation unit 502 of the data analysis display server 206 has the data analysis execution request transmitted from the data analysis display client 207 and the data analysis result transmitted from the data analysis execution reception unit 505 of the data analysis server 204. Based on the above, display information is generated (S1015).

Further, the display information generation unit 502 of the data analysis display server 206 transmits the above-mentioned generated display information to the data analysis display client 207 (S1016), and ends the process.

In FIG. 13, the intermediate data deletion server 205 of the present embodiment acquires the intermediate data deletion policy from the data storage server 203, and the intermediate data deletion server 205 selects the intermediate data to be deleted based on the deletion policy. It is a figure which shows the example of a series of operation sequences in which the data storage server 203 deletes the intermediate data selected as the deletion target, and the data storage server 203 updates the metadata of the intermediate data.

In this case, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 transmits an information acquisition request of the intermediate data deletion policy to the data acquisition / storage / deletion reception unit 403 of the data storage server 203 ( S1101).
Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires the deletion policy from the deletion policy storage unit 711 (S1102).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 transmits the deletion policy obtained in S1102 to the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 (S1103).

Next, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 transmits an information acquisition request for metadata and generation data to the data acquisition / storage / deletion reception unit 403 of the data storage server 203. (S1104).

On the other hand, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 receives the above-mentioned information acquisition request and acquires metadata from the metadata storage unit 412 (S1105).
Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 acquires the generation data from the generation data storage unit 511 (S1106).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 requests the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 for the metadata and generation obtained in S1105 and S1106, respectively. Data is transmitted (S1107).

On the other hand, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 has the deletion policy obtained via the above-mentioned acquisition information processing (S1103), and the metadata and generation data obtained via the acquisition information processing (S1107). Based on the above, the intermediate data to be deleted is selected (S1108). Further, in this process, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 receives the deletion request and the metadata update request of the intermediate data selected above in the data acquisition / storage of the data storage server 203. -Send to the deletion reception unit 403.

The process of searching for the intermediate data to be deleted in S1108 is performed for each of the nodes including the analysis data, that is, the starting data node (data node 101 and data node 111 in the case of FIG. 1). Further, when selecting the intermediate data to be deleted in S1108 described above, the search is performed in the direction from the starting data node to the terminating data node (a node including the analysis result, and in the case of FIG. 1, the data node 151). Details of the processing content of S1108 will be described later with reference to FIGS. 14 and 15.

Subsequently, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 requests the data acquisition / storage / deletion reception unit 403 of the data storage server 203 to delete the intermediate data selected in S1108 and the metadata. An update request is transmitted (S1109).

In the above-mentioned deletion request transmitted by the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205, the intermediate data selected in the process (S1108) is designated as the deletion target. Further, in the update request transmitted by the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205, the metadata of the intermediate data selected in the process (S1108) is designated as the update target.

For example, when the intermediate data 122 of FIG. 1 is selected as the deletion target, the data is stored in the record in which the value of the data node ID column 1601 of the table in the metadata storage unit 412 of FIG. 8 is "data node 121". The update request is made with the value of the path column 1603 as "-". In the data storage path column 1603 of the table in the metadata storage unit 412, the value of "-" indicates that the corresponding data has been deleted.

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 analyzes the corresponding intermediate data based on the deletion request transmitted from the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205. It is deleted from the storage unit 411 (S1110).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 is based on the update request transmitted from the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205, and the metadata of the metadata storage unit 412. Is updated (S1111).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 sends the above-mentioned intermediate data deletion result and metadata update result to the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205. Is transmitted (S1112).

On the other hand, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 has the intermediate data deletion policy information obtained via the above-mentioned acquisition information processing (S1103), the metadata obtained via the acquisition information processing (S1107), and the metadata. The intermediate data to be deleted is selected based on the generated data (S1113). Further, in this S1113, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 makes a data acquisition / storage of the deletion request and the metadata update request of the intermediate data selected in S1113. -Send to the deletion reception unit 403.

The process of searching for the deletion target in S1113 is performed on all the terminal data nodes (data node 151 in the case of the example of FIG. 1). Further, when selecting the intermediate data to be deleted in S1113, unlike the case of S1108, the search is performed in the direction from the terminal data node to the start data node. Details of the processing contents of S1113 will be described later with reference to FIGS. 16 and 17.

Subsequently, the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 requests the data acquisition / storage / deletion reception unit 403 of the data storage server 203 to delete the corresponding intermediate data and update the metadata. Transmit (S1114).

In the deletion request transmitted by the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205 described above, the intermediate data selected in the process (S1113) is designated as the deletion target. Similarly, in the update request transmitted by the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205, the metadata of the intermediate data selected in the process (S1113) is designated as the update target.

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 receives the corresponding intermediate data based on the above-mentioned deletion request transmitted from the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205. Is deleted from the analysis data storage unit 411 (S1115).

Further, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 is based on the above-mentioned update request transmitted from the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205, and the corresponding intermediate data is obtained. The metadata is updated in the metadata storage unit 412 (S1116).

Subsequently, the data acquisition / storage / deletion reception unit 403 of the data storage server 203 sends the above-mentioned intermediate data deletion result and metadata update result to the data acquisition / storage / deletion request unit 703 of the intermediate data deletion server 205. Is transmitted (S1117), and the process is terminated.

In FIG. 14, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 in the present embodiment searches with the designated starting data node as a starting point to select the intermediate data to be deleted, and the intermediate data deletion server 205 selects the intermediate data. This is an example of a series of operation flows in which the data acquisition / storage / deletion request unit 705 transmits a deletion request for intermediate data selected as described above and an update request for its metadata to the data storage server 203.

The flow shown in FIG. 14 is a process executed by the intermediate data deletion management unit 702 of the intermediate data deletion server 205 as a processing subject. Here, the starting point data node that is the starting point in the search for the deletion target of the intermediate data is set to "n".
In this case, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes "1" for the loop counter i mounted on the memory 302 or the like (S1201).

The intermediate data deletion management unit 702 of the intermediate data deletion server 205 repeatedly executes the process (S1203) to the process (S1212) until the condition of the process (S1204) described later is satisfied (S1202).

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 requests the data storage server 203 to acquire a list of data nodes in the i-th hierarchy of the loop counter when viewed from the starting data node "n". It is transmitted to the data acquisition / storage / deletion reception unit 403 to acquire a list of the corresponding data nodes (S1203). Here, the list of acquired data nodes is "l".

For example, in FIG. 1, when the origin data node "n" is "data node 101" and the loop counter i is "1", the list of acquired data nodes is the data node of the first layer when viewed from "data node 101". It becomes "data node 121". Similarly, when the origin data node "n" is "data node 101" and the loop counter i is "2", the list of acquired data nodes is "data node of the second layer from the viewpoint of" data node 101 "". Data node 141 ". When the origin data node "n" is "data node 101 and the loop counter i is" 3 ", the list of acquired data nodes is" data node 151 "which is the third layer data node when viewed from" data node 101 ". It becomes. When the origin data node "n" is "data node 101" and the loop counter i is "4", the list of acquired data nodes is the data node of the fourth layer from the viewpoint of "data node 101", but the corresponding data. The data list is empty because there are no nodes.
Next, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data node list “l” is empty (S1204).

As a result of the above determination, when the data node list "l" is not empty (S1204: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1205). On the other hand, as a result of the above determination, when the data node list "l" is empty (S1204: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 ends the process.

As described above, for example, in FIG. 1, when the starting data node "n" is "data node 101" and the loop counter i is "5", the list of acquired data nodes is the fifth layer when viewed from "data node 101". However, since there is no corresponding data node, the data list becomes empty and this process ends.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 performs a processing (processing (S1211) from S1206) for each data node of the data list "l" acquired in the above processing (S1203) (loop processing (S1211). S1205) is executed.

Here, the target data node in the loop processing is set to "j". Further, when the processing (S1211) is completed from the processing (S1206) for all the data nodes in the data list "l", the process proceeds to the processing (S1212).

Here, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the calculation result T1 (j) of the processing function for calculating the data generation time of the data node “j” into the time t (S1206). Here, the processing function for calculating the data generation time of the data node "n" is "T1 (n)".

The processing function "T1 (n)" for calculating the data information generation time of the data node "n" is a function for calculating the time required to regenerate the data node "n" from the starting data node. Details of this process (S1206) will be described later with reference to FIG.

Subsequently, in the intermediate data deletion management unit 702 of the intermediate data deletion server 205, is the time t obtained in the above process (S1206) equal to or less than the reproduction time upper limit of the deletion policy acquired in the process (S1103) of FIG. A determination is made (S1207).

As a result of the above determination, when the time t is equal to or less than the reproduction time upper limit of the deletion policy (S1207: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1208). On the other hand, as a result of the above determination, when the time t is larger than the reproduction time upper limit value of the deletion policy (S1207: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1205).

By this process (S1207), a data node whose intermediate data reproduction time is longer than a predetermined time is excluded from the deletion target of the intermediate data because the data reproduction takes too much time and reduces the convenience of the analyst. Is possible.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data size value of the metadata of the data node "j" is equal to or greater than the data size lower limit value of the deletion policy acquired in the process (S1103). Do (S1208).

As a result of the above determination, when the data size value is equal to or greater than the data size lower limit value (S1208: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1209).

On the other hand, as a result of the above determination, when the data size value is smaller than the data size lower limit value (S1208: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1205).

By this process (S1208), a data node whose data size of the intermediate data is smaller than a predetermined size is excluded from the deletion target of the intermediate data based on the policy that the deletion efficiency is low even if the intermediate data is deleted. It is possible to do.

Next, in the intermediate data deletion management unit 702 of the intermediate data deletion server 205, the value of the last access time column 1604 to the data in the metadata of the data node "j" is the final access of the deletion policy acquired in the process (S1103). It is determined whether or not the time upper limit value is exceeded (S1209).

As a result of the above determination, when the final access time to the data is earlier than the value obtained by subtracting the final access time upper limit value from the current time (S1209: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines. Proceed to processing (S1210).

On the other hand, as a result of the above determination, when the final access time to the data is after the value obtained by subtracting the final access time upper limit value from the current time (S1209: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 Proceeds to the process (S1205).

By this processing (S1209), the intermediate data is based on the policy that the analyst is likely to refer to the data node whose last access time to the intermediate data has not passed a predetermined time or more. It is possible to exclude it from the deletion target.

Subsequently, in the intermediate data deletion management unit 702 of the intermediate data deletion server 205, the value of the data generation time column 1605 in the metadata of the data node "j" is equal to or greater than the lower limit of the generation elapsed time of the deletion policy acquired in the process (S1103). Is determined (S1210).

As a result of the above determination, when the data generation time value is earlier than the value obtained by subtracting the generation elapsed time lower limit value from the current time (S1210: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 processes ( Proceed to S1211).

On the other hand, as a result of the above determination, when the data generation time value is after the value obtained by subtracting the generation elapsed time lower limit value from the current time (S1210: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines. Proceed to the process (S1205).

Based on the policy that there is a high possibility that the analyst will use the data node for which the data generation time of the intermediate data has not elapsed for a predetermined time or more by this process (S1210), and the intermediate data that has just been generated. It is possible to exclude the intermediate data from the deletion target.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 stores the data deletion request for the intermediate data of the data node "j" and the metadata update request for the data node "j" selected so far. It is transmitted to the data acquisition / storage / deletion reception unit 403 of the server 203 (S1211). This process makes it possible to reduce the used capacity of the data storage. Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 adds "1" to the loop counter "i" (S1212).

Here, a specific example of the operation flow of FIG. 14 will be described using the data analysis route conceptual diagram of FIG. 1, the table of the metadata storage unit 412 of FIG. 8A, and the table of the deletion policy storage unit 711 of FIG. The deletion policy applied here is the record of "policy 01" in the policy ID column 1701 in the table of the deletion policy storage unit 711. Further, the time when the process of FIG. 14 is executed is set to "2017/3/1 00:00".

In this case, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 has a list "data node 121" including the "data node 121" from the process (S1204) when the loop counter "i" is "1" in the "starting point data node 101". l ”is acquired.

Further, in the process (S1206), the data information generation time of the "intermediate data 122" of the "data node 121" is calculated as "33 [min]". The data information generation time "33 [min]" of the "intermediate data 122" is from the value "60 [min]" of the regeneration time upper limit column 1705 of the deletion policy (hereinafter referred to as the application policy) assumed to be applied above. Since it is small, it proceeds to the process (S1208) by the determination of the process (S1207).

Further, since the value "80 [Gbyte]" of the data size column 1602 of the "data node 121" is larger than the value "1 [Gbyte]" of the data size lower limit column 1702 of the above-mentioned application policy, the process (S1208). In the determination of, the process proceeds to the process (S1209).

Further, the value "20161001-1003" of the last access time column 1604 to the data of "data node 121" is compared with the current time, and the value of the last access time lower limit column 1703 of the application policy is "24 * 30 *". Since 2 [hour] or more has passed, the process (S1210) is determined by the determination of the process (S1209).

Further, the value "20160912-0342" of the data generation time column 1605 of the "data node 121" is compared with the current time, and the value "24 * 30 [hour]" of the generation elapsed time lower limit column 1704 of the application policy is compared. Since the above has passed, the process (S1211) is determined to proceed to the process (S1211), and the "intermediate data 122" of the "data node 121" is deleted by the process (S1211).
Next, when the loop counter "i" is "2" in the "starting point data node 101", the list "l" including the "data node 141" is acquired from the process (S1204).
In the process (S1206), the data information generation time of the "intermediate data 142" of the "data node 141" is calculated as "33 + 90 = 113 [min]".

Since the data information generation time "113 [min]" of the "intermediate data 142" is larger than the value "60 [min]" of the regeneration time upper limit column 1705 of the application policy, it is determined in the process (S1207). Proceeding to the process (S1212), the "intermediate data 142" does not correspond to the application policy and is excluded from the deletion target. Subsequent data nodes are also excluded from the deletion target because they do not fall under the application policy due to the same processing.

Therefore, in this example, the intermediate data information of "80 [Gbyte]" is deleted according to the predetermined deletion policy for the total data size "80 + 110 + 90 = 280 [Gbyte]" of the intermediate data node by the processing of FIG. It is possible to reduce the area for storing data by about 29%.

FIG. 15 is an example of an operation flow in which the intermediate data deletion management unit 702 of the intermediate data deletion server 205 calculates the generation time of the corresponding intermediate data from a forward viewpoint. The process of FIG. 15 is a process executed by the intermediate data deletion management unit 702 of the intermediate data deletion server 205 as the processing subject.

In this case, first, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 sends an information acquisition request for acquiring the original data list of the data node "n" to the data acquisition / storage / deletion reception unit 403 of the data storage server 203. Send and acquire the list of the corresponding data nodes (S1301).

Here, the list of acquired data nodes is "l". For example, in FIG. 1, when the data node "n" is the "data node 121", the acquired original data list includes the "data node 101". Similarly, when the data node "n" is the "data node 141", the original data list to be acquired includes the "data node 121" and the "data node 131". When the data node "n" is the "data node 151", the original data list to be acquired includes the "data node 141".

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 performs a loop process (S1306) from the process (S1303) to each data node of the data list (l) acquired in the above process (S1301). (S1302) is executed.

Here, the target data node in the loop processing is set to "k". When the process (S1306) is completed from the process (S1303) for all the data nodes in the data list "l", the process proceeds to the process (S1307).

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether or not the data node "k" is the starting data node (S1303). As a result of this determination, if the data node "k" is not the starting data node (S1303: Yes), the process proceeds to the process (S1304). On the other hand, as a result of the above determination, when the data node "k" is the starting data node (S1303: No), the process proceeds to the process (S1302).

In the determination of the origin data node, the value of the table generation data ID column 1608 in the metadata storage unit 412 of FIG. 8A is used. If there is no value in the table generation data ID column 1608 in the metadata storage unit 412, the corresponding data node is the starting data node. On the other hand, when there is a value of the data ID column 1608 for table generation in the metadata storage unit 412, the corresponding data node is not the starting data node.

For example, "data node 101" in FIG. 1 indicates that it is a starting data node, and in the table of the metadata storage unit 412 of FIG. 8A, the generation data ID column 1608 of "data node 101" has no value. Similarly, “data node 102” in FIG. 1 indicates a starting data node, and in the table of the metadata storage unit 412 of FIG. 8A, the generation data ID column 1608 of “data node 102” has no value.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether or not the intermediate data exists in the data node “k” (S1304). As a result of this determination, if intermediate data exists in the data node "k" (S1304: Yes), the process proceeds to the process (S1305). On the other hand, when there is no intermediate data in the data node "k" (S1304: No),
Process (S1302) proceeds.

The value of the data storage path column 1303 of the table in the metadata storage unit 412 is used for determining the existence of the intermediate data. If the data storage path column 1303 of the table in the metadata storage unit 412 has a value, the intermediate data of the corresponding data node exists, and if the data storage path column 1303 of the table in the metadata storage unit 412 has no value, the data node of the corresponding data node exists. There is no intermediate data.

For example, "data node 122" in FIG. 1 indicates that there is no intermediate data, and in FIG. 8A, the data storage path column 1303 of "data node 122" has no value. Similarly, "data node 142" in FIG. 1 indicates that there is no intermediate data, and in FIG. 8A, the data storage path column 1303 of "data node 142" has no value.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 adds the data generation time of the metadata of the data node “k” to the time t (S1305).

Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 adds the calculation result "T1 (k)" of the processing function for calculating the data information generation time of the data node "k" to the time t (S1306). ..

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 adds the data generation time of the metadata of the data node “n” to the time t (S1307).
Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the value of the time t calculated up to this point into the data generation time Tr (S1308).

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 returns the data generation time Tr obtained in the above calculation to the caller of the function (S1309), and ends the process.

In FIG. 16, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 of the present embodiment performs a search with the designated termination data node as a starting point, selects the intermediate data to be deleted, and selects the intermediate data. It is a figure which shows the example of a series of operation flow which sends the data deletion request and the metadata update request to the data storage server 203. The process of FIG. 16 is a process executed by the intermediate data deletion management unit 702 of the intermediate data deletion server 205 as the processing subject. Here, the terminal data node that is the starting point in the search for the deletion target of the intermediate data is set to "n".
In this case, first, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes 1 for the loop counter i (S1401).

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 executes a loop process in which the process (S1403) to the process (S14112) are repeated until the condition of the process (S1404) described later is satisfied (S1402). ..

Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 sends an information acquisition request for acquiring a list of data nodes in the i-th layer of the loop counter when viewed from the terminal data node "n" to the data of the data storage server 203. It is transmitted to the acquisition / storage / deletion reception unit 403 to acquire the list of the corresponding data nodes (S1403). Here, the list of acquired data nodes is "l".

For example, in FIG. 1, when the terminal data node "n" is "data node 151" and the loop counter i is "1", the list of acquired data nodes is the data node of the first layer when viewed from "data node 151". It becomes "data node 141". Similarly, when the terminal data node "n" is "data node 151" and the loop counter i is "2", the list of acquired data nodes is "data node of the second layer from the viewpoint of" data node 151 "". Includes "data node 121" and "data node 131". When the terminal data node "n" is "data node 151" and the loop counter i is "3", the list of acquired data nodes is "data node 101" which is the third layer data node when viewed from "data node 151". "And" data node 111 ". When the terminal data node "n" is "data node 151" and the loop counter i is "4", the list of acquired data nodes is the data node of the fourth layer from the viewpoint of "data node 151", but the corresponding data. The data list is empty because there are no nodes.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data node list “l” is empty (S1404). As a result of this determination, when the data node list "l" is not empty (S1404: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1405.

On the other hand, as a result of the above determination, when the data node list "l" is empty (S1404: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 ends the process.

As described above, for example, in FIG. 1, when the terminal data node "n" is "data node 151" and the loop counter i is "5", the list of acquired data nodes is the fifth layer when viewed from "data node 151". However, since there is no corresponding data node, the data list becomes empty and this process ends.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 performs a loop process (S1411) from the process (S1406) to each data node of the data list "l" acquired in the above process (S1403). (S1405) is executed. Here, the target data node in the loop processing is set to "j".
When the process (S1411) is completed from the process (S1406) for all the data nodes in the data list "l", the process proceeds to the process (S1412).

Next, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the calculation result T2 (j) of the processing function for calculating the data information generation time of the data node “j” into the time t (S1406).

Here, the processing function for calculating the data information generation time of the data node "n" is "T2 (n)". The processing function "T2" for calculating the data information generation time of the data node "n" is a function for calculating the estimated time required when the data node "n" is regenerated from the terminal data node. Details of such processing (S1406) will be described later with reference to FIG.

Next, in the intermediate data deletion management unit 702 of the intermediate data deletion server 205, the time t obtained in the above process (S1406) is equal to or less than the lower limit of the data reproduction time in the deletion policy acquired in the process (S1103) of FIG. (S1407).

As a result of the above determination, when the time t is equal to or less than the lower limit of the data reproduction time of the deletion policy (S1407: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1408).

On the other hand, as a result of the above determination, when the time t is larger than the data reproduction time lower limit value of the deletion policy (S1407: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1405).

By this process (S1407), the data node whose intermediate data reproduction time is longer than a predetermined time is removed from the intermediate data deletion target under the policy of reducing the convenience of the analyst because the data reproduction takes too much time. Allows exclusion.

Next, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data size value of the metadata of the data node "j" is equal to or greater than the data size lower limit value in the deletion policy acquired in the process (S1103). (S1408).

As a result of the above determination, when the data size value is equal to or greater than the data size lower limit value (S1408: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1409).

On the other hand, as a result of the above determination, when the data size value is smaller than the data size lower limit value (S1408: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process (S1405).

By this process (S1408), data nodes whose intermediate data data size is smaller than a predetermined size are excluded from the intermediate data deletion target based on the policy that the deletion efficiency is low even if the intermediate data is deleted. It is possible to do.

Subsequently, in the intermediate data deletion management unit 702 of the intermediate data deletion server 205, the final access time value in the metadata of the data node "j" is the final access time upper limit value to the data in the deletion policy acquired in the process (S1103). It is determined whether the time has passed (S1409).

As a result of the above determination, when the final access time value is earlier than the value obtained by subtracting the final access time upper limit value to the data from the current time (S1409: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 , Proceed to process (S1410).

On the other hand, as a result of the above determination, when the final access time value to the data is after the value obtained by subtracting the final access time upper limit value to the data from the current time (S1409: No), the intermediate data of the intermediate data deletion server 205 The deletion management unit 702 proceeds to the process (S1405).

By this process (S1409), data nodes whose last access time to the intermediate data has not passed a predetermined time or more are likely to be referred again by the analyst, and are excluded from the deletion target of the intermediate data. It is possible to do.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data generation time value of the metadata of the data node "j" is equal to or greater than the lower limit of the generation elapsed time in the deletion policy acquired in the process (S1103). A determination is made (S1410).

As a result of the above determination, when the data generation time value is earlier than the value obtained by subtracting the generation elapsed time lower limit from the current time (S1410: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 processes (S1411). ).

On the other hand, as a result of the above determination, when the data generation time value is after the value obtained by subtracting the generation elapsed time lower limit from the current time (S1410: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 processes. Proceed to (S1405).

Based on the policy that there is a high possibility that the analyst will use the data node that has not passed the data generation time of the intermediate data for a predetermined time or more by this process (S1410), and the intermediate data that has just been generated. It is possible to exclude the intermediate data from the deletion target.

Next, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 receives the data deletion request of the data node "j" and the metadata update request of the data node "j" from the data storage server 203. -Send to the storage / deletion reception unit 403 (S1411). This process makes it possible to reduce the used capacity of the data storage. Further, "1" is added to the loop counter "i" (S1412).

Here, a specific example of the operation flow of FIG. 16 will be described using the data analysis route conceptual diagram of FIG. 1, the table in the metadata storage unit 412 of FIG. 8A, and the table of the deletion policy storage unit 711 of FIG. The deletion policy applied here (hereinafter referred to as the application policy) is set as the record of "policy 01" in the policy ID column 1701 of the table in the deletion policy storage unit 711, and the time when the process of FIG. 16 is executed is "2017/3 /". It will be "13:00".
In this case, when the loop counter i is "1" in the "terminating data node 151", the list "l" including the "data node 141" is acquired from the process (S1403).

In the process (S1406), the generation time of the "intermediate data 142" of the "data node 141" is calculated as "77 * 0.25≈19 [min]". Since the generation time "" 19 [min] "of the" intermediate data 142 "is smaller than the value" 60 [min] "of the regeneration time upper limit column 1705 of the application policy, the processing (S1408) is determined by the processing (S1407). ).

Further, since the value "90 [Gbyte]" of the data size column 1602 of the "data node 141" is larger than the value "1 [Gbyte]" of the data size lower limit column 1702 of the application policy, the determination of the process (S1408). Proceed to the process (S1409).

Further, the value "20161206-1608" of the last access time column 1604 of "data node 141" is compared with the current time, and the value of the last access time lower limit column 1703 of the application policy is "24 * 30 * 2 [hour]". Since the above has passed, the process (S1410) proceeds according to the determination of the process (S1409).

Further, the value "20160913-0759" of the data generation time column 1605 of the "data node 141" is compared with the current time, and the value "24 * 30 [hour]" of the generation elapsed time lower limit column 1704 of the application policy is compared. Since the above has passed, the process (S1411) is determined by the determination of the process (S1410), and the process (S1411) deletes and executes the "" intermediate data 142 "of the" data node 141 ".

Next, when the loop counter i is "2" in the "terminating data node 151", the list "l" including the "data node 121" and the "data node 131" is acquired from the process (S1403). In the process (S1406), the data information generation time of the "intermediate data 122" of the "data node 121" is calculated as "-1 [min]".

Since the data information generation time "-1 [min]" of the "intermediate data 122" is smaller than the value "60 [min]" of the regeneration time upper limit column 1705 of the application policy, it is determined in the process (S1407). Proceed to processing (S1405). In addition, "intermediate data 122" does not correspond to the application policy and is excluded from the deletion target. Subsequent data nodes are also excluded from the deletion target because they do not fall under the application policy due to the same processing.

Therefore, in this example, by the processing of FIGS. 14 and 16 described above, the intermediate data of "80 + 90 = 110 [Gbyte]" is deleted from the predetermined deletion policy for the total data size "80 + 110 + 90 = 280 [Gbyte]" of the intermediate data node. It is possible to reduce the area for storing intermediate data by about 61%.

The process of searching for the intermediate data to be deleted from the terminal data node of FIG. 16 makes it possible to further reduce the area for storing the intermediate data as compared with the process of FIG.

FIG. 17 is an example of an operation flow in which the intermediate data deletion management unit 702 of the intermediate data deletion server 205 of the present embodiment calculates the estimated regeneration time of the corresponding intermediate data from a reverse viewpoint. The process of FIG. 17 is a process executed by the intermediate data deletion management unit 702 of the intermediate data deletion server 205 as the processing subject.
In this case, first, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the initial value "-1" into the data generation time Tr (S1501).

Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 acquires, stores, and deletes data of the data storage server 203, which acquires a list of data nodes whose original data node is the data node "n". It is transmitted to the reception unit 403 and the list of the corresponding data nodes is acquired (S1502). Here, the list of acquired data nodes is "l".

For example, in FIG. 1, when the data node "n" is the "data node 141", the "data node 151" having the "data node 141" as the original data node corresponds to the data node, so that the acquired data list is the "data node". 151 "is included.

Similarly, when the data node "n" is the "data node 121", the "data node 141" whose original data node is the "data node 121" is applicable, so that the acquired data list is "data node 141". include. When the data node "n" is the "data node 101", the acquired data list includes the "data node 121" because the "data node 121" whose original data node is the "data node 101" corresponds to the data node "n".

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 executes a loop process (S1503) in which the process S1512 is performed from the process S1504 for each data node of the data list “l” acquired in the above process S1502. .. Here, the target data node in the loop processing is set to "k". When the process S1512 is completed from the process S1504 for all the data nodes in the data list "l", the process proceeds to the process S1513.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data inverse conversion generation estimation time of the data node “k” is negative (S1504).

As a result of the above determination, when the data inverse conversion generation estimation time of the data node "k" is not negative (S1504: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1505.

On the other hand, as a result of the above determination, when the data inverse conversion generation estimation time of the data node "k" is negative (S1504: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1503.

As the value of the data inverse conversion generation estimated time of the data node, the value of the data inverse transformation generation estimated time column 1607 of the table in the metadata storage unit 412 of FIG. 8A is used. When the value of the data inverse conversion generation estimation time column 1607 of the table in the metadata storage unit 412 is negative, it means that the corresponding intermediate data cannot be inversely converted to the original intermediate data, that is, regenerated once it is lost. show. On the other hand, when the value of the data inverse conversion generation estimation time column 1607 of the table in the metadata storage unit 412 is not negative, it indicates that the corresponding intermediate data can be inversely converted to the original data information, that is, can be regenerated.

For example, in FIG. 8A, the estimated data inverse conversion generation generation time of the “data node 151” is “77 * 0.25 [hour]” and is not a negative value, so that “analysis result 152” of the “data node 151” is “analyzed”. It is shown that the reverse conversion to the "intermediate data 142" of the "data node 141" which is the original data information of the "result 152" is possible.

On the other hand, in FIG. 8A, the data inverse conversion generation water time of the “data node 141” is “-1”, which is a negative value. Therefore, from the “intermediate data 142” of the “data node 141” to the “intermediate data 142”. It shows that the reverse conversion of the original data information "data node 121" to "intermediate data 122" and "data node 131" to "intermediate data 132" is not possible.

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 adds the data inverse conversion generation estimated time of the metadata of the data node “k” to the time t (S1505).
Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data node "k" is a terminal data node (S1506).

As a result of the above determination, when the data node "k" is not the terminal data node (S1506: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1507.

On the other hand, as a result of the above determination, when the data node "k" is the terminal data node (S1506: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1512.

For the determination of the terminal data node described above, the value of the original data list column 1610 of the table in the generation data storage unit 511 of FIG. 8B is used. If the data node is not registered as the original data node in the original data list column 1610 of the table of the generation data storage unit 511, the corresponding data node is the terminal data node. On the other hand, when the data node is registered as the original data node in the original data list column 1610 of the table in the generation data storage unit 511, the corresponding data node is not the terminal data node.

For example, "data node 151" in FIG. 1 indicates that it is a terminal data node, and "data node 151" is not registered as an original data node in the table in the generation data storage unit 511 of FIG. 8B.
Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether or not the intermediate data exists in the data node “k” (S1507).

As a result of this determination, when the intermediate data does not exist in the data node "k" (S1507: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1508.

On the other hand, as a result of the above determination, when intermediate data exists in the data node "k" (S150).
7: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1512.

Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the calculation result "T2 (k)" of the processing function for calculating the data information generation time of the data node "k" into the time t'(S1508). ).
Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the time t'is a value larger than 0 (S1509).

As a result of the above determination, when the time t'is a value larger than 0 (S1509: Yes), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1510.

On the other hand, when the time t'is a value of 0 or less as a result of the above determination (S1509: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1503.
Next, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 adds the time t and the time t'to the data generation time candidate Tr'(S1510).
Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the time t into the data generation time candidate Tr'(S1511).

Subsequently, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 determines whether the data generation time Tr is a value smaller than 0 or the data generation time Tr is a value larger than the data generation time candidate Tr'(S1512). ).

As a result of the above determination, when the data generation time Tr is smaller than 0 or the data generation time Tr is larger than the data generation time candidate Tr'(S1512: Yes), the intermediate data deletion management of the intermediate data deletion server 205 Section 702 proceeds to process S1513.
As a result of the above determination, in other cases (S1512: No), the intermediate data deletion management unit 702 of the intermediate data deletion server 205 proceeds to the process S1503.
Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 substitutes the value of the data generation time candidate Tr'into the data generation time Tr (S1513).

Further, the intermediate data deletion management unit 702 of the intermediate data deletion server 205 returns the data generation time Tr obtained by the calculation up to this point to the caller of the function (S1514), and ends the process.

For the restoration, that is, the regeneration of the intermediate data, for example, any one of the data storage server 203, the data analysis server 204, the intermediate data deletion server 205, and the data analysis display server 206 can be restored as the data analysis display client 207. With respect to the intermediate data specified from, the intermediate data or analysis data, which is the original data used for the generation, is specified by the original data list column 1610 of the generation data storage unit 511, and the corresponding processing algorithm is used for this original data. It can be executed by applying the algorithm indicated by the value in column 1611 together with the parameter indicated by the value in forward data generation parameter column 1612 or reverse data generation parameter column 1613.

Although the best mode for carrying out the present invention has been specifically described above, the present invention is not limited to this, and various modifications can be made without departing from the gist thereof.

According to the present embodiment, it is possible to reduce the use of the data storage by effectively deleting the intermediate data, and to regenerate the intermediate data once deleted.

When deleting the intermediate data, the intermediate data to be deleted is appropriately selected from the viewpoint of whether the influence on the analysis process is good or bad. Therefore, the above-mentioned improvement in the utilization efficiency of the data storage and the improvement in the efficiency of the analysis process can be achieved at the same time. It will be. This leads to an increase in the number of analysis trials that can be performed per unit time as compared with the conventional method, and it can be expected that the analysis in a short time will lead to the good analysis result desired by the analyst. In addition, each of the above-mentioned effects leads to more efficient analysis resources, which may lead to cost reduction in the introduction and operation of such an analysis system.
That is, the intermediate data can be effectively managed, and the efficiency of the analysis process using the intermediate data can be improved.

The description herein reveals at least the following: That is, in the intermediate data management system of the present embodiment, the storage device further stores the deletion policy of the intermediate data, and the arithmetic unit applies the metadata and the generation data to the deletion policy. Therefore, it may be assumed that the intermediate data to be deleted is selected and the deletion process of the intermediate data is executed.
According to this, it is possible to efficiently and accurately select the intermediate data to be deleted according to the above-mentioned deletion policy.

Further, in the intermediate data management system of the present embodiment, the storage device has at least one of the data size, the last access time, the elapsed time from generation, and the time required for data restoration as the deletion policy of the intermediate data. The calculation device applies the metadata and the generation data to the deletion policy, and selects the intermediate data that matches the specified information as the intermediate data to be deleted. However, the process of deleting the intermediate data may be executed.

According to this, as the intermediate data to be deleted, the data size is less than the specified, the last access time is correspondingly old and the usage frequency is low, the elapsed time from generation is correspondingly long and old, and the time suitable for data restoration. It is possible to accurately and efficiently select intermediate data that can be recognized as having a low significance of existence.

Further, in the intermediate data management system of the present embodiment, the arithmetic unit further executes a restoration process of the intermediate data based on the analysis data, the metadata, and the generation data of the predetermined intermediate data. It may be something to do.

According to this, even if the intermediate data is once deleted, it can be restored when it is needed later, and it is possible to perform an analysis process using the intermediate data again.

Further, in the intermediate data management method of the present embodiment, the information processing system further stores the deletion policy of the intermediate data in the storage device, and applies the metadata and the generation data to the deletion policy. Then, the intermediate data to be deleted may be selected and the deletion process of the intermediate data may be executed.

Further, in the intermediate data management method of the present embodiment, the information processing system is required for the data size, the last access time, the elapsed time from generation, and the data restoration as the deletion policy of the intermediate data in the storage device. The specified information regarding at least one item of time is stored, the metadata and the generated data are applied to the deletion policy, and the intermediate data matching the specified information is selected as the intermediate data to be deleted. Then, the deletion process of the intermediate data may be executed.

Further, in the intermediate data management method of the present embodiment, the information processing system further restores the intermediate data based on the analysis data, the metadata, and the generation data regarding the predetermined intermediate data. You may do it.

101, 111, 121, 131, 141, 151 Data nodes 102, 112 Analysis data 103, 113, 123, 133, 143, 153 Meta data / generation data 122, 132, 142 Intermediate data 152 Analysis results 161 to 164 Data processing 171 to 175, 181 and 182 Related 200 Intermediate data management system 201 Data source terminal 202 Data receiving server 203 Data storage server 204 Data analysis server 205 Intermediate data deletion server 206 Data analysis display server 207 Data analysis display client 208, 209 Network 301 CPU
302 Memory 303 Auxiliary storage 304 Communication I / F
305 Input I / F
306 Keyboard 307 Mouse 308 Output I / F
309 Display 401 Data reception processing unit 402 Data acquisition / storage / deletion request unit 403 Data acquisition / storage / deletion reception unit 411 Analysis data storage unit 412 Metadata storage unit 501 Display request reception unit 502 Display information generation unit 503 Data analysis execution request Unit 504 Data acquisition / storage / deletion request unit 505 Data analysis execution reception unit 511 Generation data storage unit 512 Analysis program information storage unit 601 Analysis execution management unit 602 Analysis program 603 Data acquisition / storage / deletion request unit 701 Intermediate data deletion periodical Request unit 702 Intermediate data deletion management unit 703 Data acquisition / storage / deletion request unit 711 Intermediate data deletion policy storage unit

Claims (8)

A storage device for storing analysis data, metadata showing various attributes of the analysis data, and data for generating generated data generated based on other analysis data among the analysis data.
Based on the metadata and the generation data , a sequence of data from the start point to the end point in a series of data processing flows starting from the analysis data and ending from the analysis result of the analysis data, and the above. A computing device that selects predetermined intermediate data from the generated data by a predetermined algorithm and executes deletion processing of the intermediate data in each of a series of data that traces from the end point to the start point .
An intermediate data management system characterized by being equipped with. The storage device is
Further stores the deletion policy of the intermediate data,
The arithmetic unit is
The metadata and the generation data are applied to the deletion policy, intermediate data to be deleted is selected, and the deletion process of the intermediate data is executed.
The intermediate data management system according to claim 1. The storage device is
As the intermediate data deletion policy, the specified information regarding at least one of the data size, the last access time, the elapsed time from the generation, and the time required for data restoration is stored.
The arithmetic unit is
The metadata and the generation data are applied to the deletion policy, intermediate data matching the specified information is selected as intermediate data to be deleted, and the deletion process of the intermediate data is executed.
The intermediate data management system according to claim 2. The arithmetic unit is
The restoration process of the intermediate data is further executed based on the analysis data, the metadata, and the generation data of the predetermined intermediate data.
The intermediate data management system according to claim 1. An information processing system including a storage device for storing analysis data, metadata showing various attributes of the analysis data, and data for generating generated data generated based on other analysis data among the analysis data.
Based on the metadata and the generation data , a sequence of data from the start point to the end point in a series of data processing flows starting from the analysis data and ending from the analysis result of the analysis data, and the above. In each of the series of data tracing from the end point to the start point, a predetermined intermediate data is selected from the generated data by a predetermined algorithm, and the deletion process of the intermediate data is executed.
An intermediate data management method characterized by that. The information processing system
In the storage device, the deletion policy of the intermediate data is further stored.
The metadata and the generation data are applied to the deletion policy, intermediate data to be deleted is selected, and the deletion process of the intermediate data is executed.
The intermediate data management method according to claim 5, characterized in that. The information processing system
In the storage device, as the deletion policy of the intermediate data, the specified information regarding at least one of the items of data size, last access time, elapsed time from generation, and time required for data restoration is stored.
The metadata and the generation data are applied to the deletion policy, intermediate data matching the specified information is selected as intermediate data to be deleted, and the deletion process of the intermediate data is executed.
The intermediate data management method according to claim 6, characterized in that. The information processing system
Further executing the restoration process of the intermediate data based on the analysis data, the metadata, and the generation data of the predetermined intermediate data.
The intermediate data management method according to claim 5, characterized in that.

Images