JP2015014847A - Design assist system, design assist method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate application of a parallel distributed processing technique by assisting, in a system design taking advantage of the parallel distributed processing technique, the design of a case in which an especially large amount of data is handled.SOLUTION: In a data extraction/processing design assist system 100, a mode evaluation subsystem 300 extracts a feature amount from inputted processing mode data and evaluates a data processing mode from the feature amount and processing mode metadata. A sample execution subsystem 400 extracts a candidate as distributed processing from the processing mode data, and predicts a processing time for data to be processed from a sample data processing time which is a processing time based on the processing mode data for sample data obtained by adding the evaluation result of a mode evaluation unit to the extraction result. A data management subsystem 500 generates a processing mode feature amount on the basis of the feature amount extracted by the mode evaluation unit, and outputs the processing mode feature amount and the processing time for the data to be processed as predicted by the sample execution subsystem 400.

Description

  The present invention relates to a design support system, a design support method, and a program, and more particularly, to a design support technology when utilizing a parallel distributed processing technology.

  In recent years, data processing for large amounts of data that could not be processed in a realistic time has become possible with parallel distributed processing technology. Data processing by this parallel distributed processing technique is very effective in terms of speeding up processing for a large amount of data, and is applied to statistical processing, mining processing, or large-scale batch processing. In application, it is implemented after judging whether parallel distributed processing is possible.

  In this type of parallel distributed processing technology, for example, parallel distributed processing (for example, refer to Patent Document 1) or COBOL (COmmon Business Oriented Language) batch processing is performed in parallel for series data that is not good at parallel distributed processing. A device that can be distributed and speeded up (for example, see Non-Patent Document 1) is known.

JP 2011-150503 A

Hadoop blind spot, parallelization of COBOL batch processing, (URL: http://www.atmarkit.co.jp/fjava/rensai4/trouble_knowhow08/01.html)

  The parallel distributed processing technology is very effective in that a large amount of data can be handled at high speed. However, in implementing data extraction / processing such as data analysis, it is technically difficult to appropriately design how to distribute the processing and in what procedure.

  This is because there are cases where there are restrictions on conditions that can be specified for the data extraction process, and there are cases where output data becomes too large and I / O (Input / Output) performance deteriorates.

  Although there are special methods for solving such problems and services for consulting data processing, there are cases where trials themselves are difficult due to the "mass data processing".

  As described above, Non-Patent Document 1 describes that existing batch processing can be distributed in parallel, and Patent Document 1 describes a technique for performing parallel distribution. Judgment as to whether or not the method can be applied to data to be handled in the future is not clear, and application requires an engineer who is familiar with these advanced methods.

  For example, from the viewpoint of performance verification, it is necessary to develop a prototype or the like in order to confirm the design and to experiment with a large amount of data. If an appropriate design is not made, the processing time naturally becomes longer, the number of trials decreases, and the quality deteriorates relatively.

  This may become a bottleneck in providing value when providing data processing services such as analysis to customers, and it is necessary to apply consulting and advanced engineers for each verification.

  An object of the present invention is to provide a technology that can facilitate the application of a parallel distributed processing technique by supporting a design of a case that handles a large amount of data in a system design that utilizes a parallel distributed processing technique. It is in.

  The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

  That is, a typical outline is applied to a design support system and has the following characteristics.

  The design support system is not an advanced engineer who is proficient in parallel distributed algorithms in addition to data processing technology by supporting the design of cases that handle large amounts of data in system design that utilizes parallel distributed processing technology. However, this facilitates the application of the parallel distributed processing technology.

  This design support system includes a method evaluation unit, a sample execution processing unit, and a data management unit. The method evaluation unit extracts a feature amount from the input processing method data, and evaluates a data processing method from the extracted feature amount and processing method metadata.

  The sample execution processing unit needed to extract a candidate for distributed processing as sample data from the processing method data, and to execute processing based on the processing method data on the sample data to which the evaluation result by the method evaluation unit was added. The processing time of the processing target data is predicted from the sample data processing time which is time.

  The data management unit generates a processing method feature value based on the feature value extracted by the method evaluation unit, and outputs the processing method feature value and the processing time of the processing target data predicted by the sample execution processing unit.

  The present invention is also applicable to a method by a system that provides appropriate parallel distributed processing selection candidates and a selection processing method verification method using various types of data as input, and a program that causes a computer system to function as the system. be able to.

  Among the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

  Parallel distributed processing design can be made highly efficient.

It is explanatory drawing which shows an example of a structure of the data extraction and process design assistance system in one Embodiment. It is a flowchart which shows an example of a process of the system evaluation subsystem which the data extraction and process design assistance system of FIG. 1 has. It is explanatory drawing which showed an example of the data format of the processing method feature-value stored in the process target data group which the data extraction and process design support system of FIG. 1 has. It is description which showed an example of the data format of the data feature-value stored in a process target data group. It is explanatory drawing which shows an example of the data format in the processing system feature-value extracted by the format feature extraction part which the data extraction and process design support system of FIG. 1 has. It is a flowchart which shows an example of the process of the candidate process extraction part which the sample execution subsystem of FIG. 1 has. It is explanatory drawing which shows an example of the data format by the sample data process of a candidate process extraction part. It is a flowchart which shows an example of the process of the sample execution part which a sample execution subsystem has. It is explanatory drawing which shows an example of the display in the display part which a design control subsystem has.

  In the following embodiments, when it is necessary for the sake of convenience, the description will be divided into a plurality of sections or embodiments. However, unless otherwise specified, they are not irrelevant to each other. There are some or all of the modifications, details, supplementary explanations, and the like.

  Further, in the following embodiments, when referring to the number of elements (including the number, numerical value, quantity, range, etc.), especially when clearly indicated and when clearly limited to a specific number in principle, etc. Except, it is not limited to the specific number, and may be more or less than the specific number.

  Further, in the following embodiments, the constituent elements (including element steps and the like) are not necessarily indispensable unless otherwise specified and apparently essential in principle. Needless to say.

  Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., the shape of the component is substantially the case unless it is clearly specified and the case where it is clearly not apparent in principle. And the like are included. The same applies to the above numerical values and ranges.

  In all the drawings for explaining the embodiments, the same members are denoted by the same reference symbols in principle, and the repeated explanation thereof is omitted. In order to make the drawings easy to understand, even a plan view may be hatched.

  Hereinafter, the embodiment will be described in detail based on the above-described outline.

<Outline of data extraction / processing design support system>
FIG. 1 is an explanatory diagram illustrating an example of a configuration of a data extraction / processing design support system 100 according to the present embodiment.

  The data extraction / processing design support system 100 is a design support system configured by one or more computers having a calculation capability such as a central processing unit and a network, and processes a large amount of series data in parallel.

  As shown in FIG. 1, the parallel distributed processing data extraction / processing design support system 100 includes a design control subsystem 200, a method evaluation subsystem 300, a sample execution subsystem 400, a data management subsystem 500, a processing target data group 601, And a processing method metadata group 602.

  The design control subsystem 200 is connected to the method evaluation subsystem 300 and the sample execution subsystem 400, respectively. The method evaluation subsystem 300, which is a method evaluation unit, is connected to the sample execution subsystem 400 serving as a sample execution processing unit. Has been.

  The data management subsystem 500, which is a data management unit, is connected to the method evaluation subsystem 300, the sample execution subsystem 400, the processing target data group 601, and the processing method metadata group 602, respectively.

  The design control subsystem 200 includes an input unit 201 such as a keyboard and a display unit 202 such as a display, and various requests including processing method definitions and processing target data obtained from the user via the input unit 201. Are used to transmit various requests to the system evaluation subsystem 300 and the sample execution subsystem 400, and the results are displayed on the display unit 202.

  The method evaluation subsystem 300 includes a data evaluation unit 301 and a processing method feature amount extraction unit 302. The method evaluation subsystem 300 extracts the feature amount of the processing method data including the processing method definition and the processing target data input from the input unit 201 of the design control subsystem 200, and the processing method from the data management subsystem 500. Evaluate the data processing method along with the metadata. The processing method definition may be only the input / output definition of processing, or may include a progress log of processing such as specifying the processing flow.

  The sample execution subsystem 400 includes a sample execution unit 401 and a candidate process extraction unit 402. In addition to the evaluation of the processing executed by the method evaluation subsystem 300 on the processing method definition and processing target data input from the input unit 201, the sample execution subsystem 400 receives sample data for execution from the processing target data. Extraction is performed, and processing based on the processing method definition is executed using the sample data. The executed trial result is stored through the data management subsystem 500.

  The data management subsystem 500 includes a format feature extraction unit 501 and a data browsing unit 502. The data management subsystem 500 extracts features based on the data format in response to a data storage request from the method evaluation subsystem 300 or the sample execution subsystem 400, and the results, metadata, etc. are processed as a database. It is a subsystem that stores data in the data group 601 and processing method metadata group 602 and sends stored data in response to various requests.

  The processing functions of the method evaluation subsystem 300, the sample execution subsystem 400, and the data management subsystem 500 described below are, for example, a program storage memory (not shown) provided in the data extraction / processing design support system 100. ) And the like in a program format stored by the CPU (Central Processing Unit) (not shown) of the data extraction / processing design support system 100 is executed.

<Example of system evaluation subsystem processing>
Next, processing of the system evaluation subsystem 300 will be described using FIG.

  FIG. 2 is a flowchart showing an example of processing of the method evaluation subsystem 300 included in the data extraction / processing design support system 100 of FIG.

  Here, the processes of steps S101, S102, S106, and S107 in FIG. 2 are processes executed by the data evaluation unit 301, and the processes of steps S103, S104, and S105 in FIG. It is a process to be executed.

  First, among the processing method definitions input via the input unit 201, the processing log is analyzed, and the processing name, processing time, and detailed history of data use are extracted (step S101). Subsequently, the input / output definition for the process is acquired from the processing method definition input via the input unit 201 and the detailed history of data use that is the output of the process of step S101 (step S102). From the input / output definition, the type and set of input / output data, the order of use, the operation type between input values, and the processing time are extracted.

  At this time, if there is a similar processing method, the similar processing method metadata already stored in the processing method metadata group 602 is acquired through the data browsing unit 502 of the data management subsystem 500 (step S103). ). If there is no similar processing method, the process of step S106 is performed.

  When similar processing method metadata is acquired in the processing of step S103, the feature amount is extracted by matching with the analysis result of the processing method definition extracted in the processing of steps S101 and S102 (step S104). At this time, the matching of the feature amounts may be an average of numerical values or a combination of character strings.

  Thereafter, the calculated processing method feature quantity is stored in the processing target data group 601 through the data management subsystem 500 (step S105), and an input / output management unit and an operation unit are extracted from the input / output definition (step S106). The management unit may be a name acquired from a file name or may be an application determined from an extension. Subsequently, the calculated data feature amount is stored together with the processing target data through the data management subsystem 500 (step S107).

<Data format example of processing method feature value>
FIG. 3 is an explanatory diagram showing an example of the data format of the processing method feature amount stored in the processing target data group 601 included in the data extraction / processing design support system 100 of FIG. As described above, a format having “method ID”, “detail ID”, “processing name”, “input type”, “output type”, “trial history”, “processing time / data amount”, and “group operation” It has become.

<Data format example of data feature value>
FIG. 4 is an explanation showing an example of the data format of the data feature quantity stored in the processing target data group 601. As shown in the figure, “processing ID”, “argument ID”, “management unit”, “shared” The format has “unit”, “operation unit”, and “data”.

  When storing data feature amounts, feature amounts such as data type and value distribution are automatically calculated as processing method feature amounts through the format feature extraction unit 501 of the data management subsystem 500 and stored in the processing target data group 601. . At this time, the calculated feature value may be a data type such as a character string type or an integer type, or may be a total or maximum statistical value such as a maximum value, a minimum value, or a distribution format.

<Data format example of extracted processing method features>
FIG. 5 is an explanatory diagram showing an example of a data format in the processing method feature amount extracted by the format feature extraction unit 501 included in the data extraction / processing design support system 100 of FIG. 1. As shown in FIG. , “Attribute”, “attribute type”, and “number of records”.

<Example of candidate process extraction unit>
Next, the process of the candidate process extraction unit 402 will be described with reference to FIG.

  FIG. 6 is a flowchart showing an example of processing of the candidate process extraction unit 402 included in the sample execution subsystem 400 of FIG.

  The candidate process extraction unit 402 executes a process of calculating candidates as a distributed process based on the processing target data input to the input unit 201, and is output to the display unit 202 as selectable input items.

  First, when processing target data is input from the input unit 201, the candidate process extraction unit 402 acquires a processing method feature quantity group in the format shown in FIG. 3 from the data management subsystem 500 (step S201).

  Subsequently, a data evaluation request is made to the method evaluation subsystem 300, and as a result, a data feature amount is obtained in the format shown in FIG. 4 (step S202). Further, a format evaluation request is made to the data management subsystem 500, and as a result, data feature amounts are obtained in the format shown in FIG.

  A set of feasible values is extracted from the processing method feature amount group obtained in the process of step S201 and the data feature amount obtained in the process of step S202 (step S203), and data of each value included in the extracted set is extracted. The minimum number of samples is acquired from the feature quantity.

  Using the extracted sample, the sample execution unit 401 is requested to execute processing on the sample data for each processing method (step S204). When the processing of the sample execution unit 401 is completed, the log at the time of execution of the process is analyzed, the processing time, the date, etc. are extracted and held as data in the format shown in FIG. 7 (step S205).

  It is determined whether or not the sample execution time exceeds a threshold value that is a preset time (step S206), and processing is performed when the sample execution time exceeds the threshold value or when a predetermined number of times have been executed. finish. If the sample execution time does not exceed the threshold value, the process returns to step S203 to request the sample execution by increasing the number of samples again.

<Sample execution unit processing example>
Next, processing of the sample execution unit 401 will be described.

  FIG. 8 is a flowchart illustrating an example of processing of the sample execution unit 401 included in the sample execution subsystem 400.

  The sample execution unit 401 is requested to execute from the input unit 201 or starts processing in response to a request from the candidate process extraction unit 402. The sample execution unit 401 starts with input of processing target sample data to be executed and a processing method definition.

  First, the sample execution unit 401 makes a format feature extraction request to the data management subsystem 500, and acquires the data format feature shown in FIG. 5 as a result (step S301). Then, the method evaluation subsystem 300 is requested to extract the feature quantity of the processing target sample data, and as a result, the data feature quantity shown in FIG. 4 is acquired (step S302).

  When the data feature amount is acquired, processing based on the processing method definition is executed with the processing target sample data as input (step S303), and execution history information is output (step S304).

<Display examples on the display>
FIG. 9 is an explanatory diagram illustrating an example of display on the display unit 202 included in the design control subsystem 200.

  On the display unit 202, a data feature display window 2022 is displayed at the upper left of FIG. 9, and a processing method display window 2023 is displayed at the lower left of the data feature display window 2022.

  A processing input method display window 2024 is displayed on the right side of the processing method display window 2023. An execution button 2025 is displayed below the process input method display window 2024.

  An output sample display window 2026 is displayed on the right side of the data feature display window 2022, and a processing time estimation display window 2027 is displayed below the output sample display window 2026.

  The data feature display window 2022 displays data features and data format features stored in the format shown in FIGS. 4 and 5 in a list format. The processing method display window 2023 and the processing input method display window 2024 display a processing method list saved in the format shown in FIG. 3 and an input set (tuple) for the processing method. Only pairs that can be combined with the data format feature amount are specified.

  The processing method definition selected in the processing method display window 2023 or the processing input method display window 2024 is executed by the sample execution unit 401 when an execution button 2025 is pressed. The execution results in the sample execution unit 401 are displayed in an output sample display window 2026 and a processing time estimation display window 2027, respectively.

  The output sample display window 2026 displays the output of the selected processing method. The display format may be text or graph. The processing time estimation display window 2027 displays data in the form of sample execution monitoring results shown in FIG. 7 obtained as execution results, and displays approximate curves. Estimating the processing time for a large amount of data using the processing time for a small amount of sample helps to determine feasibility.

  According to the above, if the parallel distributed processing method definition is accumulated in advance by an engineer, analysis method extraction and performance evaluation can be performed in parallel distributed processing technology design, even in cases where large amounts of data are handled. Implementation can be facilitated.

  In addition, since the provision of screens enables conceptual understanding of the effects of distributed parallel processing, it is possible to concentrate on effective use verification of distributed technology and data analysis technology without direct instruction from advanced analysts. .

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  In addition, this invention is not limited to above-described embodiment, Various modifications are included. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to one having all the configurations described.

  Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. . In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

100 Data Extraction / Processing Design Support System 200 Design Control Subsystem 201 Input Unit 202 Display Unit 2022 Data Feature Display Window 2023 Processing Method Display Window 2024 Processing Input Method Display Window 2025 Execution Button 2026 Output Sample Display Window 2027 Processing Time Estimation Display Window 300 Method Evaluation Subsystem 301 Data Evaluation Unit 302 Processing Method Feature Extraction Unit 400 Sample Execution Subsystem 401 Sample Execution Unit 402 Candidate Processing Extraction Unit 500 Data Management Subsystem 501 Format Feature Extraction Unit 502 Data Browsing Unit 601 Processing Target Data Group 602 Processing Method metadata group

Claims (12)

A method evaluation unit that extracts feature values from the input processing method data and evaluates a data processing method from the extracted feature values and processing method metadata;
The time required when processing based on the processing method data is performed on the sample data obtained by extracting candidates for distributed processing as sample data from the processing method data and adding the evaluation result by the method evaluation unit. A sample execution processing unit for predicting the processing time of processing target data from a certain sample data processing time;
A data management unit that generates a processing method feature value based on the feature value extracted by the method evaluation unit, and outputs a processing time of the processing target data predicted by the processing method feature value and the sample execution processing unit;
Design support system with The design support system according to claim 1,
The design support system further includes a database that stores the processing method feature amount output by the data management unit and a processing time of the processing target data predicted by the sample execution processing unit. The design support system according to claim 1,
The processing method data has a processing method definition that defines input / output of data processing,
The system evaluation unit is a design support system that extracts the feature amount by comparing the processing method definition and processing method metadata similar to the processing method definition. The design support system according to claim 1,
The sample execution processing unit generates an approximate curve based on the sample data processing time, and predicts the processing time of the processing target data from the approximate curve. A method evaluation unit that evaluates a data processing method, a sample execution processing unit that predicts a processing time of processing target data, a processing method feature amount, and a processing time of the processing target data predicted by the sample execution processing unit are output. A design support method by a computer system comprising a data management unit,
Extracting a feature amount from the input processing method data in the method evaluation unit;
Evaluating a processing method based on the extracted feature quantity and processing method metadata;
In the sample execution processing unit, candidates for distributed processing are extracted as sample data from the processing method data, and processing based on the processing method data is executed on the sample data to which the evaluation result by the method evaluation unit is added. Calculating sample data processing time, which is the time required for
Predicting the processing time of the data to be processed from the calculated sample data processing time;
In the data management unit, generating a processing method feature amount based on the feature amount extracted by the method evaluation unit;
Outputting the processing method feature and the processing time of the processing target data predicted by the sample execution processing unit;
A design support method. The design support method according to claim 5,
The design support method further includes a step of storing the processing method feature amount output by the data management unit and a processing time of the processing target data predicted by the sample execution processing unit in a database. The design support method according to claim 5,
The processing method data has a processing method definition that defines input / output of data processing,
The step of extracting the feature amount is a design support method for extracting the feature amount by comparing the processing method definition and processing method metadata similar to the processing method definition. The design support method according to claim 5,
The step of predicting the processing time of the processing target data is a design in which the sample execution processing unit generates an approximate curve based on the sample data processing time and predicts the processing time of the processing target data from the approximate curve. Support method. A method evaluation unit that evaluates a data processing method, a sample execution processing unit that predicts a processing time of processing target data, a processing method feature amount, and a processing time of the processing target data predicted by the sample execution processing unit are output. A program to be executed by a computer system comprising a data management unit,
Extracting a feature amount from the input processing method data in the method evaluation unit;
Evaluating a processing method based on the extracted feature quantity and processing method metadata;
In the sample execution processing unit, candidates for distributed processing are extracted as sample data from the processing method data, and processing based on the processing method data is executed on the sample data to which the evaluation result by the method evaluation unit is added. Calculating sample data processing time, which is the time required for
Predicting the processing time of the data to be processed from the calculated sample data processing time;
In the data management unit, generating a processing method feature amount based on the feature amount extracted by the method evaluation unit;
Outputting the processing method feature and the processing time of the processing target data predicted by the sample execution processing unit;
A program with The program according to claim 9, wherein
The program further includes a step in which the data management unit stores in the database the processing method feature amount output by the data management unit and the processing time of the processing target data predicted by the sample execution processing unit. The program according to claim 9, wherein
The step of extracting the feature amount extracts the feature amount by comparing a processing method definition that defines input / output of data processing included in the processing method data and processing method metadata similar to the processing method definition, program. The program according to claim 9, wherein
The step of predicting the processing time of the processing target data is a program in which the sample execution processing unit generates an approximate curve based on the sample data processing time and predicts the processing time of the processing target data from the approximate curve. .

Images