WO2007072566A1

WO2007072566A1 - Metadata retriever

Info

Publication number: WO2007072566A1
Application number: PCT/JP2005/023649
Authority: WO
Inventors: Hitoshi Uehara; Hideharu Sasaki; Yoshikazu Sasai
Original assignee: Japan Agency For Marine-Earth Science And Technology
Priority date: 2005-12-22
Filing date: 2005-12-22
Publication date: 2007-06-28
Also published as: JP4905989B2; JPWO2007072566A1

Abstract

A metadata retriever comprises an acceptor for accepting a designation of a file identifier containing both information on the storage location of a data file and a keyword for metadata retrieval, a metadata storage for storing metadata corresponding to the data file, an extractor for extracting the keyword from the designated file identifier, and a retriever for retrieving metadata corresponding to the extracted keyword from the metadata storage.

Description

Specification

Metadata search device

Technical field

[0001] The present invention relates to a technology for processing a large amount of data files such as large-scale simulation data in the computer science field.

Background art

Simulation results performed in a large-scale simulation system such as a marine simulation system consist of a large amount of data files. It is rare for a large number of data files to be several tens of terabytes or more in size and 10,000 or more in number of files. In general, such a large amount of data files are not created in the same format, and often have a slightly different format depending on the content of the data.

In addition, in numerical simulation, generally, the intersections of line segments as shown in FIG. 16A are defined as calculation grids, and processing (for example, calculation of physical quantities) on numerical data is advanced based on this calculation grid. Be Depending on the physical characteristics of the numerical data and the circumstances of the calculation formula, processing (calculation of physical quantities) for some numerical data is based on different calculation grids as shown in FIG. 16B. Is often done.

Here, a first data file calculated based on the calculation grid (referred to as “first calculation grid”) shown in FIG. 16A, and a calculation grid shown in FIG. 16B (“No. Assume that there is a second data file calculated based on (2) “Calculation grid”. Furthermore, it is assumed that, for example, data in an area (cutout range) from 120.1 ° to 121.6 ° of the east longitude is cut out from the first and second data files.

In this case, data corresponding to two calculation grids are cut out from the first data file, and data corresponding to four calculation grids are cut out from the second data file. However, if it is desired that the first data file and the like also cut out the data corresponding to the four calculation grids, the data (cutout data) cut out according to the above-described cut out range will be insufficient.

[0006] The first data file as described above is calculated based on what calculation grid Data indicating the details of the data in the data file, whether it contains data or not, is referred to herein as "metadata".

[0007] Conventionally, metadata was stored separately from simulation data, or a user designated and input corresponding metadata for each simulation data file to be processed.

Disclosure of the invention

Problem that invention tries to solve

[0008] When simulation data generated as a result of large-scale simulation is to be further processed after simulation, user power S metadata is designated and input for each of those large data files. That will force the user to spend a lot of effort. In addition, there was also a possibility that user power S metadata was specified erroneously

An object of the present invention is to provide a technology that eliminates the need for a user to specify metadata for processing target data.

Means to solve the problem

[0010] The present invention adopts the following means in order to achieve the above object.

That is, according to the present invention, receiving means for receiving specification of a file identifier including storage location information of a data file and a keyword for metadata search;

Metadata storage means storing metadata representing details of the data file, extraction means for extracting the keyword from the specified file identifier, and metadata corresponding to the extracted keyword are searched from the metadata storage means Search means to output

Is a metadata search device including

Preferably, in the present invention, each data file is stored on a directory structure, and the storage location information indicates a file path.

Preferably, in the present invention, the metadata storage means is provided on a storage area different from the storage area of the data file to which the file identifier is assigned.

The present invention also provides a metadata search having the same features as the above-described metadata search device. It can be specified as a search method, a program, or a recording medium on which the program is recorded. Effect of the invention

According to the present invention, it is not necessary for the user to specify metadata for processing target data.

Brief description of the drawings

FIG. 1 is a view showing a configuration example of a simulation system to which the present invention can be applied.

FIG. 2 is a view showing an example of the configuration of a control computer shown in FIG. 1;

[FIG. 3] A diagram showing an example of the configuration of the node shown in FIG.

4 is a view showing an example of the directory structure of a file database storing the processing object data file shown in FIG.

FIG. 5 is a view showing an example of the data structure of the metadata table shown in FIG. 2;

6 is a view showing an example of the data structure of a usage and load distribution situation table shown in FIG. 2;

[FIG. 7] A diagram showing a display example of a user interface (designated screen) provided to a user of the system.

FIG. 8 is a diagram showing an example of description of a file of parallel processing designation information input using a user interface.

[FIG. 9] A flowchart showing a main routine of creation processing of a parallel processing job script and a parallel processing program setting file.

FIG. 10 is a flowchart showing a main routine of processing for creating a parallel processing job script and a parallel processing program setting file.

FIG. 11 is a flowchart showing a main routine of processing for creating a parallel processing job script and a parallel processing program setting file.

[FIG. 12] A flowchart showing a subroutine related to acquisition of analysis of metadata.

FIG. 13 is a flowchart showing a subroutine of search and determination processing of a node to be a placement destination of a processing target data file.

FIG. 14 is a diagram showing an example of description of a parallel processing program setting file.

FIG. 15 is a flowchart showing execution processing of a parallel processing program. FIG. 16A is a diagram showing an example of a calculation grid prepared as metadata for processing target data.

[FIG. 16B] A diagram showing an example of a calculation grid different from the calculation grid of FIG. 16A, which is prepared as metadata for processing target data.

Explanation of sign

X ...... Parallel computer group

Υ · · · Computer for control

1, 11 --- CPU

2, 12 · "main memory

3, 14 · · · External storage device

7 · "Input device

8 ··· Display

6, 15 ··· Communication interface

31 · · · file database

32 · · · Metadata table

33 · · 'Use and load balancing information table

BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The configuration in the embodiment is an example, and the present invention is not limited to the configuration of the embodiment.

[Simulation system]

FIG. 1 is a view showing a configuration example of a simulation system to which the present invention can be applied. Figure

In the example shown in 1, the simulation system comprises a parallel computer group X and a control computer (information processing apparatus) Y connected to the parallel computer group X via a communication line (network).

[0020] The parallel computer group X is made up of a plurality of nodes # 0 to #n (n is a natural number) performing parallel processing on a large number of data files that constitute large-scale simulation data such as ocean general circulation model. .

Computer Y is a simulation data to be processed by parallel computer group X (processing It manages the target data), and performs control in the case of making the parallel computer group X execute parallel processing using simulation data according to the user's operation.

A user of the simulation system performs parallel processing to execute parallel processing of a large amount of processing target data (processing target data group) using parallel computer group X through a UI (user interface) provided by computer Y. Enter the specified information.

Here, the parallel processing specification information includes a plurality of simulation data files (processing target data file group) to be subjected to parallel processing, and processing contents of parallel computer group Y for the processing target data file group (processing type, It can include specification of processing detail parameters), multiple nodes (number of nodes) performing parallel processing, and storage location of file (processed data file (processing result file)) generated as a result of parallel processing.

Computer Y is a parallel processing job script for giving control instructions related to parallel processing to parallel computer group X based on the input parallel processing specification information (control program for parallel computer group Y: It may be written as “script” and a parallel processing program setting file (hereinafter referred to as “setting file”) that is referred to when each node that executes parallel processing processes the data file to be processed. And automatically generate.

[0025] Computer Y generates a script during the process of generating a script, acquires metadata (detailed information of process target data) for each process target data file, and processes a process target data file group for parallel computer group X. Make placement decisions. Metadata and placement decision results will be reflected in the description content of the script.

Computer Y distributes (distributes) processing target data files to a plurality of nodes through script execution, and instructs these nodes to execute a parallel processing program (job). . Each node executes the parallel processing program according to the description of the setting file, and performs processing on the distributed processing target data file based on the corresponding metadata. A processing result file is created through the processing. The processing result file is stored in the storage location specified as parallel processing specification information.

FIG. 2 is a diagram showing an example of the configuration of a computer. In FIG. 2, the computer Υ is connected to the CPU 1, the main memory (MM: for example RAM) 2, the external memory through the bus Β. Memory (eg hard disk) 3, input / output interface (I / F) 4 and 5, and communication interface 6 are provided.

The I / F 4 is connected to an input device (keyboard, pointing device (for example, a mouse), etc.) as an input means, and a display device (display) 8 as an output means is connected to the IZF 5. It is done. Furthermore, the communication IZF 6 is connected to each node # 0 to #n via a communication line (network).

The external storage device 3 includes a file database (file DB) 31 storing a large amount of simulation data files constituting large-scale simulation data, and metadata (simulation data corresponding to each data file). Metadata table 32 storing detailed information and usage and load distribution situation table 33 (hereinafter referred to as “situation table 33”) of each node to be referred to when distributing the processing target data file group to a plurality of nodes. Notation) is stored. File DB 31 and metadata table 32 are created on different storage areas.

Further, a program for causing the external storage device 3 to function the computer Y as a management device of simulation data and metadata and also to function as a control device of the parallel computer group Y (nodes # 0 to #n). Is stored.

The CPU 1 implements the following function, for example, by loading a program recorded in the external storage device 3 into the MM 2 and executing the program.

(1) Provide input (specified) environment (UI: user interface) of parallel processing specification information using the input device 7 and the display device 8 to the user of the simulation system.

(2) Create a script and a setting file based on parallel processing specification information.

(3) Before creation of a script, search and acquire metadata corresponding to each of a plurality of simulation data files (processing target data file group) designated by the user as the processing target.

(4) At the time of creation of the script, determine the nodes (arrangement of the processing target data file) to process each processing target data file constituting the processing target data file group.

(5) Transfer control of processing result files generated by parallel processing of the processing target data file group and the processing target data file group. The CPU 1 corresponds to a receiving unit, an extracting unit, and a searching unit according to the present invention, and the external storage device 3 corresponds to a metadata storage unit (storage unit) according to the present invention.

The nodes # 0 to #n constituting the parallel computer group X have the same configuration. FIG. 3 is a diagram showing an example of the configuration of a node. The node includes a CPU 11, a main memory 12, a calculation processor 13, an external storage device (for example, a hard disk) 14, and a communication interface (communication IZF) 15 mutually connected via a bus B 1. Communication IZF 15 is connected to computer Y and other nodes via a network.

The node receives the processing target data file transferred from the computer Y by the communication I / F 15 and stores it in the external storage device 14. The node also receives parallel processing instructions and configuration files from computer Y via communication I / F 15.

Then, the CPU 11 starts the execution of the parallel processing program stored in advance in the external storage device 3 in accordance with the description of the setting file. The calculation processor 13 is used for calculation using data to be processed. The calculation processor 13 reads the processing target data file stored in the external storage device 14 onto the MM 12 and executes predetermined processing (for example, cutting out a predetermined area in the data file, calculation of physical quantity) using this. This predetermined process is performed based on the metadata.

By a predetermined process, a process result file is generated and stored in the external storage device 14. The processing result file stored in the external storage device 14 is moved (transferred) to a predetermined storage position.

The CPU 11 corresponds to the receiving means, the extracting means and the searching means according to the present invention, and the external storage device 14 corresponds to the metadata storing means according to the present invention.

[0038] Data structure of DB and table>

Next, details of the file DB 31 and the metadata table 32 and the usage and load status table (status table) 33 shown in FIG. 2 will be described.

<< File DB 31 >>

The file DB 31 classifies and stores a large number of simulation data files (hereinafter sometimes referred to simply as “data file”) using a directory structure. FIG. 4 is a diagram showing an example of the directory structure of the file DB 31. As shown in FIG. In the file DB 31, a directory tree starting from the root directory (the directory "data" in FIG. 4) is formed, and directories of each hierarchy are given predetermined directory names. Data files are stored in a directory located at the end of the directory tree, and given a specified data file name.

Data files are identified using file identifiers. The file identifier is represented by a list of data file names and the names (path names) of each directory located on the path of the directory tree from the root directory to the end directory.

For example, the file identifier of the data file having the data file name “timeXXX.0000.000.dat” in FIG. 4 is 7 data / experiment A / 3 D / statistics A / variable B / time XXX. 000.000. Thus, the file The identifier contains storage location information (file path) of the data file.

In addition, the directory name ("3D", "statsticsA", "variableB", etc.) and the data file name ("timeXXX.0000.000") in the file identifier indicate the details (properties, etc.) of the data in the data file. It is specified as a keyword to show. Keywords consist of any one or more characters, and are placed in at least one place in directory names and data file names. However, no keyword is set in the extension part of the file name. A keyword functions as a search key for searching metadata corresponding to processing target data.

The data files may be distributed and stored in a plurality of storage areas arranged inside or outside the computer Y which need not necessarily be stored in one storage area.

<< Metadata Table >>

The metadata table 32 stores metadata corresponding to the keywords in the file identifier. FIG. 5 is a view showing an example of the data structure of the metadata table 32. As shown in FIG.

In the example shown in FIG. 5, the metadata table 32 is composed of a plurality of records storing a search key (keyword) and metadata corresponding thereto. The keyword is used as a search key from the file identifier of the data file (data file to be processed) specified by the user. It is extracted.

Metadata is information indicating details (properties and attributes, etc.) of simulation data (data to be processed), for example, information indicating physical properties of data to be processed, statistical processing, and time-space (vertical , Height, and time (date and time)). For example, the information on the calculation grids shown in FIGS. 16A and 16B is information on space. As a keyword representing such information of the calculation grid, for example, a variable name represented by an arbitrary number of characters is applied.

Note that FIG. 5 shows a case where one force of the directory name included in the file identifier corresponds to one metadata. Alternatively, for example, one metadata may be retrieved from a combination of a plurality of keywords included in one file identifier. Also, some of directory names and data file names (except for the extension) include a key code, and keywords may be extracted from file identifiers in partial match search. Also, it is possible to adopt a configuration in which keywords are set only for the file path part in the file identifier.

<< Status Table 33 >>

FIG. 6 is a view showing an example of the data structure of the situation table 33. As shown in FIG. The status table 33 is composed of a plurality of small tables 34 prepared for each node. Each small table 34 has the same data structure. The small table 34 contains identification information (user ID) of the user authorized to use the node, the maximum size of the external storage device (hard disk) of the node usable by the user, and the external currently used by the user. It consists of a set of records whose elements (items) are the storage capacity (load). Each small table 34 is assigned a node identifier, and information corresponding to the node identifier is stored in the small table.

In the computer Y shown in FIG. 2, the CPU 1 provides the user of the computer Y with an input environment (UI) of parallel processing specification information through execution of a program.

The user uses the UI to process the processing target data file group (file identifier), the plurality of nodes for processing the processing target data file group, and the processing target data file group, which are elements (items) of parallel processing specification information. Processing content (processing type and detailed parameters), processing result The storage location of the file can be specified.

FIG. 7 is a diagram showing an example of a designation screen of parallel processing designation information provided as a UI.

The designated screen is displayed on the screen of the display device 8 through the execution of the program by the CPU 1.

In the example shown in FIG. 7, the designation screen includes a file path display field 81, a file list display field 82, and a command input field 83. The file path display field 81 displays the directory (file path) in the file DB 31 selected by the user using the input device 7.

In the file list display field 82, a list of data files (data files stored in the end directory in the file path) corresponding to the file path displayed in the file path display field 81 (file list Display). Also, the command input field 83 is used to input a command related to the process for the data file to be processed.

The user can operate the input device 7 to display a desired file path in the file path display field 81 (select a file path). The display contents of the file list display field 82 are changed according to the selection result of the file path, and the file list corresponding to the file path is displayed in the display field 82.

The user designates the file identifier of the data file to be processed by designating the desired file name from the file list displayed in the file list display field 82 by the cursor operation using the input device 7. Can. At this time, it is possible to specify multiple data files at one time through cursor operation. Thus, the user can designate the file identifier of the data file to be processed using the file path display field 81 and the file list display field 82.

In addition, the user can use the command input field 83 to designate and input nodes (number of nodes) used for parallel processing, processing contents for processing target data file group, storage location of processing result file, etc. .

When specifying the number of nodes, processing parameters, and storage location, options for the specified content are displayed on the screen, and these are specified by the user selecting the desired option by cursor operation. It can be configured as follows.

<Generation of Script and Setting File> When the user designates each element of the parallel processing specification information using the UI as described above, and performs the confirmation operation of the specified content, the parallel processing specification information is a parallel processing specification information file described in a predetermined format. Are stored in a predetermined position of the external storage device 3.

FIG. 8 is a diagram showing an example of description of a parallel processing specification information file. In FIG. 8, the parallel process specification information file includes a specification line of computer resources, a specification line of process details (process content), and a specification line of a processing target data file and a storage position of a process result thereto.

In the computer resource specification line (line 1 in FIG. 8), the identifier (“NODE”) and an argument (“3” in the example of FIG. 7) representing the number of nodes used for parallel processing are described Be done.

Further, in the specification line of the process details (the second line in FIG. 8), the identifier ("PROC") and the process type ("

Described are arguments representing PR〇C_A ") and a processing parameter (" 120.0 150.0 20.0 50.0 ") representing processing details.

In addition, in the processing target data file and storage location specification lines (lines 3 and 4 in FIG. 8), the identifier (“DATA”), the file identifier of the processing target data file, and the corresponding processing result The identification information of the storage location of the file (indicated by "xxxxx" or "xxxxy") is described. The designated line is created for each processing target data file.

Such a description (parallel processing specification information file) is automatically set by the CPU 1 by the user specifying the number of nodes, processing content, processing target data file group, and storage location using the UI. It is created.

FIG. 9, FIG. 10 and FIG. 11 are flowcharts showing an example of a main routine of script and setting file creation processing executed by the CPU 1 (FIG. 2). The execution of the processing is started, for example, triggered by the completion of creation of the parallel processing specification information file or the input of a processing start instruction from the user.

When the process shown in FIG. 9 is started, CPU 1 first performs an initialization process (step SO

01). Next, the CPU 1 sends the parallel processing specification information file stored in the external storage device 3 (see FIG. 8).

) To MM2 (step S002).

Next, the CPU 1 executes analysis loop processing of parallel processing specification information. In this analysis loop process, CPU 1 takes one specified line from the parallel processing specification information file. The extracted line is set as the analysis target line, and the analysis target line is analyzed.

The CPU 1 determines whether or not the analysis target line extracted from the parallel processing specification information file is a specification line of computer resources (step S 003).

At this time, if the analysis target line is a designated line of computer resources (S 003; YES), CPUl takes arguments (the number of nodes: “3” in the example of FIG. 8) in this analysis target line. It is determined as a computer resource parameter related to parallel processing, and is stored in a predetermined position (a predetermined work area on MM2) (step S004). Thereafter, the CPU 1 determines the next designated line as the analysis target line, and returns the process to step S003.

If it is determined in step S 003 that the analysis target line is not a designated line of computer resources (SO 03; N)), CPU 1 determines whether the analysis target line is a designated line of processing details. Yes (Step S005).

At this time, if the analysis target line is a specification line of processing details (S 005; YES), the CPU 1 specifies the processing type specification and arguments in this analysis target line (specified processing parameters: example of FIG. 8) If so, "PROC-A" (procedure A) corresponds to the process type specification, and "120.0 150.0 20.0 50.0" corresponds to the process parameter), and this process type and arguments are processed parameters related to parallel processing And store it in a predetermined position (work area) (step S006). Thereafter, the CPU 1 determines the next designated line as a line to be analyzed, and returns the process to step S003.

If it is determined in step S 005 that the analysis target line is not a designated line of process details (S 005; NO), CPU 1 determines that the analysis target line is a designated line of a processing target data file and a storage location, According to this judgment, the file identifier in the analysis target line and identification information of the storage position are taken out and stored in a predetermined position (work area) (S 007).

The analysis loop process described above ends when the processing for the last line of the parallel processing specification information file is completed. Subsequently, the CPU 1 advances the process to step S008 in FIG.

In step S 008, CPU 1 outputs the header portion of the parallel processing job script. The header is stored in advance at a predetermined position of the external storage device 3 as a fixed text. The header contains a configuration file transfer instruction. In step S 008, nodes to be used for parallel processing are determined based on the number of data files to be processed and the specified number of nodes used for parallel processing. Usage and load for each node # 0 to #n The situation is managed by, for example, the OS (Operating System) of the computer Y. The number of processing target data files in the parallel processing specification information file and the number of nodes are transferred to the OS.

For example, the OS extracts nodes that are permitted to be used by the user from nodes # 0 to #n, and takes into consideration the usage of multiple extracted nodes, the load status, and the number of files. Select the number of nodes. For example, from the extracted nodes, the nodes with the specified number of nodes are determined as the nodes to be used for parallel processing in the order of smaller load. The determined use and load status of each node is set as a small table 34 in the status table 33. As a result, the processing target data file group is processed in parallel by the nodes of the designated node number determined by 〇S.

Note that small table 34 for all nodes # 0 to #n is stored in status table 33 (FIG. 6), and S refers to small table 34, and the number of specified nodes is in the order of small load. A minute node may be selected, and the mask may be set (disallowed) in the small table 34 corresponding to the non-selected node.

Subsequently, the CPU 1 executes loop processing of analysis' processing of the processing target data file. The loop processing is executed for each file identifier (processing target data file) obtained in step S 007. In this loop, CPU 1 first identifies one of the designated processing target data files (processing target data files having the file identifier obtained in step S 007) (referred to as analysis target files). . Subsequently, the CPU 1 starts a subroutine of metadata analysis processing of the analysis target file (step S 009),

[0078] FIG. 12 is a flowchart showing an example of a metadata analysis' acquisition subroutine.

In FIG. 12, first, the CPU 1 receives an input of data file designation (step S101). That is, the CPU 1 receives the file identifier of the analysis target file.

[0079] Next, CPU 1 determines whether or not the file identifier has the correct format (step S102). At this time, if the file identifier does not have the correct format (S 102; NO), the processing for creating the script and the setting file ends, assuming that the processing is unsuccessful (NG). In this case, an error display process is performed to notify the user of an error. It is possible to do S.

On the other hand, if the file identifier is in the correct format (S 102; YES), the CPU 1 starts the keyword acquisition loop process. In the loop processing, first, the CPU 1 determines whether or not a keyword representing metadata is included in the file identifier (S 103).

For example, the CPU 1 extracts the directory name next to the root directory in the file identifier, and this directory name and a list of keywords in the metadata table 32 (FIG. 5) (keywords stored in the metadata table 32) Match the group name) and search for keywords that match the extracted directory name.

At this time, when the keyword can not be searched, the CPU 1 extracts the next directory name and collates with the keyword list. In this way, CPU 1 repeats the extraction process of the directory name or data file name as described above and the collation process with the keyword list until a directory name or data file name matching one of the keywords is found. .

If the CPU 1 finds a keyword that matches the extracted directory name or data file name (S 103; YES), the CPU 1 interrupts the extraction process, and the metadata corresponding to the keyword is extracted from the metadata table 32. Take out and acquire (Step S104).

For example, the file identifier 7 data / experiment A / 3D / statisticsA / variableB / timeXXX.

When the above process is performed using the metadata table 32 of the storage content shown in FIG. 5 with respect to 0000.000.dat "(FIG. 4), the directory name" 3D "is extracted from the file identifier and collated with the keyword list When is performed, metadata "metaOl" corresponding to "3D" is to be acquired from the metadata table 32.

When the CPU 1 acquires metadata from the metadata table 32, the CPU 1 resumes extraction of the directory name or data file name and collation with the keyword list for the file identifier. Thus, for example, with the directory name "statisticsA" next to the directory name "3D" as a keyword, the corresponding metadata "metal" is acquired from the metadata table 32 power.

Thereafter, when the matching process for the data file name is completed (a matching key ヮ If a keyword is retrieved, the keyword acquisition loop processing ends (S 104; N)) when acquisition of the corresponding metadata ends), and the subroutine (S 009) shown in FIG. 12 ends. The process returns to step SO 10 (Fig. 10) of the main routine.

In this manner, when the user specifies the file identifier of the processing target data, the computer Y automatically uses the property information (keyword) included in the file identifier for the metadata corresponding to the processing target data. Identify (acquire)

In step S010, CPU 1 analyzes the metadata in a row, and in parallel processing of the processing target data file (analysis target file) targeted by loop processing, this analysis is not performed on the analysis target file alone. Determine whether data related to the target file (related data file) is necessary.

[0089] For example, when fluid flow velocity calculation is performed in parallel processing, X component, Y component, and Z component of velocity are required. Here, if it is a data file that indicates the X component of the analysis target file power speed that is the target of the determination processing in step SO 10, each data file that indicates the Y and Z components is required as the related data file. .

Here, the file identifier is an X component and a Y component in the directory name or data file name.

, z component can include a character or a character string indicating component information indicating whether it is offset or not. It is created by routinely changing the character or character string description of the Y component and z component data information corresponding to the data file of a certain component (for example, the X component). For example, if the letter “X” of the component information contained in the file identifier is replaced with the letter “Y” or “Z” indicating the Y component or the Z component, the corresponding Y component or Z component data file It becomes a file identifier.

[0091] In step S010, if the CPU 1 finds that the analysis target file is, for example, an X component data file through analysis of the metadata obtained in step S009, the related data file is required. It is determined that (S010; YES), and the process proceeds to step soil. If not (S010; N)), the CPU 1 advances the process to step S012.

In step S011, the CPU 1 generates a file identifier of the related data file.

The file identifier of the related data file is, for example, the analysis target file as described above It can be generated by changing part of the file identifier of. The generated association is stored in the work area on MM2.

As the actual related data file is stored on the file path indicated by the file identifier of the created related data file, the related data file is stored in the database DB 31 and is written. Thereafter, the process proceeds to step S012.

In step S012, a subroutine for determining the arrangement of the analysis target file (specified data file) or the analysis target file and the related data file is executed.

FIG. 13 is a flowchart showing an example of the arrangement determination subroutine (S012). In FIG. 13, when the process is started, the CPU 1 first estimates the size of the data file allocated to the node and the computer resource A required for the process (step S201).

That is, the CPU 1 obtains the size of the analysis target file (for example, obtained from metadata). Subsequently, the CPU 1 estimates the size of the processing result file created when the processing specified by the processing detail parameter obtained in step S 006 (FIG. 9) is executed on the analysis target file according to the corresponding metadata. Do. The CPU 1 calculates the sum of the size of the analysis target file and the size of the processing result file as the computer resource A.

[0097] The size of the processing result file is, for example, extracted from the extraction range if the processing content specified by the processing detail parameter, and a process of extracting a part of the analysis target file from the extraction range specified. .

[0098] When there is a related data file for the analysis target file, it is preferable that the analysis target file and the related data file be processed on the same node in order to improve processing efficiency. For this reason, in step S201, if there is a related data file, the size of the related data file and the size of the processing result file for the related data file are also included in the computer resource A. The size of the related data file and the size of the processing result file for this can be estimated, for example, the size of the analysis target file and the size of the processing result file for this.

Next, CPU 1 can provide the user with a capacity corresponding to computer resource A with reference to situation table 33 (FIG. 6), and the load is the largest in the current load distribution situation. Search for nodes expected to be light (step S202).

That is, the CPU 1 refers to the status table 33 and refers to the user's record in each small table 34. The user ID has already been input to the computer Y by the user, for example, when starting to use the simulation system, and the CPU 1 refers to the record corresponding to this user ID.

Next, the CPU 1 subtracts the load (current usage size) from the maximum size in each record to determine the remaining usable size of the user at each node. Subsequently, the CPU 1 determines the node with the largest available size and the node with the smallest load as the node to which the analysis target file (and the related data file) should be placed.

Next, the CPU 1 updates the situation table 33 based on the computer resource A (step S 203). That is, the CPU 1 adds the value of the computer resource A to the load value (use size) of the small table 34 corresponding to the determined node.

For example, in the example shown in FIG. 6, when it is decided to allocate user A's computer resource A (for example, assuming 10 gigabyte) to node # 0, the corresponding small table 34 The load value of is updated to "20 Gbyte".

When updating of the status table 33 is completed, the CPU 1 ends the processing of the subroutine, and passes the identifier of the node determined as the file allocation destination to the main routine.

Processing Power When the processing proceeds to step S013 of the S main routine, the CPU 1 outputs an instruction statement (referred to as “data arrangement instruction statement”) related to data arrangement to nodes.

That is, the CPU 1 reads a template of data arrangement command statements (previously stored in the external storage device 3). The template is configured such that the command statement is completed if a file identifier to be placed and a node identifier are described at a predetermined position of the fixed command statement. The CPU 1 describes the identifier of the file to be analyzed (and the related data file) at a predetermined position of the template and describes the node identifier obtained in step S012. In this way, the completed data allocation statement becomes part of the parallel processing job script.

Next, CPU 1 outputs a command statement (referred to as “processing result transfer command statement”) for moving processed data (processing result file) to the storage position after completion of parallel processing (step s).

014). That is, the CPU 1 reads a template of the processing result movement instruction statement (previously stored in the external storage device 3). The template is configured such that the command statement is completed if the storage position specified by the UI is described at a predetermined position of the fixed command statement. The CPU 1 writes the storage location of the processing result file for the analysis target file obtained in step S 007 in a predetermined position of the template. In this way, the completed processing result transfer statement becomes part of the parallel processing job script.

Next, the CPU 1 stores data arrangement information (step S015). That is, the CPU 1 stores the correspondence between the file identifier and the node identifier as data arrangement information in a predetermined storage area.

When step S015 is finished, if there is a file identifier of the processing target data file that is not a file to be analyzed, processing power returns to step S009, and processing power S of steps S009 to S015 described above is executed. Ru. When the process for the file identifiers of all process target data files is completed, the process proceeds to step S016.

By the above-described loop processing, it is determined such that the load on the arrangement destination of each processing target data file included in the processing target data file group is minimized.

At step S016, the CPU 1 outputs a parallel processing program execution statement. That is, the CPU 1 reads out a parallel processing program execution statement stored in advance in the external storage device 3 and sets it as a part of a parallel processing job script. In this way, a parallel processing job script including a header, a data placement instruction statement, a processing result transfer instruction statement, and a parallel processing program execution statement is automatically generated.

Next, the CPU 1 starts the process of creating the setting file of the parallel processing program (step S017: FIG. 11). The CPU 1 starts parallel processing program setting creation loop processing. This loop process is executed for each data file to be processed.

When the process is started, the CPU 1 creates the setting for the process target data file based on the data arrangement information (the correspondence between the file identifier and the node identifier) (S018).

That is, CPU 1 extracts a portion related to one processing target data file from the data arrangement information obtained in step S 015, and processes the processing parameter corresponding to this file identifier. Combine with the meter (obtained in step S006). The CPU 1 describes the result of the combination in a predetermined format for the setting file.

The CPU 1 performs such a process for each data file to be processed, and ends the main routine when the process of step S 019 on all the data files to be processed is completed.

FIG. 14 is a diagram showing an example of description of a setting file for parallel processing program. In the example shown in FIG. 14, the setting file consists of a plurality of lines described for each data file to be processed.

In each line, from the left in FIG. 14, a node identifier, specification of processing ("PROC_A" in this example), file identifier of processing target data file, and processing parameter are described. Such configuration files are referred to when each node executes a parallel processing program.

When creation of the script and configuration file is completed, CPU 1 starts script execution. By executing the script, the computer Y transfers the configuration file to each node of the parallel computer group X according to the configuration file transfer instruction statement of the header.

In addition, computer Y transfers each processing target data file (processing target data file group) stored in file DB 31 to the node at the placement destination according to the data placement information by executing the data placement command statement. Do.

In addition, the computer Y executes, for each node, the processing result file (processed data) created by processing the processing target data file in each node by executing the processing result moving instruction statement. It instructs to store in the specified storage location (for example, prepared in file DB 31).

Further, the computer Y instructs each node to start the execution of the parallel processing program by executing the parallel processing program execution statement.

Each node (FIG. 3) at which the processing target data file group is arranged receives the setting file and the processing target data file from the computer Y via the network. These are stored in the external storage device 14 in the node. After that, the CPU 11 of each node When the execution instruction of the force parallel processing program is received, the execution of the parallel processing program is started.

FIG. 14 is a flowchart showing an execution process of a parallel processing program executed by the CPU 11. When the CPU 11 starts the process shown in FIG. 14, it first executes an initialization process (step S301). When the initialization is completed, the CPU 11 reads the setting file stored in the external storage device 14 into the MM 12 (step S302).

Next, the CPU 11 executes a processing loop of the processing target data file according to the setting file. In this processing loop, the CPU 11 sets one line in the setting file as the line to be processed, and executes the process on the data file to be processed according to the setting contents described in the line to be processed.

In the loop, first, the CPU 11 refers to the node identifier in the configuration file and determines whether this node identifier is equal to the identifier of the own node (step S303).

At this time, if the node identifiers are not equal (S 303; NO), the next line in the setting file is set as the line to be processed, and the process of step S 303 is executed.

On the other hand, if the node identifiers are the same (S 303; YES), the CPU 11 performs processing for acquiring metadata corresponding to the file identifier described in the processing target line (step S 304).

The process of step S304 is the same process as the subroutine shown in FIG. That is, the CPU 11 refers to the metadata table 32A (the data structure is the same as the metadata table 32 (FIG. 5)) stored in the external storage device 14 and retrieves the corresponding metadata.

Next, the CPU 11 executes the process on the process target data file in accordance with the process type specification in the process target line, the process parameter, and the metadata (step S305). That is, the CPU 11 gives the calculation processor 13 specification of processing type, processing parameters, file identifier and metadata. Then, the calculation processor 13 reads the processing target data file corresponding to the file identifier from the external storage unit 14 to the MM 12 and executes processing according to the processing type specification and the processing parameter based on the metadata.

After that, when the processing by the calculation processor 13 is completed, the CPU 11 outputs the data of the processing result. Data (processed data) is output as a processing result file (step S306). The processing result file is transferred, for example, to the computer Y, and the computer Y stores the processing result file in the storage location designated by the user (for example, prepared in the file DB 31).

The above-described processing is performed on each line in the setting file as a processing target line, and when the processing on all the lines is completed, the parallel processing program execution processing is ended.

Modification Example

In the above-described embodiment, the case where the computer Y and each node strength S metadata table are provided has been described. Instead of such a configuration, a configuration may be applied in which the metadata acquired by the computer Y is transferred to each node.

Further, in the present embodiment, an example in which the storage area of the process target data file (simulation data file) is provided on the external storage device 3 of the computer Y has been described. The storage area may be provided on a file server independent of the computer Y and the parallel computer group X which each node may have.

According to the present embodiment, when the user designates the file identifier, the number of nodes, the process type, the process detail parameter, and the storage position using the input environment (UI) of the parallel process specification information, the parallel processing for the data file group to be processed is performed. Processing control programs (scripts) and configuration files for parallel program execution are created automatically.

In the past, the user had to write a script description that would sometimes be several hundred lines or more, including the transfer control of the data file, by mistake for execution of parallel processing.

According to the present embodiment, a desired script and setting file are automatically created simply by designating or inputting information that is an element of the above-described parallel processing designation information using the UI. This can greatly reduce the effort of the user. In addition, since the time required to describe a script is shortened, the time required to obtain parallel processing results can be shortened. Furthermore, it is possible to eliminate the possibility of redoing parallel processing due to a user's description error. Further, the metadata for the processing target data is automatically searched for and acquired by the designation of the file identifier by the user. That is, when the user designates a file identifier, a keyword is extracted from the file identifier, and metadata corresponding to this keyword is treated as designated metadata. This eliminates the need for the user to input metadata specifications for each data file to be processed. Therefore, the effort of the user can be reduced, the processing time can be shortened, and the user's input error can be prevented.

In the automatic specification of metadata, in this embodiment, a file identifier including data storage location information (file path) is applied to the processing target data file, and a keyword indicating the nature of the processing target data (for metadata search) The keywords of) are included.

That is, data associating the processing target data with the metadata is incorporated in the file identifier. This eliminates the need to manage the data to be associated separately from the data to be processed and metadata. Therefore, effective use of storage area and reduction of management burden can be achieved. The file identifier can include multiple keywords.

Furthermore, when the user designates the data file to be processed, the user is configured to designate the file identifier including the file path. In this way, specification of the file identifier doubles as keyword input. Therefore, the burden on the user can be reduced.

Further, in the present embodiment, the metadata is configured to be stored in a storage area different from the processing target data file. As a result, the data file to be processed can be efficiently stored in the storage area.

Industrial applicability

The present invention is applicable to, for example, data processing in various numerical simulation systems.

Claims

The scope of the claims

[1] A receiving unit that receives specification of a file identifier including storage location information of a data file and a keyword for metadata search;

Metadata storage means for storing metadata representing details of the data file, extraction means for extracting the keyword from the specified file identifier, and metadata corresponding to the extracted keyword are searched from the metadata storage means Search means and

A metadata search device that includes

[2] The metadata storage means is stored in a storage area different from the storage area of the data file to which the data file identifier is given.

The metadata search device according to claim 1.

[3] The computer is

Accepts specification of file identifier including storage location information of data file and keyword for metadata search,

Extract the keyword from the specified file identifier,

A metadata search method including: searching metadata from metadata storage means corresponding to an extracted keyword.

[4] on the computer,

Accepting specification of a file identifier including storage location information of the data file and a keyword for metadata search;

Extracting the keyword from the specified file identifier; searching for metadata corresponding to the extracted keyword from the metadata storage means;

A program that runs

[5] Each data file is stored in a directory structure,

The storage location information indicates a file path

The program according to claim 4.