JP2012160013A - Data analysis and machine learning processing unit, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reuse intermediate data utilized in a past processing in Map Reduce.SOLUTION: Mapper and Reducer detect, during processing of data which is input, a parameter value, a use position, and a usage source function from the data, as information relating to a parameter utilized in a mapping means and a reducing means, store them in a distributed file storage means, refer to a parameter given to a job, collate it with information relative to the parameter stored in the distributed file storage means and store intermediate data as a cache in the distributed file storage means or a local storage means if a prescribed condition is satisfied, and refer to the parameter given to the job and make the Reducer execute if the intermediate data is stored in the distributed file storage means or the local storage means.

Description

  The present invention relates to a data analysis and machine learning processing apparatus, method, and program, and more particularly, to a data analysis and machine learning processing apparatus and method for improving the efficiency of machine learning processing for analysis processing of large-scale data, and Regarding the program.

  As a technique for analyzing large-scale data, there is “MapReduce” (for example, see Non-Patent Document 1).

  FIG. 23 shows the flow of map reduce processing. MapReduce uses multiple computers connected together by a network. MapReduce is a distributed processing framework in which processing is performed as shown in FIG.

  When the MapReduce process is started, the "Mapper" arbitrarily defined by the user is started on each computer, each computer reads the distributed data stored in advance in the distributed file system (HDFS), and the map (Map ) Processing is performed.

  In the map processing, a map function defined by the user is applied once for each row in order from the top to the distributed data assigned to each computer. In the Map function, a set of records in a key-value format is output as intermediate data as a result of processing. Note that the key and value types are arbitrarily defined by the user under certain restrictions.

  Next, the Reducer arbitrarily defined by each user is activated in each computer, and the intermediate data is moved to each other via the network so that records having the same key part of the intermediate data are collected in one computer. This is called “Shuffle processing”.

  Records of intermediate data with the same key are finally sorted, and Value is given to the reduce function defined by the user as an iterator. The Reduce function outputs a set of records in multiple key-value formats as a result of processing.

  FIG. 24 shows a processing flow of a general application using MapReduce.

  Step 100) Input an initial setting file and generate a job to start MapReduce.

  Step 110) The number of mappers, reducers and reducers to be used for the generated job, other settings required for the MapReduce process itself, and the mapper and reducer defined by the user Give parameters that can be changed at runtime to the MapReduce job by analyzing parameters, command line arguments, etc.

  Step 120) MapReduce processing is performed. Details will be described with reference to FIG.

  Step 130) Any processing (processing defined by the user) is performed on the result output by the MapReduce processing.

  Next, the MapReduce process in step 120 of FIG. 24 will be described. MapReduce processing is a distributed processing framework that runs on a cluster in which multiple nodes (computers) are connected to each other via a network.

  FIG. 25 shows a flow of general MapReduce processing.

  Step 200) When MapReduce processing is started, first, Mapper arbitrarily defined by the user is started in each node, each node reads the distributed data stored in the distributed file system in advance, and Map processing is performed. . At this time, arbitrary Map processing defined by you is performed. A map function defined by the user is applied once to each row of the distributed data assigned to each node in order from the top, and any intermediate data in the key-value format is output.

  Step 210) Only when the user explicitly specifies the class to be used for the combine (Combine), as soon as the map processing is completed at each node, the combiner (combiner) is started and output by the respective map processing. Combined processing (local reduction) that combines intermediate data with a common key into a single target is performed, and multiple intermediate data in the form of a Key / Value list are output. .

  Step 220) Shuffle (Shuffle) for moving intermediate data to each other through a network so that a user-defined reducer is activated at each node and records having the same key part of the intermediate data are collected in one node. ) Processing is performed. During Shuffle, the intermediate data is sorted based on the Key part, and a list of multiple Value formats is output for one Key.

  Step 230) Reduce processing is performed on each of the shuffled intermediate data.

  The above technology performs multiple MapReduce processing (Grep processing) using the same data as input, adds tags to intermediate data keys, and reduces intermediate data (pre-job integration and simultaneous multiple jobs) (For example, see Non-Patent Document 2).

MRShare: Sharing Across Multiple Queries in MapReduce [Tomasz Nykiel, Michalis Potamias, Chaitanya Mishra, George Kollios, Nick Koudas, VLDB2010, September 2010] MapReduce: Simple_ed Data Processing on Large Clusters [Jeffrey Dean, Sanjay Ghemawat, OSDI2004] http://static.googleusercontent.com/external content / untrusted dlcp / labs.google.com / en // papers / mapreduce-osdi04.pdf

  In machine learning processing, the accuracy of the processing result obtained depending on the setting value given to the machine learning algorithm in advance may vary greatly, so the best result can be obtained unless the processing is repeated multiple times while adjusting the setting value. Can not. In particular, when performing processing for large-scale data using MapReduce, for example, "Mahout", which is a machine learning library, does not take into account the adjustment of setting values, so it takes a long time to obtain good results. This process is very inefficient.

  The technique of Non-Patent Document 1 described above can reduce the Shuffle cost due to the reduction of intermediate data by sharing the data reading portion of each process in a plurality of processes, but by reusing the intermediate data of the previous process. There is a problem that the processing amount cannot be significantly reduced.

  The present invention has been made in view of the above points, solves the problem that the intermediate data used in the past processing cannot be reused, and enables data analysis and machine learning processing apparatus and method capable of greatly reducing the processing amount. And to provide a program.

In order to solve the above problems, the present invention (Claim 1) includes a map means for generating one or more key / value pairs from one key / value pair, and a plurality of keys having the same key. Reduce means for generating one or a plurality of key / value pairs from key / value pairs, and distributed file storage means for storing given teacher data, paralleling large-scale data A data analysis / machine learning device for distributed processing,
The map means and the reduce means are:
Detection of parameter values, use positions, and use source functions from the data as information related to parameters used by the map means and the reduce means during processing of input data, and storing in the distributed file storage means Having means,
Refer to the parameter given to the job, collate with the information about the parameter stored in the distributed file storage means, and when the predetermined condition is satisfied, the intermediate file is used as a cache to the distributed file storage means or local Intermediate data storage means for storing in the storage means;
With reference to the parameters given to the job, if the intermediate data is stored in the distributed file storage means or the local storage means, a skip processing means for executing only the reduce means,
Have
The reducing means includes
When the intermediate data is stored in the distributed file storage unit or the local storage unit, the distributed file storage unit or a unit that reads the intermediate data from the local storage unit and performs processing including.

Further, the present invention (Claim 2) includes the detection means according to Claim 1,
A class name of the mapping unit and the reducing unit, a mapping unit that uses a predetermined parameter name and value, a usage order during processing of the reducing unit, and a unit that detects a function to be used.

Further, the present invention (Claim 3) provides the intermediate data storage means according to Claim 1,
The intermediate data is stored as a cache when the processing of the mapping means depends on the parameter, the usage frequency of the parameter exceeds a predetermined threshold, or the user designates saving. Including means.

According to the present invention (Claim 4), there is provided a map means for generating one or more key / value pairs from one key / value pair, and one or more key / value pairs having the same key. Has a reduction means for generating a plurality of key / value pairs and a distributed file storage means for storing given teacher data, and is a data analysis / machine in a device for parallel and distributed processing of large-scale data A learning method,
The map means and the reduce means detect a parameter value, a use position, and a use source function from the data as information on parameters used by the map means and the reduce means during processing of input data, Detecting in the distributed file storage means;
The intermediate data storage means refers to the parameter given to the job, collates with the information about the parameter stored in the distributed file storage means, and when the predetermined condition is satisfied, the intermediate data is used as a cache for the distributed file. An intermediate data storage step for storing in a storage means or a local storage means;
A skip processing unit refers to a parameter given to a job, and when the intermediate data is stored in the distributed file storage unit or the local storage unit, a skip process for executing only the reduce unit Steps,
When the intermediate data is stored in the distributed file storage unit or the local storage unit, the reducing unit reads the intermediate data from the distributed file storage unit or the local storage unit. And the step of performing the process.

Further, the present invention (Claim 5) is the detection step of Claim 4,
The class names of the map means and the reduce means, the order of use of the parameter names and values given in advance during the processing of the map means and the reduce means, and the functions used are detected.

Further, the present invention (Claim 6) is the intermediate data storage step of Claim 4,
The intermediate data is stored as a cache when the processing of the mapping means depends on the parameter, the usage frequency of the parameter exceeds a predetermined threshold, or the user designates saving. .

  The present invention (Claim 7) is a data analysis / machine learning program for causing a computer to function as each means constituting the data analysis / machine learning apparatus according to any one of Claims 1 to 3.

  As described above, in the present invention, during the MapReduce process, the use order of parameters and the use function are detected, it is determined whether to use the information to leave intermediate data as a cache, and when a predetermined condition is satisfied Stores intermediate data in a distributed file system and reuses intermediate data cached in subsequent processing, skipping map processing and shuffle processing, and reducing (Reduce) By starting only the processing, the processing amount can be reduced.

It is a system configuration figure in a 1st embodiment of the present invention. It is a flowchart of the whole operation | movement in the 1st Embodiment of this invention. It is a flowchart (S310) of the process using MapReduce which preserve | saves the intermediate data in the 1st Embodiment of this invention. It is a flowchart (S620) of the process of the intermediate data preservation | save determination in the 1st Embodiment of this invention. It is a flowchart (S320) of the process using MapReduce which reuses the intermediate data in the 1st Embodiment of this invention. It is a flowchart (S920) of the intermediate data reuse determination process in the 1st Embodiment of this invention. It is a flowchart (S300) of the process using MapReduce which detects the parameter information in the 2nd Embodiment of this invention. It is a flowchart (S420) of MapReduce processing which detects parameter information in a 2nd embodiment of the present invention. It is an image of the parameter information detection in the 2nd Embodiment of this invention. It is a flowchart (S630) of the MapReduce process which preserve | saves the intermediate data in the 3rd Embodiment of this invention. It is a flowchart (S930) of the MapReduce process which reused the intermediate data in the 3rd Embodiment of this invention. It is a figure which shows the motion of each node at the time of the MapReduce process start by Hadoop of one Example of this invention. It is a flowchart of a general process of a client node. It is a flowchart (S1450) of processing when adding a job. It is a flowchart (S1450) of the process at the time of the job addition of one Example of this invention. It is a flowchart of the process of the job scheduler of one Example of this invention. It is a flowchart of a process of the task acquisition loop of TaskTracker of one Example of this invention. It is a flowchart (S1821, S1831) of the process at the time of the task start of one Example of this invention. It is a flowchart (S1920) of the process of the mapper for parameter detection of one Example of this invention. It is a flowchart of the process at the time of the parameter call of the Configuration class for a detection of one Example of this invention. It is a flowchart (S1920) of the process of the detection reducer of one Example of this invention. It is a flowchart (S1920) of the process of the mapper for skip of one Example of this invention. This is the flow of MapReduce processing. It is a flowchart of general application processing using MapReduce. It is a flowchart of the process of general MapReducer.

  Embodiments of the present invention will be described below with reference to the drawings.

[First embodiment]
FIG. 1 shows a system configuration in the first embodiment of the present invention.

  As in FIG. 23, the configuration of the present invention is a system that uses a plurality of computers connected to each other via a network and performs a distributed processing framework in which processing is performed in the same flow as in FIG.

  In the present embodiment, in the MapReduce process, information related to the parameters is detected and stored in the totaling data creation unit (Mapper) 20 and the totaling unit (Reducer) 40. Thereafter, the controller 50 compares the saved information with the Mapper, Reducer, and parameters used for the processing at the start of the MapReduce process, and saves the intermediate data as a cache in the distributed file system 10. Is determined by user designation or the like. Further, in the subsequent processing, it is determined whether or not the cache stored in the distributed file system 10 is to be used, and the map / shuffle processing is skipped by reusing the intermediate data.

  Details of the operation will be described below.

  FIG. 2 is a flowchart of the overall operation in the first embodiment of the present invention.

  Step 300) In this step, MapReduce processing similar to that shown in FIG. 24 is performed. In this process, parameter values, usage positions, and usage source functions used by the Mapper 20 and Reducer 40 are detected and stored in the distributed file system 10.

  Step 310) In the process of MapReduce processing shown in FIG. 24, the control unit (Controller) 50 refers to the parameter given to the job, and collates with the information about the parameter obtained in Step 300, thereby caching the intermediate data. In the case of saving, the intermediate data is saved in the distributed file system 10 or locally.

  Step 320) In the process of the MapReduce process shown in FIG. 24, the Controller 50 refers to the parameter given to the job, determines whether the intermediate data cache obtained in Step 310 can be reused, and if possible Skips Map processing and Shuffle processing by reusing the cache, and only performs Reduce processing of Reducer 40.

  Next, the detailed operation of step 310 will be described.

  FIG. 3 is a flowchart of processing using MapReduce for storing intermediate data according to the first embodiment of this invention.

  Step 600) As in FIG. 24, a MapReduce job is generated.

  Step 610) As in FIG. 24, parameters are given to the job.

  Step 620) Based on the parameters held by the job that is the output of Step 610, the Controller 50 refers to the information related to the parameters stored in the distributed file system 10 that is the output of Step 300, and uses the intermediate data as a cache in the job. It is determined whether or not to save, and in the case of saving, a parameter indicating a save flag “true” is newly added.

  Step 630) In the course of the MapReduce process, immediately before performing the Reduce process, the intermediate data is stored in the distributed file system 10 as a cache.

  Step 640) The same processing as step 130 in FIG. 24 is performed.

  Next, the processing in step 620 will be described in detail.

  FIG. 4 is a process flowchart of intermediate data storage determination in the first embodiment of the present invention.

Step 700) When the information about the mapper class name and the parameter is given, the controller 50 refers to the information about the parameter based on the mapper class name. If there is no parameter call in the processing of the mapper 20, “true” is set. If not, “false” is output. Step 710) The controller 50 determines whether the frequency with which the same value is set exceeds the threshold specified by the user in all the parameters used in the map processing. Specifically, based on the Mapper class name, the information about the given parameter is referred to, and the same value is set with a frequency that all of the parameters used in the processing of Mapper 20 exceed the threshold set by the user. If this is the case, “true” is output. Otherwise, “false” is output.

  Step 720) The controller 50 obtains a set of parameters, and checks whether there is any in the set that explicitly enables saving in the intermediate data storage unit 30, and if there is, it does not have “true”. In this case, “false” is output.

  Step 730) In any of the above Steps 700, 710, and 720, if "true", the Controller 50 validates the intermediate data for the parameter held by the MapReduce job, and the location for storing the intermediate data Is granted.

  Next, the process of step 320 in FIG. 2 will be described in detail.

  FIG. 5 is a flowchart of processing using MapReduce for reusing intermediate data according to the first embodiment of this invention.

  Step 900) The Controller 50 generates a MapReduce job as in Step 100 of FIG.

  Step 910) Parameters are given to the generated job in the same manner as Step 110 in FIG.

  Step 920) Based on the parameters held by the job which is the output of Step 910, it is determined whether intermediate data usable in the job is stored in the distributed file system 10 as a cache. A parameter whose usage flag indicates “true” is newly added to the job.

  Step 930) The Reducer 40 uses the intermediate data stored in the intermediate data storage unit 30 as a cache, skips the Map process and the Shuffle process, and performs the MapReduce process with only the Reduce process.

  Step 940) The same processing as step 130 in FIG. 24 is performed.

  Next, the processing of step 5 920 will be described.

  FIG. 6 is a flowchart of intermediate data reuse determination processing according to the first embodiment of this invention.

  Step 1000) Based on the parameters held by the job, it is determined whether there is a cache (same input data, intermediate data when processed using the same parameters) that can be used in the distributed file system 10.

  Step 1010) If there is, a cache use flag including the cache storage location used for the job is assigned.

[Second Embodiment]
In this embodiment, in the processing of step 300 in FIG. 2 of the first embodiment, the names of Mapper class and Reduce class used in MapReduce processing, parameter names and values given in advance, which are in MapReduce processing In this order, it is detected in which function it is used, and the information is stored in the distributed file system 10 so that it can be used in subsequent processing.

  The system configuration in this embodiment is the same as that in the first embodiment.

  FIG. 7 is a flowchart of processing using MapReduce for detecting parameter information according to the second embodiment of the present invention.

  The following processing corresponds to the processing in step 300 in FIG. 2 of the first embodiment.

  Step 400) Same as step 100 in FIG.

  Step 410) The same as step 110 in FIG.

  Step 420) The Mapper 20 and the Reducer 40 detect parameter values, use positions, and use source functions used internally from arbitrary input data, and perform MapReduce processing while storing them in the distributed file system 10.

  Step 430) Same as step 130 in FIG.

  Next, the process of step 420 will be described in detail.

  FIG. 8 is a flowchart of MapReduce processing for detecting parameter information according to the second embodiment of the present invention.

  Step 500) When the data assigned to each node is input, the Mapper 20 performs an arbitrary Map process defined by the user. The map function defined by the user is applied once to each row of the distributed data assigned to each node in order from the top. In the meantime, when the parameter is used, the used Mapper class name, function, parameter name, and parameter value are stored in the distributed file system 10.

  Step 510) When the data in the key-value format generated in step 500 is input, the keys having the same key are collected (a single key is a list-like value), and an arbitrary combine process defined by the user is performed. Do. Meanwhile, when the parameter is used, the used Combiner class name, function, parameter name, and parameter value are stored in the distributed file system 10.

  Step 520) In the Reducer 40, the same processing as in Step 220 of FIG. 25 is performed.

  Step 530) The Reducer 40 performs an arbitrary Reduce process defined by the user on the input. In the meantime, when the parameter is used, the used Reducer class name, function, parameter name, and parameter value are stored in the distributed file system 10.

  FIG. 9 shows a processing image at the time of mounting. MapReduce, which is open software, can be easily realized by using “Hadoop”. In Hadoop, all parameters given to a job in advance are managed by a class named “Configuration”, and Mapper and Reducer are controlled from the controller (control unit 50 or JobTracker (FIG. 12 described later)) in the process of MapReduce processing. Receives the location where Configuration is saved as a file, and reads and uses that file. Configuration stores all parameters including parameters used by JobTracker, in addition to parameters used by Mapper and Reducer (for example, regarding machine learning algorithms). Configuration manages parameters as name / value pairs. When name is queried, it returns the corresponding value. In the present invention, as shown in the lower part of the figure, the parameter information is detected by replacing it with a detection one. Specifically, when name is sent to Configuration for parameter use, not only simply return value, but also acquire information such as class name, function name, parameter value of inquiry source at the same time. Save to HDFS.

[Third Embodiment]
This embodiment uses the information obtained by the second embodiment, when the map processing does not depend on parameters, when the same parameter is used statistically frequently, or when specified by the user The intermediate data as a result of the map processing is stored in the distributed file system 10 as a cache. In the subsequent processing, if the saved intermediate data can be used as a cache, it is read and only the Reduce processing is performed, and the Map processing and Shuffle processing are not performed.

  The system configuration in this embodiment is the same as that in the first embodiment.

  FIG. 10 is a flowchart of MapReduce processing for storing intermediate data according to the third embodiment of the present invention. The process shown in the figure corresponds to the process of step 630 in FIG. 3 of the first embodiment.

  Step 800) Map processing is performed in the same manner as Step 200 in FIG.

  Step 810) Combine processing is performed in the same manner as Step 210 in FIG.

  Step 820) Shuffle processing is performed as in step 220 of FIG.

  Step 830) The intermediate data obtained by the above processing is output to the distributed file system 10 as a cache.

  Step 840) The same processing as step 230 in FIG. 25 is performed. However, Map processing and Shuffle processing are skipped by directly flowing Shuffled intermediate data stored in the intermediate data storage unit 30 into the Reducer 40.

  Next, the processing in step 930 in FIG. 5 will be described.

  FIG. 11 is a flowchart of MapReduce processing that reuses intermediate data according to the third embodiment of the present invention.

  Step 1100) The Reducer 40 acquires intermediate data stored in the distributed file system 10 as a cache.

  Step 1110) The Reducer 40 performs the same processing as Step 230 of FIG. 25 using the acquired intermediate data.

  Although the embodiment of the present invention has been described above, in the present invention, MapReduce is performed at least three times unless the user z explicitly specifies the intermediate data cache (when automatically performing the cache). The first time is detection of parameter information, the second time is cache determination and caching of intermediate data using the first time information, and the third time is reduced the amount of processing using the intermediate data cached in the second time.

  Hereinafter, an example in which the present invention is applied to “Hadoop” of an open source distributed system will be described.

  Hereinafter, in Hadoop, “parameter” specifically refers to the Configuration class. The Configuration class includes all parameters such as parameters of Hadoop itself, parameters defined by the user-defined Mapper and Reducer, variables such as variables used by Mapper and Reducer, etc. in the series of MapReduce processing flow by Hadoop. It is a class for storing and managing without distinction, and various classes related to MapReduce use it by inheriting the Configuration class or holding the Configuration class. Parameters are stored in the Configuration class in the form of “name: value”. When a parameter name is given to a method corresponding to a required type, a value of that type is returned. Has a method to convert to XML format for output to HDFS and local files.

  “Job (job or JobConf)” specifically refers to the Job class or the JobConf class in Hadoop. When starting MapReduce processing with Hadoop, you can create a job, give the Mapper class or Reduce class used for processing using various methods provided by the job, and start MapReduce processing with the start method. It contains the Configuration class, and basically all the given parameters are stored there.

  Note that “JobTracker” to be described later corresponds to “control unit (Controller) 50” in the above-described embodiment.

  FIG. 12 shows the operation of each node at the start of MapReduce processing by Hadoop.

  The system shown in the figure includes a JobClient 1302, a client node 1300 having a MapReduce application 1301, a JobTracker node 1310 having a JobTracker 1311, a TaskTracker 1321, and a Track Tracker node 1320 having a Map or Reduce task 1322.

  First, the operation of a general client node 1300 will be described.

  FIG. 13 is a flowchart of general processing of the client node.

  Step 1400) When the JobClient 1302 of the client node 1300 generates an instance of the Job class included in Hadoop, the initial parameter in the XML format is read and given to the job as the Configuration class.

  Step 1410) Give parameters to parameters such as Mapper / Reducer / Key / Value classes required by Hadoop and initial values of variables used during processing (giving parameters to the configuration held by the job) Output a job instance given a fixed parameter.

  Step 1420) The location of the input data and the parameters that can be changed by the user at the time of execution are obtained by analyzing the command line arguments and the like are given to the job (the parameter is given to the configuration held by the job).

  Step 1430) Using JobClient 1302, the ID of the new job is acquired from the server on which JobTracker is operating and given to the job.

  (Step 1440) The job generated and set by Steps 1400, 1410, and 1420 is output as XML to a predetermined location (path) on the distributed file system 1330 based on the job ID acquired in Step 1430 (or A file indicating input data is also output to a predetermined location).

  Step 1450) The job tracker 1311 is notified of the addition of a new job using the JobClient 1302. Details will be described later.

  Step 1460) The job added in JobTracker 1311 becomes an executing job and is accumulated in the queue. MapReduce processing is executed based on the queue as appropriate, and waits until the end while displaying the processing log on the standard output (all the Map or Reduce tasks 1322 of the Task Tracker node 1320 wait until the end).

  Step 1460) The processing of Step 1460 ends, the log information of MapReduce processing is acquired as the result of MapReduce processing, and arbitrary processing defined by the user is performed.

  FIG. 14 is a flowchart of processing when a job is added, and shows details of step 1450 described above.

  Step 1500) The job is initialized based on the input job ID, and an execution job (JobInProgress) is generated.

  Step 1510) The execution job is added to the queue.

  The process shown above is a general process, but in the present invention, the process shown in FIG. 15 is performed instead of the process shown in FIG.

  FIG. 15 is a flowchart of processing when a job is added according to an embodiment of the present invention.

  Step 1600) In the JobClient 1302 of the client node 1300, an execution job is generated as in Step 200 of FIG.

  Step 1610) With reference to Configuration held by the job, it is determined whether the parameter detection setting is valid. If it is valid, the process proceeds to step 1620. If it is not valid, the process proceeds to step 1611.

  Step 1611) The directory on the distributed file system 1330 is searched using the Mapper / Reducer class name, the input data name, and the parameter, and it is determined whether there is a reusable intermediate data cache. If there is, the process proceeds to step 1612, and if not, the process proceeds to step 1640.

  Step 1612) An operation is performed so that the Reducer processing is not performed on the configuration of the job, the Mapper class is replaced with a skipping one, and the process proceeds to Step 1640.

  Step 1620) The set mapper and reducer are saved as different setting names and replaced with the parameter detection mapper and reducer. The contents of the replaced Configuration are also reflected in the XML stored on the distributed file system 1330.

  Step 1630) When the cache is explicitly specified by the user, always "true", and the parameter stored on the distributed file system 1330 using the Mapper / Reducer cluster name, input data name, and parameter name Searches the usage class / function information of, and outputs “true” when Mapper does not depend on the parameter or when the same parameter value is used more than the number of times specified by the user. Otherwise, “false” is output and the process proceeds to step 1640.

  Step 1631) In the case of “true” in Step 1630, a flag for caching the intermediate data is given to the configuration of the execution job.

  Step 1640) Similar to step 1510 of FIG. 14, the execution job is stored in the queue.

  Next, the job scheduler process of the JobTracker 1311 of the JobTracker node 1310 in FIG. 13 will be described.

  FIG. 16 is a flowchart of processing of the job scheduler according to the embodiment of this invention.

  Step 1700) The JobTracker 1311 determines whether or not there is an execution job in the queue. If there is an execution job, the jobtracker 1311 proceeds to Step 1710, and if not, waits for the queue.

  Step 1710) An execution job is acquired from the queue.

  Step 1720) The job resource is acquired from an appropriate location of the distributed file system 1330 based on the execution job, and the necessary number of Map tasks and Reduce tasks are generated.

  Step 1730) As an end determination, it is determined whether there is an end flag. If there is an end flag, the process ends. If not, the process proceeds to step 1700.

  Next, the task acquisition process of the TaskTracker 1321 of the TaskTracker node 1320 will be described.

  FIG. 17 is a flowchart of the task acquisition loop processing of the TaskTracker according to the embodiment of this invention.

  Step 1800) The TaskTracker node 1320 of the TaskTracker node 1320 sends a heart beat (Heat Beat) to the JobTracker 1311 of the JobTracker node 1310 to confirm its own existence, and receives a response from the JobTracker 1311.

  Step 1810) The TaskTracker 1321 determines whether or not a task is included in the response from the JobTracker 1311. If it is included, the process moves to Step 1820. If not, the process moves to Step 1800.

  Step 1820) The TaskTracker 1321 determines whether or not the task is a Map task. If it is a Map task, the process proceeds to step 1821; otherwise, the process proceeds to step 1830.

  Step 1821) The TaskTracker 1321 executes the given Map task, and proceeds to Step 1800. Details will be described later with reference to FIG.

  Step 1830) It is determined whether the task is a Reduce task. If it is a Reduce task, the process proceeds to step 1831; otherwise, the process proceeds to step 1840.

  Step 1831) The given Reduce task is executed, and the process proceeds to Step 1800. Details will be described later with reference to FIG.

  Step 1840) If there is an end flag, the process is ended; otherwise, the process proceeds to Step 1800.

  Next, processing for starting the tasks (Map task, Reduce task) in steps 1821 and 1831 will be described.

  FIG. 18 is a flowchart of processing at the start of a task according to an embodiment of the present invention.

  Step 1900) The TaskTracker 1321 generates a Context class including parameters (Configuration), job progress (Status), input / output, and the like according to the task.

  Step 1910) A class (Mapper class or Reducer class) corresponding to a given task is generated.

  Step 1920) Give the generated Context class to the Mapper class or Reducer class and execute it. In the case of the Mapper class, the process of FIG. 19 is executed, and in the case of the Reducer class, the process of FIG. 20 is executed.

  When the Mapper class is generated in the above step 1910, the following processing is performed.

  FIG. 19 is a flowchart of processing when a Mapper class according to an embodiment of the present invention is generated.

  Step 2200) When the Context class (class with settings, task ID, data input / output, job status, counter, etc.) and the class used as Mapper are input, the Configuration class held by the Context class is extracted and Is replaced with Configuration for detecting the class, function, and order in which the parameter is used.

  Step 2210) The name of the Mapper class is extracted from the Configuration, and a normal Mapper class is generated.

  Step 2220) Only any pre-processing defined by the user is performed. Here, the function names in Mapper and Reducer are set to function names (function names that are easy to detect) that can be clearly identified in Mapper and Reducer. Thus, it can be easily determined whether the parameter is used.

  Step 2230) When a part of the assigned input data and the Context class are input, only the arbitrary Map processing defined by the user is performed on the next line of the input using a function name that is easy to detect. .

  Step 2240) It is determined whether or not the line read in Step 2230 is the last line of data.

  Step 2250) Only an arbitrary end process defined by the user is performed using a function name that is easy to detect.

  Next, processing when requesting a parameter for the detection context generated in step 2200 of FIG. 19 will be described.

  FIG. 20 is a flowchart of processing when a parameter of the detection Configuration class is called according to an embodiment of the present invention.

  Step 2300) Using Stack Trace, which is a function of Java (registered trademark), the function that called the function is traced, and the Mapper class name and function name of the call source are detected.

  Step 2310) On the distributed file system 1330, generate the file on the distributed file system 1330 using the caller class name, order (numerical value), function name, parameter name as the directory name, value as the file name, and frequency as the content. Output to system 1330.

  Step 2320) A value is acquired from the retained property class using the parameter name as a key, and converted into a type appropriate for the called method.

  Next, processing in the case of executing the detection reducer in step 1920 of FIG. 18 will be described.

  FIG. 21 is a flowchart of the processing of the detection reducer according to the embodiment of the present invention.

  Step 2500) When the Context class (setting, task ID, data input / output, job status, counter, etc. class) and the name of the class used as a reducer are entered, the Configuration class held by the Context class Take it out and replace it with Configuration to detect the class, function, and order in which the parameters were used.

  Step 2510) The name of the Reducer class is acquired from the Configuration, and a normal Reducer class is generated.

  Step 2520) Only arbitrary pre-processing defined by the user is performed using a function name that is easy to detect.

  Step 2530) It is determined whether there is a cache saving flag in the Configuration in the Context class. If there is, the process proceeds to step 2540, and if not, the process proceeds to step 2550.

  Step 2540) The Mapper class name, Reduce class name, input data name, and Key are set to a directory at a predetermined location in the distributed file system 1330, and all contents of the Value iterator are output to the directory.

  Step 2550) Reduce processing is performed on the assigned intermediate data (Key, Value) using a function that is easy to detect.

  Step 2560) Only an arbitrary end process defined by the user is performed.

  Next, processing in the case where the skip mapper is executed in step 1920 of FIG. 18 will be described.

  FIG. 22 is a flowchart of the processing of the skip mapper according to the embodiment of this invention.

  Step 2600) The Reducer class name is acquired from the Configuration, and a normal Reducer class is generated.

  Step 2610) Only any pre-processing defined by the user is performed.

  Step 2620) The intermediate data cache stored in the distributed file system 1330 previously stored is read and acquired in the form of a Key and Value iterator.

  Step 2630) Reduce processing is performed on the assigned intermediate data (Key, Value).

  Step 2640) Only an arbitrary end process defined by the user is performed.

  The operations of the components shown in FIG. 1 can be constructed as a program and installed in a computer used as a data analysis / machine learning device for execution, or distributed via a network.

  The present invention is not limited to the above-described embodiments and examples, and various modifications and applications are possible within the scope of the claims.

10 Distributed file system (HDFS)
20 Data creation part for aggregation (Mapper)
30 Intermediate data storage section 40 Total section (Reducer)
50 Controller (Controller)
60 Combiner
70 Shuffle
1300 Client node 1301 MapReduce application 1302 Job client (JobClient)
1310 Job Tracker Node 1311 Job Tracker Job Tracker
1320 Task Tracker node 1321 Task Tracker
1322 Map or Reduce Task 1330 Distributed File System (HDFS)

Claims (7)

Map means for generating one or more key / value pairs from one key / value pair, and one or more key / value pairs from a plurality of key / value pairs having the same key A data analysis / machine learning apparatus for generating a reduction (Reduce) means and a distributed file storage means for storing given teacher data, and for performing parallel distributed processing of large-scale data,
The map means and the reduce means are:
Detection of parameter values, use positions, and use source functions from the data as information related to parameters used by the map means and the reduce means during processing of input data, and storing in the distributed file storage means Having means,
Refer to the parameter given to the job, collate with the information about the parameter stored in the distributed file storage means, and when the predetermined condition is satisfied, the intermediate file is used as a cache to the distributed file storage means or local Intermediate data storage means for storing in the storage means;
With reference to the parameters given to the job, if the intermediate data is stored in the distributed file storage means or the local storage means, a skip processing means for executing only the reduce means,
Have
The reducing means includes
When the intermediate data is stored in the distributed file storage unit or the local storage unit, the distributed file storage unit or a unit that reads the intermediate data from the local storage unit and performs processing A data analysis / machine learning apparatus characterized by including: The detection means includes
The class means of the said map means and the said reduction means, The usage order in the process of the said map means and the said reduction means of the parameter name and value given previously, The means to detect the function used are included, The means used Data analysis and machine learning device. The intermediate data storage means includes
The intermediate data is stored as a cache when the processing of the mapping means depends on the parameter, the usage frequency of the parameter exceeds a predetermined threshold, or the user designates saving. The data analysis / machine learning apparatus according to claim 1, further comprising means. Map means for generating one or more key / value pairs from one key / value pair, and one or more key / value pairs from a plurality of key / value pairs having the same key A data analysis / machine learning method in a device that has a reduction means to generate and a distributed file storage means for storing given teacher data and performs parallel distributed processing of large-scale data,
The map means and the reduce means detect a parameter value, a use position, and a use source function from the data as information on parameters used by the map means and the reduce means during processing of input data, Detecting in the distributed file storage means;
The intermediate data storage means refers to the parameter given to the job, collates with the information about the parameter stored in the distributed file storage means, and when the predetermined condition is satisfied, the intermediate data is used as a cache for the distributed file. An intermediate data storage step for storing in a storage means or a local storage means;
A skip processing unit refers to a parameter given to a job, and when the intermediate data is stored in the distributed file storage unit or the local storage unit, a skip process for executing only the reduce unit Steps,
When the intermediate data is stored in the distributed file storage unit or the local storage unit, the reducing unit reads the intermediate data from the distributed file storage unit or the local storage unit. The steps to process in
A data analysis / machine learning method characterized by In the detection step,
5. The data analysis method according to claim 4, wherein a class name of the map means and the reduce means, a use order of the parameter names and values given in advance during the processing of the map means and the reduce means, and a function to be used are detected. Machine learning method. In the intermediate data storing step,
The intermediate data is stored as a cache when the processing of the mapping means depends on the parameter, the usage frequency of the parameter exceeds a predetermined threshold, or the user designates saving. The data analysis / machine learning method according to claim 4.   A data analysis / machine learning program for causing a computer to function as each means constituting the data analysis / machine learning device according to any one of claims 1 to 3.

Images