Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F8/00—Arrangements for software engineering
  • G06F8/40—Transformation of program code
  • G06F8/41—Compilation
  • G06F8/42—Syntactic analysis
  • G06F8/427—Parsing
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F9/00—Arrangements for program control, e.g. control units
  • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
  • G06F9/46—Multiprogramming arrangements
  • G06F9/48—Program initiating; Program switching, e.g. by interrupt
  • G06F9/4806—Task transfer initiation or dispatching
  • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system

Definitions

• Embodiments of the disclosure relate generally to the field of data processing systems.
• embodiments of the disclosure relate to systems and methods for processing machine learning algorithms in a MapReduce environment (e.g., Apache® Hadoop!).
• MapReduce environment e.g., Apache® Hadoop!
• Parallel processing is used to increase speed of execution and amounts of data to be processed.
• using a distributed network or plurality of processors means there will exist larger plurality of possible execution strategies for a job.
• One problem is that selecting a good execution strategy from the plurality, especially for implementing a plurality of ML algorithms, falls on the programmer.
• the method includes receiving a ML algorithm to be executed in the MapReduce environment.
• the method further includes parsing the ML algorithm into a plurality of statement blocks in a sequence, wherein each statement block comprises a plurality of basic operations (hops).
• the method also includes automatically determining an execution plan for each statement block, wherein at least one of the execution plans comprises one or more low-level operations (lops).
• the method further includes implementing the execution plans in the sequence of the plurality of the statement blocks.
• Figure 1 illustrates an exemplary system for implementing machine learning algorithms in a MapReduce environment.
• Figure 2 illustrates an exemplary method performed by the system of Figure 1 in implementing machine learning algorithms in a MapReduce environment.
• Figure 4 illustrates an exemplary method for generating a parsed representation as in the method illustrated in Figure 2.
• Figure 5 illustrates an example directed acyclic graph of basic operations as created by the system illustrated in Figure 1 performing the method illustrated in Figure 2.
• Figure 6 illustrates an example of corresponding lops for a binary hop representing C/D
• Figure 7 illustrates an example of packaging lops into MapReduce jobs by the system illustrated in Figure 1 performing the method illustrated in Figure 2.
• Figure 8 illustrates the replication based matrix multiplication algorithm.
• Figure 9 illustrates the cross product based matrix multiplication algorithm.
• Figure 10 illustrates an example computer architecture for implementing the system illustrated in Figure 1 and the method illustrated in Figure 2.
• Embodiments of the disclosure relate generally to the field of data processing systems.
• embodiments of the disclosure relate to systems and methods for processing machine learning algorithms in a MapReduce environment.
• MapReduce environment For example, embodiments of the disclosure relate to systems and methods for processing machine learning algorithms in a MapReduce environment.
• numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. It will be apparent, however, to one
• MapReduce is a generic parallel programming paradigm for large clusters of machines. Combined with the growing need to run machine learning (ML) algorithms on massive data-sets, the present disclosure describes novel methods and systems for implementing ML algorithms on MapReduce. This disclosure describes systems and methods in which machine learning algorithms are expressed in a higher-level language and executed in a
• MapReduce environment The higher level language exposes several constructs that constitute key building blocks for a broad class of supervised and unsupervised machine learning algorithms.
• the algorithms expressed in the higher-level language are compiled and optimized into a set of MapReduce jobs running on a cluster of machines.
• the disclosure also describes a number of optimization strategies for efficiently executing these algorithms on MapReduce frameworks (e.g., Apache® Hadoop!).
• the declarative higher-level language for writing ML algorithms frees a user from low-level implementation details and performance- tuning of implementing the algorithm in MapReduce. Additionally, the systems and methods provides performance that scales to very large datasets. The performance is comparable to hand- tuned implementation for individual algorithms.
• MapReduce consists of three phases: “Map” phase where the input data is separated out into different key-value pairs; “Shuffle” phase where the same key from different mappers are brought together; and “Reduce” phase where all values associated with an individual key are analyzed in union.
• Map and the Reduce phases are exposed while the Shuffle phase is internal to the platform.
• Shuffle phase is internal to the platform.
• cost of the Shuffle phase is an important aspect of optimizations described in this disclosure.
• FIG. 1 illustrates an exemplary embodiment of a system 101 (called “SystemML” in this disclosure) for implementing ML algorithms in a MapReduce environment.
• SystemML 101 generally includes a Language Component 102 to accept Declarative Machine learning Language (DML) scripts 103, a High-Level Operator (HOP) Component 104 coupled to the Language Component 102, a Low-Level Operator (LOP) Component 105 coupled to the HOP Component 104, and a Runtime Component 106 coupled to the LOP Component 105.
• the Runtime Component 106 connects with a distributed network or cluster 107- 108a-n to execute the scripts.
• DML Declarative Machine learning Language
• ARC920100077PCT1 5 script 1 (below), which is the Gaussian Non-Negative Matrix Factorization (GNMF) algorithm
• V read("in/V");
• H H * (W T V/W T WH);
• H H*(t(W)%*%V)/(t(W)%*%W%*%H);
• DML supports three main data types: matrices, vectors, and scalars. Supported scalar data types are integer, double, string and logical. The cells in a matrix or vector may consist of integer, double or string values.
• a DML program consists of a sequence of statements, with the default computation semantics being sequential evaluation of the individual statements. The following constructs are currently supported in DML:
• the user can provide additional properties of the matrix or vector, such as dimensionality and sparsity (number of non-zero entries or nnzs).
• Control structures supported in DML include the while statement, for statement and if statement. Steps 6-9 in Script 1 show an example while statement.
• Assignment An assignment statement consists of an expression and the result of which is assigned to a variable (e.g., Steps 7 - 9 in Script 1). The assignment can be to a scalar, vector or matrix.
• DML supports the following main types of operators:
• Arithmetic operators include multiplication, division, addition, and
• Arithmetic operators extend naturally to vectors and matrices where the semantics dictate that the operator is applied to the corresponding cells.
• Relational/Boolean operators include less than, greater than, and equal to.
• Internal functions include aggregate functions (e.g., sum, rowSum), mathematical functions such as sin and log, and linear algebra functions (e.g., transpose t, diag).
• ARC920100077PCT1 7 DML also allows users to define their own functions using the syntax "function (arglist) body.”
• the arglist includes a set of formal input and output arguments.
• the body is a group of valid DML statements.
• DML does not support advanced programming features such as object oriented features, advanced data types (e.g., lists and arrays), and advanced function support (e.g., accessing variables in the caller function and further up in the call-stack). Additionally, DML does not support extensive graphical procedures that are supported by R.
• Figure 2 illustrates an exemplary embodiment of method 200 performed by
• FIG. 1 SystemML 101 in implementing ML algorithms in a MapReduce environment.
• the Language Component 102 of SystemML 101 receives DML scripts 103 for an ML algorithm to be implemented. Proceeding to 202, the Language Component 102 breaks the algorithm into a parsed representation of smaller units called statement blocks.
• Figure 4 illustrates an exemplary embodiment of method 400 for generating a parsed representation.
• data types are assigned to each variable in the DML script. For example, from Script 1, ReadMM statements (Steps 1-3) are used to type V, W and H as
• H H*(t(W)%*%V)/(t(W)%*%W%*%H);
• the different types of statement blocks may include: (1) a simple statement block (e.g., including basic script that is
• ARC920100077PCT1 9 run one time during execution, such as Statement Block 1 above); (2) a repeating statement block (statement blocks whose script may be executed more than once, e.g., loops in code, including for, while, do-while, etc. , such as Statement Block 2 above); and (3) skip statement blocks (statement blocks whose script may not be executed, e.g., a conditional statement, such as if, etc.).
• the type of statement block may be used to determine how and whether to execute the low-level operations (e.g., skipping lops for a skip statement block whose if condition is not met for execution).
• SystemML 101 determines what variables need to be passed across statement blocks.
• variable W used in Step 7 refers to the output of Step 2 (for the first iteration of the loop) and Step 8 for second iteration onwards.
• Step 8 for second iteration onwards.
• each use of a variable in the script is connected with the immediately preceding write(s) for that variable across different evaluation paths.
• the HOP Component 104 determines a high-level execution plan for the plurality of statement blocks.
• the HOP Component 104 receives the parsed representation of a statement block as input and produces a directed acyclic graph of basic operations (hops) over matrices, vectors, and scalars (HOPDag).
• hops basic operations
• HOPDag scalars
• Binary e.g., b(/): A binary hop takes two inputs (where each input is a matrix, vector, or scalar) and performs one of the following operations: *, +, -, /, min, max, etc..
• Unary e.g., u(+): An unary hop takes as input two operands: (1) a matrix, vector or a scalar, and (2) a scalar constant. The unary hop then performs one of the following operations: *, +, -, /.
• AggUnary (e.g., a(+)): An AggUnary hop aggregates a set of cells in a matrix or vector.
• colSum computes the sum of every column in a matrix and returns a vector of the sums.
• AggBinary e.g., a(+*)
• An AggBinary hop takes two inputs where each input is a matrix or a vector.
• the AggBinary hop then performs a Binary operation followed by an aggregate operation (e.g., +, Min, Max, etc.).
• Reorg e.g., r(T)
• a reorg hop changes the index of every cell in a matrix such as the transpose of a matrix.
• Data hop reads or writes a matrix, vector or scalar. Data hops can be transient or persistent with respect to the lifetime of a script.
• a statement block is represented in one HOPDag.
• Figure 5 illustrates the HOPDag using the hops introduced above for the body of the While loop in Statement Block 2 from Script 1.
• multiple statements in the statement block may be combined into a single HOPDag.
• the Read W data hop feeds into the r T reorg hop, which in turn feeds into the a(+*) AggBinary hop together with the Read W data hop to represent the t(W)% * %W part of the statement block.
• hops are instantiated and connected to build a HOPDag for the statement blocks in the program. While the figure illustrates the HOPDag being connected, it should be noted that a HOPDag does not have to be connected.
• the grayed Read data hops represent the live-in variables for matrices W, H, and V, and scalar i at the beginning of an iteration.
• the grayed Write data hops represent the live-out variables at the end of an iteration that need to be passed onto the next iteration.
• These data hops - which are transient - implicitly connect HOPDags of different statement blocks with each other by mapping the transient Write data hops (sinks) of the HOPDag of one statement block to the transient Read data hops (sources) of the HOPDag of the next statement block, or the next iteration of the while loop.
• the LOP Component 105 translates the HOPDag into low-level physical execution graphs (LOPDags).
• a low-level operator (lop) represents a basic operation in a MapReduce environment. Each lop takes one or more sets of key-value pairs as input and generates one or more key-value pairs as output. The following are lops supported in SystemML 101 :
• a binary lop performs computations involving two inputs where each input is a set of key -value pairs.
• unary A unary lop performs computations where one input is a set of key -value pairs and the other input is a scalar value.
• transform A transform lop applies a transformation function (e.g., transpose) on each input key.
• transformation function e.g., transpose
• a data lop represents an input data source or output data sink.
• a mmcj lop corresponds to cross product computation in cross product based matrix multiplication (CPMM).
• a mmrj lop corresponds to computation in replication based matrix
• RPM multiplication
• the HOPDag is processed in a bottom-up fashion, wherein each hop is converted into one or more lops.
• Figure 6 illustrates an example of corresponding lops 600 for the binary hop C / D (illustrated in Figure 3, from algorithm 1).
• each of the two data lops returns one set of key -value pairs for the input matrices.
• one entry for each cell in the individual matrices would be returned.
• a data lop typically returns multiple cells for each key (the number of cells returned is determined by an appropriate blocking strategy).
• a group lop then groups the multiple entries for the two inputs. The results are then passed to a binary lop to perform the division of the corresponding cell-values from the two input matrices.
• cost-based optimization may be used in creating a LOPDag.
• a plurality of choices may exist for translating a hop into one or more lops. Therefore, cost-based optimizations that consider various data characteristics of involved matrices may be used to lower the transaction cost for the chosen group of lops for the hop.
• cost-based optimization includes selecting from multiple methods of performing matrix multiplication, as later described.
• the LOP Component 105 compiles the LOPDags into at least one MapReduce job. Translating each lop into a separate MapReduce job would result in multiple scans of input data and intermediate results. Therefore, if multiple lops are packaged into a single MapReduce job, the resulting reduction in scans typically result in an improvement in efficiency.
• a greedy piggybacking heuristic algorithm (below as algorithm 2) is used to analyze and group multiple lops into one MapReduce job.
• Al orithm 2 Piggybacking— Packing lops that can be evaluated together in a single
• Figure 7 illustrates an example 700 of packaging lops into MapReduce jobs for step 7.
• 701 (left portion of 700) illustrates the generated LOPDag for the part of the HOPDag 500 ( Figure 5) labeled "H Assignment.”
• 702 (right portion of 700) illustrates the corresponding MapReduce jobs for the LOPDag 701.
• the Runtime Component 106 optimizes execution of the MapReduce job(s) during runtime.
• SystemML 101 partitions matrices and vectors into blocks (called blocking) and exploits local sparsity within a block to optimize the number of key -value pairs representing matrices and vectors.
• Blocks are smaller rectangular sub-matrices using a designated block-size. Each block is represented in a key-value
• ARC920100077PCT1 15 pair The key denotes the block id.
• the value carries all of the cell values in the block.
• Local Sparsity refers to the sparsity of an individual block. The layout of the values in a block is decided based on the sparsity in the block (i.e., the fraction of non-zero values in the block).
• dynamic block-level operations are based on local sparsity of the block.
• local sparsity information is used to decide on the appropriate execution at runtime per block.
• there is a separate algorithm inside every lop to account for the fact that individual blocks may be dense or sparse.
• the actual multiplication algorithm chosen in the lop is decided based on the local sparsity of the two input blocks. If both blocks are dense, the runtime chooses an algorithm that cycles through every cell in both blocks. However, if one of the blocks is sparse, the runtime chooses an algorithm that cycles through only the nonzero cells in the sparse block, which are multiplied with the values in the corresponding cells in the dense block.
• G-MR MapReduce job
• the G-MR is instantiated by the piggybacking algorithm (algorithm 2 above) with one or more lops.
• the MapReduce job marked 1 in 702 of Figure 7 includes three lops: data W; transform; and mmcj .
• G-MR is parameterized as follows:
• the Map phase of the MapReduce job runs data, transform, and the map portion of mmcj in sequence as indicated by the LOP Component 105.
• the control module of the Runtime Component 106 orchestrates the execution of all MapReduce jobs for a DML script.
• the control module performs the following operations: (i) instruction scheduling and (ii) instruction execution.
• Such operations performed in the control module include scalar computations in the script (e.g., scalar arithmetic operations and predicate evaluations) and metadata operations (e.g., deletion of intermediate results) during the execution of DML scripts.
• SystemML 101 may execute the resulting MapReduce jobs by sending the resulting jobs to a framework for running applications on a distributed network 107.
• a framework for running applications on a distributed network 107 is Apache® Hadoop! for processing the jobs on distributed nodes 108a-n.
• SystemML 101 supports at least two matrix multiplication algorithms, RMM and CPMM. To illustrate RMM and CPMM, let A and B be blocked matrices with M b x K b blocks in
• FIG. 8 illustrates the RMM algorithm 800, which requires only one MapReduce job for execution.
• the LOPDag for the execution plan including RMM contains a single mmrj lop (implemented in G-MR).
• a reducer is to access all blocks of A and B that are necessary to compute each result block of C. Since each block A contributes to the
• the mapper sends the number of copies of A to compute the several result blocks to the corresponding reducers. The same is done for each block B, too, with the reducer now having all required blocks to compute C.
• FIG. 9 illustrates the CPMM algorithm 900, which requires two MapReduce jobs for execution.
• CPMM is represented in LOPDags as a sequence of three lops: mmcj ; group; and aggregate.
• example 700 illustrates a CPMM evaluation for the computation of W T W.
• the mapper In the second MapReduce job, the mapper reads the results from the previous MapReduce job and groups all Cf ' s by the key (i,j).
• SystemML 101 may include an optimized implementation of mmcj .
• the optimized implementation is a local aggregator that enables partial aggregation in the reducer.
• the first MapReduce output is Cf j for 1 ⁇ k ⁇ K b .
• K b is larger than the number of available reducers r, each reducer may process multiple groups.
• a disk-based local aggregator may be implemented.
• the disk-based local aggregator is configured to use an in-memory buffer pool to perform local aggregation. If cross product results spill to disk, the results may be sorted to ensure that partial aggregation for subsequent groups is performed efficiently.
• SystemML 101 selects between CPMM and RMM.
• SystemML 101 optimizes the selection through comparing cost models for using the different algorithms.
• RMM mappers replicate each block of A and B the number of times equal to the aggregate number of blocks of C to be computed for each block of A and B (noted as number N b for A and number M b for B).
• mappers read blocks of A and B and send the blocks to reducers.
• the reducers perform cross products for each k and apply a local aggregator to partially aggregate the results across different values of k within a reducer.
• the size of the result set produced by each reducer is bounded by
• the second MapReduce job the data from the first MapReduce job is read, shuffled, and fed into the reducers to produce a final result.
• cost(CPMM) ⁇ shuffle(
• SystemML 101 compares cost(RMM) to cost(CPMM) to determine an appropriate algorithm for a particular matrix multiplication.
• RMM cost
• CPMM typically will perform better than RMM (since the shuffle overhead for RMM would be large).
• the overhead is low enough such that RMM typically will perform better than CPMM. It should be noted that when a data shuffle and IOdfs operation are of the same size, the data shuffle is a more expensive operation because it involves network overhead and local file system 10 and external sorting.
• Figure 10 illustrates an example computer architecture for implementing the systems and methods described in this disclosure.
• the exemplary computing system of Figure 10 includes: 1) one or more processors 1001; 2) a memory control hub (MCH) 1002; 3) a system memory 1003 (of which different types exist such as DDR RAM, EDO RAM, etc,); 4) a cache 1004; 5) an I/O control hub (ICH) 1005; 6) a graphics processor 1006; 7) a display/screen 1007 (of which different types exist such as Cathode Ray Tube (CRT), Thin Film Transistor (TFT), Liquid Crystal Display (LCD), DPL, etc.); and/or 8) one or more I/O devices 1008.
• CTR Cathode Ray Tube
• TFT Thin Film Transistor
• LCD Liquid Crystal Display
• the one or more processors 1001 execute instructions in order to perform whatever software routines the computing system implements.
• the instructions frequently involve some sort of operation performed upon data. Both data and instructions are stored in system memory 1003 and cache 1004.
• Cache 1004 is typically designed to have shorter latency times than system memory 1003. For example, cache 1004 might be integrated onto the same silicon chip(s) as the processor(s) and/or constructed with faster SRAM cells whilst system memory
• ARC920100077PCT1 20 1003 might be constructed with slower DRAM cells. By tending to store more frequently used instructions and data in the cache 1004 as opposed to the system memory 1003, the overall performance efficiency of the computing system improves.
• System memory 1003 is deliberately made available to other components within the computing system.
• the data received from various interfaces to the computing system e.g., keyboard and mouse, printer port, LAN port, modem port, etc.
• an internal storage element of the computing system e.g., hard disk drive
• system memory 1003 prior to their being operated upon by the one or more processor(s) 1001 in the implementation of a software program.
• data that a software program determines should be sent from the computing system to an outside entity through one of the computing system interfaces, or stored into an internal storage element is often temporarily queued in system memory 1003 prior to its being transmitted or stored.
• the ICH 1005 is responsible for ensuring that such data is properly passed between the system memory 1003 and its appropriate corresponding computing system interface (and internal storage device if the computing system is so designed).
• the MCH 1002 is responsible for managing the various contending requests for system memory 1003 access amongst the processor(s) 1001, interfaces and internal storage elements that may proximately arise in time with respect to one another.
• I/O devices 1008 are also implemented in a typical computing system. I/O devices generally are responsible for transferring data to and/or from the computing system (e.g., a networking adapter); or, for large scale non-volatile storage within the computing system
• ARC920100077PCT1 21 (e.g., hard disk drive).
• ICH 1005 has bi-directional point-to-point links between itself and the observed I/O devices 1008.
• Components of the different embodiments of a claimed system may include software, hardware, firmware, or any combination thereof.
• the components may be software programs available to the public or special or general purpose processors running proprietary or public software.
• the software may also be specialized programs written specifically for signature creation and organization and recompilation management.
• storage of the system may include, but is not limited to, hardware (such as floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, flash, magnetic or optical cards, propagation media or other type of media/machine-readable medium), software (such as instructions to require storage of information on a hardware storage unit, or any combination thereof.
• elements of the present invention may also be provided as a machine- readable medium for storing the machine-executable instructions.
• the machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, flash, magnetic or optical cards, propagation media or other type of media/machine-readable medium suitable for storing electronic instructions.
• embodiments of the invention may include the various processes as set forth above.
• the processes may be embodied in machine-executable instructions which cause a general-purpose or special-purpose processor to perform certain steps. Alternatively, these processes may be performed by specific hardware
• ARC920100077PCT1 22 components that contain hardwired logic for performing the processes, or by any combination of programmed computer components and custom hardware components.
• Embodiments of the invention do not require all of the various processes presented, and it may be conceived by one skilled in the art as to how to practice the embodiments of the invention without specific processes presented or with extra processes not presented.
• embodiments of the disclosure may be embodied across a plurality of machines.
• translation may be performed in a distributed computing environment, wherein portions of the methods may be performed on different computers situated within the distributed network.
• execution of the jobs may be performed in a distributed computing environment.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Software Systems (AREA)
• General Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Data Mining & Analysis (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Artificial Intelligence (AREA)
• Evolutionary Computation (AREA)
• Medical Informatics (AREA)
• Computing Systems (AREA)
• Mathematical Physics (AREA)
• Devices For Executing Special Programs (AREA)
• Data Exchanges In Wide-Area Networks (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Complex Calculations (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=46753913

Family Applications (1)

Country Status (6)

Cited By (2)

Families Citing this family (85)

Citations (1)

Family Cites Families (12)

• 2011
  • 2011-03-01 US US13/038,086 patent/US8612368B2/en active Active - Reinstated
• 2012
  • 2012-02-29 WO PCT/CA2012/050123 patent/WO2012116449A1/en not_active Ceased
  • 2012-02-29 DE DE112012000628T patent/DE112012000628T5/de not_active Ceased
  • 2012-02-29 CN CN201280010904.1A patent/CN103403700B/zh active Active
  • 2012-02-29 JP JP2013555715A patent/JP5705338B2/ja active Active
  • 2012-02-29 GB GB1314958.8A patent/GB2502020A/en not_active Withdrawn

Patent Citations (1)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events