JPWO2015125452A1 - Data management device, data analysis device, data analysis system, and analysis method - Google Patents

Abstract

The CD method can be used even when the size of the training data exceeds the memory size of the computer. A data management device (101) of the present invention divides training data representing matrix data into a plurality of blocks, and generates metadata indicating which column values of the original training data each block holds When the partial component of the parameter learned from the training unit (20) and training data converges to 0, an old block including an unnecessary column in each block is replaced with a block from which the unnecessary column is removed. And a reblocking unit (40) for regenerating the metadata.

Description

  The present invention relates to a data management device, a data analysis device, a data analysis system, and an analysis method for solving an optimization problem using an optimization algorithm.

  Machine learning is used in the fields of data analysis and data mining. Many methods such as logistic regression in machine learning and SVM (Support Vector Machine) define an objective function when learning parameters from training data (for example, called a design matrix or feature amount). Then, the optimum parameters are learned by optimizing the objective function. Such parameters may have so many dimensions that they cannot be analyzed by the human eye. Therefore, a technique called a sparse learning method (sparse regularization learning, lasso) is used. Here, “lasso” is an abbreviation for “last absolute shrinkage and selection operator”. In the sparse learning method, learning is performed so that the values of most dimensions of the parameters become 0 in order to facilitate analysis of the learning result. In the framework of the sparse learning method, many components of the parameter converge to 0 in the learning process. Components that converge to zero are ignored as meaningless in the analysis.

  In order to perform machine learning efficiently, the efficiency of optimization problems has become an inseparable issue. Here, in the action recognition apparatus described in Patent Document 1, the coordinated descendent method (hereinafter referred to as the CD method) is used for matching motion feature amounts, and the minimum DR and C for the rotation matrix R and the correspondence matrix C are used. (X, Y) is calculated. The CD method is one of the methods for solving the optimization problem, and is a class of algorithm called a descent method.

  Here, the operation of the CD method, which is a kind of optimization method called the gradient method, will be described with reference to FIG. FIG. 15 is a diagram illustrating the movement of the CD method in a two-dimensional space. FIG. 15 conceptually shows the operation of the CD method in a two-dimensional space. In the example of FIG. 15, the parameter w is a two-dimensional vector having the component w1 and the component w2 as elements. The plurality of ellipses are contour lines indicating combinations of the component w1 and the component w2 in which the objective function f (w) has the same value. The star mark indicates the point where the value of the objective function f (w) is minimum or maximum, that is, the objective solution w *. When the objective function f (w) is given, the CD method is a point (objective solution) w * at which f (w) is minimum or maximum along each coordinate axis (each dimension) of the space of f (w). Will continue to explore. Specifically, after the starting point for search (start in FIG. 15) is determined at random, the following processing is repeated. That is, the coordinate axis (dimension) j is selected, the movement direction d and the movement width (step width) α of the search point are determined based on the training data, and the component wj of the dimension j is the component wj + α · d (hereinafter referred to as Δ). Updated). In the next process, another coordinate axis (dimension) is selected. Such processing is repeated in order for all coordinate axes (dimensions) until the value of the objective function f (w) sufficiently approaches the objective solution w *.

  As described above, when the objective function f (w) is given, the CD method calculates the objective solution w * that minimizes or maximizes the objective function f (w) along each coordinate axis of the space of f (w). I will explore. Then, when the target solution w * is sufficiently approached, the processing is stopped.

  In addition, the CD method does not require a high-cost matrix operation, unlike the Newton method, in the update calculation of the parameter w, and the calculation is low cost. Also, the CD method is a simple algorithm and can be implemented relatively easily. Therefore, many main methods of machine learning such as regression and SVM are implemented based on the CD method.

JP 2006-340903 A

  However, in the action recognition apparatus using the CD method described in Patent Document 1, when the size of the training data is larger than the memory size of the computer, the CD method cannot be applied by reading all the training data into the memory. There is.

  In view of the above problems, an object of the present invention is to provide a data management device, a data analysis device, a data analysis system, and a data analysis that can use the CD method even when the size of training data exceeds the memory size of a computer. It is to provide a method.

  The data management device according to an aspect of the present invention divides training data representing matrix data into a plurality of blocks, and generates metadata indicating which column values of the original training data each block holds. When a component of a parameter learned from training data converges to 0, an old block including an unnecessary column in each block is replaced with a block from which the unnecessary column is removed, and the meta Re-blocking means for regenerating data.

  A data analysis apparatus according to an aspect of the present invention includes a queue management unit that reads a predetermined block out of a plurality of blocks that are data obtained by dividing training data representing matrix data, and stores the predetermined block in a queue. In addition, iterative calculation means for reading the predetermined block and performing the CD method iterative calculation, and training data corresponding to the component converged to 0 when one component of the parameter converges to 0 during one iterative calculation Flag management means for transmitting a flag indicating that the column can be removed.

A data analysis system according to an aspect of the present invention is a block that divides training data representing matrix data into a plurality of blocks, and generates metadata indicating which column values of the original training data each block holds. When a component of a parameter learned from training data converges to 0, an old block including an unnecessary column in each block is replaced with a block from which the unnecessary column is removed, and the meta Re-blocking means for regenerating data, and queue management means for reading a predetermined block out of a plurality of blocks that are data obtained by dividing training data representing matrix data and storing it in a queue When,
An iterative calculation means for reading out the predetermined block stored in the queue and performing an iterative calculation of a CD method, and a component that has converged to 0 when one component of the parameter has converged to 0 during one iterative calculation A flag management means for transmitting a flag indicating that it is possible to remove the train data column corresponding to the data analysis apparatus.

  In a first recording medium readable by a computer according to one embodiment of the present invention, training data representing matrix data is divided into a plurality of blocks, and each column holds a value of which column of the original training data. A block obtained by removing an unnecessary block from an old block including an unnecessary column among the blocks when a component of a parameter learned from training data and a process of generating metadata to indicate converge to 0 And a program for causing the computer to execute the process of regenerating the metadata.

  The second recording medium readable by the computer in one aspect of the present invention is a process of reading a predetermined block out of a plurality of blocks, which is data obtained by dividing training data representing matrix data, and storing it in a queue; The process of reading the predetermined block stored in the queue and performing the CD method iterative calculation, and when one component of the parameter converges to 0 during one iteration, it corresponds to the component converged to 0 A program for causing the computer to execute a process of transmitting a flag indicating that the train of training data to be removed can be removed.

  The data management method in one aspect of the present invention divides training data representing matrix data into a plurality of blocks, generates metadata indicating which column values of the original training data each block holds, When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. To do.

  In the data analysis method according to an aspect of the present invention, a predetermined block is read out from a plurality of blocks, which are data obtained by dividing training data representing matrix data, is stored in a queue, and the predetermined data stored in the queue is stored. When the block is read and the CD method is repeatedly calculated, and when some component of the parameter converges to 0 during one iteration, the train of training data corresponding to the component converged to 0 can be removed. Send a flag indicating.

  The analysis method according to an aspect of the present invention divides training data representing matrix data into a plurality of blocks, generates metadata indicating which column values of the original training data each block holds, and trains the training data. When some components of the parameters learned from the data converge to 0, the old block including the unnecessary column in each block is replaced with the block from which the unnecessary column is removed, and the metadata is regenerated. , Out of a plurality of blocks that are data obtained by dividing training data representing matrix data, a predetermined block is read and stored in a queue, the predetermined block stored in the queue is read, and the CD method is repeatedly calculated. When a component of a parameter converges to 0 during one iteration calculation, a train of training data corresponding to the component converged to 0 can be removed Transmitting a flag indicating and.

  The effect of the present invention is that the CD method can be used even in a situation where the size of training data exceeds the memory size of the computer.

It is a block diagram which shows the structure of the data management apparatus 101 in the 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the data management apparatus 101 in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the data analyzer 102 in the 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the data analyzer 102 in the 2nd Embodiment of this invention. It is a block diagram which shows the structure of the data analysis system 103 in the 3rd Embodiment of this invention. It is a block diagram which shows an example of the computer which implement | achieves the structure of the data analysis system 103 in the 3rd Embodiment of this invention. It is a figure which shows an example of the training data in the 3rd Embodiment of this invention, and its block division. It is a figure which shows an example of the metadata in the 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of blocking in the 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of queue management in the 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of iterative calculation in the 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of flag management in the 3rd Embodiment of this invention. It is a flowchart which shows the operation | movement of the reblocking in the 3rd Embodiment of this invention. It is a figure which shows an example of the new block and metadata produced | generated by reblocking in the 3rd Embodiment of this invention. It is a figure which shows the operation example of Coordinate Descent method.

<Embodiment 1>
Embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the data management apparatus 101 in the first embodiment of the present invention.

  A data management apparatus 101 according to the first embodiment of the present invention will be described with reference to FIG. Note that the drawing reference numerals attached to FIG. 1 are added to the respective elements for convenience as an example for facilitating understanding, and are not intended to limit the present invention.

  As shown in FIG. 1, the data management apparatus 101 according to the first embodiment of the present invention includes a blocking unit 20 and a reblocking unit 40. The blocking unit 20 divides training data represented by given matrix data (for example, a matrix of N rows and M columns represented by integers N and M) into a plurality of blocks, and each block is the original training data. Metadata that is information indicating how many rows and columns of values are held is generated. The reblocking unit 40 monitors parameters learned from the training data. The parameter is a component learned from the training data, and corresponds to a vector component of an objective function defined by the CD method, for example. The reblocking unit 40 uses an unnecessary column in each block when a component of the parameter (for example, a component wj of j dimension (j column of training data)) converges to 0 by the training processing of the training data. Replace the old block containing with the block from which unnecessary columns are removed. Here, the unnecessary column is, for example, a column corresponding to an axis converged to 0. A block from which unnecessary columns are removed is also referred to as an update block. Then, the reblocking unit 40 regenerates the above-described metadata (information indicating how many rows and columns of the original training data each block holds) for the updated block.

  Next, the operation of the data management apparatus 101 according to the first embodiment of the present invention will be described with reference to FIG.

  FIG. 2 is a flowchart showing the operation of the data management apparatus 101 in the first embodiment of the present invention. Note that the flowchart shown in FIG. 2 and the following description are examples of processing, and the processing order and the like may be changed or the processing may be returned or repeated depending on the processing that is appropriately obtained.

  As shown in FIG. 2, the blocking unit 20 divides training data representing given matrix data into a plurality of blocks, and how many rows and columns of each block hold values of the original training data. Metadata that is information to represent is generated (step S101). When the partial component of the parameter learned from the training data converges to 0, the reblocking unit 40 replaces an old block including an unnecessary column in each block with a block from which an unnecessary column is removed, Data is regenerated (step S102).

  The data management apparatus 101 according to the first embodiment of the present invention can use the CD method even in a situation where the size of training data exceeds the memory size of the data management apparatus or the computer. The reason is that by dividing the training data into blocks, the size of each block is reduced, and even when the training data is larger than the memory size, the processing related to the CD method is performed in units of blocks that can be processed by the data management device or the computer. This is because it can be done.

<Embodiment 2>
The configuration of the data analysis apparatus 102 according to the second embodiment for carrying out the present invention will be described with reference to the drawings. FIG. 3 is a block diagram showing a configuration of the data analysis apparatus 102 according to the second embodiment of the present invention.

  As illustrated in FIG. 3, the data analysis apparatus 102 according to the second exemplary embodiment of the present invention includes a queue management unit 90, an iterative calculation unit 110, and a flag management unit 100.

  The queue management unit 90 reads a predetermined block among a plurality of blocks that are data obtained by dividing the training data represented by the matrix data, and stores it in the queue. The iterative calculation unit 110 reads a predetermined block stored in the queue and performs a CD method iterative calculation (corresponding to learning in the first embodiment). When one component of a parameter converges to 0 during one iteration, the flag management unit 100 transmits a flag indicating that a column (for training data) corresponding to the component can be removed. .

  Next, the operation of the data analysis apparatus 102 according to the second embodiment of the present invention will be described with reference to FIG.

  FIG. 4 is a flowchart showing the operation of the data analysis apparatus 102 according to the second embodiment of the present invention. As shown in FIG. 4, the queue management unit 90 reads a predetermined block among a plurality of blocks that are data obtained by dividing the training data representing the given matrix data, and stores the predetermined block in the queue (step S201). The iterative calculation unit 110 reads a predetermined block stored in the queue and performs iterative calculation by the CD method (step S202). The flag management unit 100 transmits a flag indicating that the train of training data corresponding to some components can be removed when some components of the parameters converge to 0 during one iteration (step S203). ).

  The data analysis apparatus 102 according to the second embodiment of the present invention can use the CD method even in a situation where the size of the training data exceeds the memory size of the computer. This is because dividing the training data into blocks reduces the size of each block, and even when the training data is larger than the memory size, the process related to the CD method can be performed in units of blocks.

<Embodiment 3>
First, problems to be solved in the embodiment of the present invention will be clarified.

  In the behavior recognition apparatus using the CD method described in Patent Document 1, when the size of training data is larger than the memory size of the computer, the problem is that the training method cannot be read into the memory and the CD method cannot be applied (No. There is one problem). Due to recent developments in information technology, it has become easier to obtain huge training data that exceeds the memory size of the machine, so the number of cases where the processing related to the CD method cannot be executed because the training data cannot be stored in the memory has increased. It is coming.

  Furthermore, in the action recognition apparatus using the CD method described in Patent Document 1, the CD method has a problem (second problem) that it takes a long processing time because repeated calculation occurs many times. In the CD method, it is necessary to refer to each row of training data by one update. In particular, when facing the first problem, an out-of-core correspondence that reads and processes the training data as much as it can be read into the memory and reads the next one becomes essential as a countermeasure. At that time, data is frequently read, and the processing time is increased.

  Therefore, the data analysis system 103 according to the third embodiment for carrying out the present invention solves the first and second problems. The configuration and operation of the data analysis system 103 according to the third embodiment for carrying out the present invention will be described below.

  First, the configuration of the data analysis system 103 according to the third embodiment for carrying out the present invention will be described with reference to the drawings. FIG. 5 is a block diagram showing the configuration of the data analysis system 103 according to the third embodiment of the present invention.

  A data analysis system 103 according to the third embodiment of the present invention includes a data management device 1, a data analysis device 6, and a training data storage unit 12. The data management device 1, the data analysis device 6, and the training data storage unit 12 are communicably connected via a network 13 or a bus. Here, the training data storage unit 12 stores training data. Moreover, the training data storage unit 12 may store training data as a storage device outside the data analysis system 103, for example. In that case, the data analysis system 103 and its storage device are communicably connected via the network 13 or the like.

  The data management device 1 includes a blocking unit 2, a metadata storage unit 3, a reblocking unit 4, and a block storage unit 5. Here, the blocking unit 2 and the reblocking unit 4 have the same configurations and functions as the blocking unit 20 and the reblocking unit 40 included in the data management apparatus 101 in the first embodiment of the present invention described above. .

  The blocking unit 2 reads the training data stored (given) in the training data storage unit 12 and divides the training data into a plurality of blocks. Further, the blocking unit 2 stores the data of each divided block in the block storage unit 5. Further, the blocking unit 2 generates metadata indicating how many rows and columns in the original training data each block has, and stores the metadata in the metadata storage unit 3.

  The block storage unit 5 stores the data of each block of the divided training data. The metadata storage unit 3 stores the metadata generated by the blocking unit 2.

  The reblocking unit 4 replaces an old block including an unnecessary column in each block with a block from which the unnecessary column is removed when a part of parameters of a parameter learned from training data converges to 0. The above-mentioned metadata is regenerated for each block.

  The data analysis device 6 includes a parameter storage unit 7, a queue 8, a queue management unit 9, a flag management unit 10, and an iterative calculation unit 11. Here, the queue management unit 9, the iteration calculation unit 11, and the flag management unit 10 are the queue management unit 90, the iteration calculation unit 110, and the flag management included in the data analysis apparatus 102 in the second embodiment of the present invention described above. It has the same configuration and function as the unit 100.

  The parameter storage unit 7 stores variables to be updated such as parameters. The queue 8 stores blocks.

  The iterative calculation unit 11 reads out blocks or representative values necessary for the columns calculated by the iterative calculation unit 11 from the queue 8 and performs update calculation. The iterative calculation unit 11 reads a predetermined block stored in the queue 8 and performs iterative calculation by the CD method. The iterative calculation unit 11 determines whether or not each parameter component has converged to 0 for each iterative calculation. When there is a component wj that has converged to 0, the iterative calculation unit 11 calls the flag management unit 10 and transmits information indicating that the component wj has converged to 0.

  The queue management unit 9 discards unnecessary blocks from the queue 8 and acquires (for example, fetches) newly necessary blocks from the block storage unit 5. The flag management unit 10 receives information that the component wj has converged to 0 from the iterative calculation unit 11 and outputs an unnecessary column to the data management device 1.

  The computer which implement | achieves the data management apparatus 1 and the data analysis apparatus 6 which the data analysis system 103 in the 3rd Embodiment of this invention contains is demonstrated using FIG.

  FIG. 6 is a typical hardware configuration diagram of the data management device 1 and the data analysis device 6 included in the data analysis system 103 according to the third embodiment of the present invention. As shown in FIG. 6, the data management device 1 and the data analysis device 6 each include, for example, a CPU (Central Processing Unit) 21, a RAM (Random Access Memory) 22, and a storage device 23. In addition, the data management device 1 and the data analysis device 6 include, for example, a communication interface 24, an input device 25, and an output device 26, respectively.

  The blocking unit 2 and the reblocking unit 4 included in the data management device 1 and the queue management unit 9, the flag management unit 10 and the iterative calculation unit 11 included in the data analysis device 6 read the program into the RAM 22 and execute it. This is realized by the CPU 21. The metadata storage unit 3 and the block storage unit 5 included in the data management device 1 and the parameter storage unit 7 and the queue 8 included in the data analysis device 6 are, for example, a hard disk or a flash memory.

  The communication interface 24 is connected to the CPU 21 and is connected to a network or an external storage medium. External data may be taken into the CPU 21 via the communication interface 24. The input device 25 is, for example, a keyboard, a mouse, or a touch panel. The output device 26 is a display, for example. Note that the hardware configuration illustrated in FIG. 6 is merely an example, and each component of the data management device 1 and the data analysis device 6 may be configured by independent logic circuits.

  Next, the operation of the data analysis system 103 according to the third embodiment of the present invention will be described with reference to FIGS.

  FIG. 9 is a flowchart (flow chart) showing the operation of the blocking unit 2 in the third embodiment of the present invention. First, the blocking unit 2 acquires the size of the queue 8 of the data analysis device 6 (step S301). Next, the blocking unit 2 divides the training data into blocks of a size that can fit in the queue 8 (step S302). For example, the division may be performed in the row direction, in the column direction, or in both the matrix directions.

  Next, the blocking unit 2 generates information as to which value of the training data each block holds as metadata (step S303). Then, the blocking unit 2 stores the data of each block in the block storage unit 5, and stores the generated metadata in the metadata storage unit 3 (step S304).

  FIG. 10 is a flowchart showing the operation of the queue management unit 9 in the third embodiment of the present invention. First, the queue management unit 9 obtains a sequence (j1, j2,..., Jk) of columns to be processed from the iterative calculation unit 11 (step S401). Here, k is an integer of 1 or more. The order relationship of the numerical sequences of the columns to be processed may be descending or ascending order of the column numbers, may be random, or may be other order relationships. Next, the queue management unit 9 initializes the counter r with 1 (step S402). Here, the value of the counter r can be 1 to k. The queue management unit 9 refers to the metadata stored in the metadata storage unit 3 and identifies an unprocessed block including the jr-th column stored in the block storage unit 5 (step S403).

  Next, when the queue 8 is full (YES in step S404), the queue management unit 9 waits while periodically checking the queue 8 until it becomes available (step S405). When the queue 8 is empty (No in step S404), the queue management unit 9 reads the block from the block storage unit 5 and puts the block in the queue 8 (step S406). If there is another unprocessed block including the jr-th column (YES in step S407), the above process is repeated (return to step S403). When there is no other unprocessed block including the jr-th column (No in Step S407), the queue management unit 9 updates the value of the counter r (Step S408). For example, the queue management unit 9 adds 1 to the value of the counter r. When the process of the iterative calculation unit 11 ends (YES in step S409), the process of the queue management unit 9 ends. When the process of the iterative calculation unit 11 does not end (No in step S409), the above process is repeated until the process ends (return to step S404).

  FIG. 11 is a flowchart showing the operation of the iterative calculation unit 11 in the third embodiment of the present invention. First, the iterative calculation unit 11 determines the number of columns to be processed (j1, j2,...) And transmits it to the queue management unit 9 (step S501). The iterative calculation unit 11 initializes the counter r with 1 (step S502) and initializes the update difference Δ with 0 (step S503). Next, the iterative calculation unit 11 acquires a block including the jr-th column from the queue 8 (step S504), and updates the update difference Δ while reading the block line by line (step S505). Here, the update difference Δ is, for example, training data of N rows and M columns (N and M are natural numbers), i row and j columns (i is an integer of 1 to N, j is 1 to M) It is calculated by adding a product xij × g (w) of a value xij of an integer) and a function g (w) of w from the first line to the Nth line.

  If the update processing has not been completed for all the rows in the jr column of the block (No in step S506), the iterative calculation unit 11 performs the processing from step S504 to step S505 on all the rows in the jr column of the block. Is repeated (return to step S504).

  When the update processing has been completed for all the rows in the jr column of the block (YES in step S506), the iterative calculation unit 11 determines the jr-th (jr-th column) component wjr of the parameter w of the objective function f (w). Is updated to wjr + Δ (step S507). When the update difference Δ of the parameter w is smaller than the predetermined value (hereinafter, described as sufficiently small) (YES in step S508), the iterative calculation unit 11 ends the operation (step process). The predetermined value may be any value as long as the value indicates that the update difference Δ is sufficiently small, for example, 0.0001.

  When the update difference Δ of the parameter w is larger than the predetermined value (No in step S508), the iterative calculation unit 11 determines that there is still room for updating and determines whether the component wjr has converged to zero (step) S509). If wjr has converged to zero (YES in step S509), the iterative calculation unit 11 transmits to the flag management unit 10 that wjr has converged to zero (step S510). Next, the iterative calculation unit 11 updates the value of the counter r to r + 1 (step S511), and repeats the above until the update difference Δ is sufficiently small (return to step S503).

  Here, if the component wjr has not converged to zero (No in step S509), the iterative calculation unit 11 updates the value of the counter r to r + 1 (step S511) and repeats the above until the update difference Δ is sufficiently small. Repeat (return to step S503).

  FIG. 12 is a flowchart showing the operation of the flag management unit 10 in the third exemplary embodiment of the present invention. As shown in FIG. 12, the flag management unit 10 manages a snapshot of the number of non-zero components of the parameter w as a variable z (step S601). Then, the flag management unit 10 repeatedly receives the position of the component that has converged to zero (step S602), and determines whether the number of pieces of zero component position information received so far is equal to or greater than z / 2 (step S603). . If the number of pieces of zero component position information is greater than or equal to z / 2 (YES in step S603), the flag management unit 10 sends the position information of the component wjr converged to zero to the reblocking unit 4 and the reblocking information. An instruction is transmitted (step S604). When the process of the iterative calculation unit 11 ends (YES in step S605), the process of the flag management unit 10 ends.

  If the process of the iterative calculation unit 11 does not end (No in step S605), the flag management unit 10 repeats the above process until the process ends (returns to step S601). If the number of pieces of zero component position information is less than z / 2 (No in step S603), the flag management unit 10 proceeds to the process of step S605. Further, the denominator of z / 2 is not necessarily 2 and may be parameterized so that the user can specify an arbitrary integer.

  FIG. 13 is a flowchart showing the operation of the reblocking unit 4 in the third embodiment of the present invention. As illustrated in FIG. 13, the reblocking unit 4 obtains a reblocking command from the flag management unit 10 and position information of components that have converged to zero in the parameter w (step S701). Next, the reblocking unit 4 reconstructs the blocks by connecting adjacent blocks while excluding the column corresponding to the component that has converged to zero, in a size range that fits in the queue 8 sufficiently. Replace with the old block in the block storage unit 5 (step S702). For example, the reblocking unit 4 reconstructs a block by replacing adjacent blocks with each other while excluding a column corresponding to a component that has converged to zero, and replaces the old block. Then, the reblocking unit 4 generates metadata corresponding to the reconfigured block, and replaces the old metadata in the metadata storage unit 3 (step S703). Thus, the operation of the reblocking unit 4 is completed.

  Next, detailed operations in the data analysis apparatus 6 for carrying out the invention of the present application will be described.

  First, an example of an operation performed by the blocking unit 2 of the data management device 1 will be described with reference to FIG. FIG. 7 is a diagram illustrating an example of training data and its block division according to the third embodiment of the present invention.

  The 8-by-8 matrix shown in FIG. 7 is an example of training data. For example, it is assumed that the queue 8 of the data analysis device 6 has only a half data size of training data. The blocking unit 2 divides the training data into blocks of an appropriate size so that the maximum size of the block is equal to or smaller than the size of the queue 8. Here, as an example, the training data is divided into two equal parts in the row and column directions, and a block that is divided into four equal parts as a whole is generated.

  As shown in FIG. 7, dotted lines described in a matrix of 8 rows and 8 columns represent block boundary lines. Each of the four equally divided blocks is referred to as blocks 1, 2, 3, and 4. In block 1, for example, the data in the row x 1 is “0.36 0.26 0.00 0.00”, and the data in the row x 2 is “0.00 0.00 0.91 0.00”. The data in the row x3 of block 1 is “0.01 0.00 0.00 0.00”, and the data in the row x4 is “0.00 0.00 0.09 0.00”.

  Here, the method of dividing the block is not limited to this example. For example, only the row or column direction may be divided, the blocks may be divided so as to have different sizes, or the rows and columns may be rearranged by an arbitrary method in advance.

  The blocking unit 2 calculates block metadata simultaneously with block division. FIG. 8 is a diagram illustrating an example of metadata in the third exemplary embodiment of the present invention. FIG. 8 shows, for example, the metadata of the four blocks in FIG. That is, each row of metadata indicates to which block each column of training data is distributed. As shown in FIG. 8, the first row of metadata indicates that, for example, the value corresponding to the first column in the training data is distributed to blocks 1 and 2.

  Here, the format of the metadata is not limited to this example, and any format can be used as long as the information of which block the training data value belongs to is included.

  Next, a specific operation example regarding reblocking will be described with reference to FIGS. 7 and 14.

  The data analysis device 6 optimizes the parameter w by the iterative calculation unit 11 while sequentially reading out the blocks to the queue 8. Here, when the initial value of the parameter w is randomly determined as (1, 10, 2, 3, 4, 8, 3) and optimization is started, for example, the number of non-zero components managed by the flag management unit 10 z is 8. If the iterative calculation unit 11 determines that the component of the second column of the parameter w has converged to zero after several iterations, the flag management unit 10 stores the position information of the second column. Assume that iterative calculation further proceeds and it is determined that the third, fourth, and sixth columns have converged to zero. Similarly, the flag management unit 10 also stores position information of the third, fourth, and sixth columns. Furthermore, since the number of components equal to or greater than z / 2 has converged to zero, the flag management unit 10 reblocks the reblocking unit 4 of the data management device 1 together with the position information (2, 3, 4, 6). Send instructions.

  Upon receiving the instruction, the reblocking unit 4 excludes the column of the position information (2, 3, 4, 6) sent to the block in the block storage unit 5 and is sufficiently large to fit in the queue 8. Re-block so that

  FIG. 14 is a diagram illustrating an example of new blocks and metadata generated by reblocking according to the third embodiment of the present invention. FIG. 14 shows an example in which the four blocks shown in FIG. 7 are reblocked based on position information (2, 3, 4, 6). In this case, two blocks from which the second, third, fourth and sixth columns are excluded are generated and replaced with the old block (FIG. 7) in the block storage unit 5. Then, as shown in FIG. 14, new metadata (right diagram in FIG. 14) is generated from the new block (left diagram in FIG. 14).

  By excluding unnecessary columns from the block, the ratio of the blocks read to the queue 8 out of all the blocks is increased, and there is an advantage that necessary information can be easily put on the buffer or the cache.

  As described above, in the data analysis system 103 according to the third embodiment of the present invention, the blocking unit 2 of the data management device 1 reads the training data stored in the training data storage unit 12 and divides the training data into blocks. And stored in the block storage unit 5. Further, the blocking unit 2 generates metadata indicating how many rows and columns in the original training data each block has, and stores the metadata in the metadata storage unit 3. Based on the position information of the parameter component that has converged to zero during the iterative calculation, the reblocking unit 4 reconstructs the block and replaces the old block so as to exclude the train data column corresponding to that position. Hold.

  The data analysis device 6 includes a parameter storage unit 7, a queue 8, a queue management unit 9, a flag management unit 10, and an iterative calculation unit 11. The parameter storage unit 7 stores variables to be updated such as parameters. The queue 8 stores blocks. The iterative calculation unit 11 reads out blocks or representative values necessary for the columns calculated by the iterative calculation unit 11 from the queue 8 and performs update calculation. The iterative calculation unit 11 reads a predetermined block stored in the queue 8 and performs iterative calculation by the CD method. The queue management unit 9 discards unnecessary blocks from the queue 8 and acquires newly required blocks from the block storage unit 5. The flag management unit 10 receives information indicating that the component wj has converged to 0 from the iterative calculation unit 11 and outputs an unnecessary column to the data management device 1. Therefore, the data analysis system 103 can use the CD method even in a situation where the size of the training data exceeds the memory size of the computer, and can shorten the processing time of the CD method under the situation.

  The reason is as follows. That is, by dividing the training data into blocks and performing the processing in units of blocks, the CD method can be executed even when the training data does not fit in the memory. Also, some components of the parameter often converge to 0 during the iterative calculation by optimization. The parameter component that has converged to 0 does not change in subsequent iterations. That is, it is not necessary to read the data string corresponding to the component. Therefore, by removing the data strings that do not need to be read by reblocking, more necessary data strings can be read at a time and the calculation is shortened.

  Here, in order to explain specifically the mechanism by which the calculation is shortened, consider the CD method using training data shown in FIG. The training data is read from the secondary storage device into the main storage device and processed. However, the computer assumes that, for example, only half of the training data can be read into main memory at a time due to capacity issues. At this time, as one of the countermeasures, a method of reading the training data into the main memory every four lines and processing can be considered. That is, in order to update the component wj for the column j, the first to fourth lines are read and processed, and then the fifth to eighth lines are read and processed. In this case, two IOs are generated. Assuming that update calculation is performed in order from the first column to the eighth column in one repetitive calculation, 16 IOs are generated. If the first, second, third, and fourth components of the parameter w converge to 0 when the calculation is repeated 50 times, and the parameter w is optimized when the calculation is repeated 100 times, the IO is 2 × 8 × in total. 50 + 2 × 4 × 50 = 1200 occurrences.

  At this time, when it is repeated 50 times, the first to fourth columns in the training data are never referred to again. The reason is as follows. That is, as described above, in the calculation for the column j in the CD method, the component wj of the parameter w is updated to wj + α · d. Here, d is the moving direction of the starting point in FIG. 15, and α is the moving width (step width). α · d is a value obtained by summing the value xij of the i-th row and j-th column of the training data and the function g (w) of w with respect to the row i of the product xij × g (w). The value in the column is used only for updating wj.

  Therefore, if the training data on the secondary storage device is replaced with training data from which the first to fourth columns are removed, the data size is halved. For this reason, in the repetitive processing from the 51st time to the 100th time, it is only necessary to read the replaced data once. In this case, IO occurs 2 × 8 × 50 + 1 × 4 × 50 = 1000 times in total, and the number of IOs is reduced as compared with the case where no replacement is performed.

  Thereby, the effect of shortening the entire processing time can be obtained.

  Although the present invention has been described above using the embodiment, the present invention is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  A part or all of the above-described embodiment can be described as in the following supplementary notes, but is not limited thereto.

[Appendix 1]
A blocking unit that divides training data representing matrix data into a plurality of blocks, and generates metadata indicating which column values of the original training data each block holds;
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. And a re-blocking unit.

[Appendix 2]
The reblocking unit connects adjacent blocks of the blocks while excluding a column corresponding to the component converged to 0 among columns included in the block, and re-blocks the blocks. The data management device according to Supplementary Note 1, which is configured.

[Appendix 3]
A metadata storage unit for storing the metadata;
The data management apparatus according to appendix 2, wherein the reblocking unit generates metadata corresponding to the reconstructed block, and updates the metadata stored in the metadata storage unit.

[Appendix 4]
Divide the training data representing the matrix data into multiple blocks, generate metadata that indicates which column values in the original training data each block holds,
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. Data management method.

[Appendix 5]
A process of dividing the training data representing the matrix data into a plurality of blocks, and generating metadata indicating which column values of the original training data each block holds;
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. A program that causes a computer to execute

[Appendix 6]
A queue management unit that reads a predetermined block out of a plurality of blocks that are data obtained by dividing training data representing matrix data, and stores it in a queue;
An iterative calculation unit that reads the predetermined block stored in the queue and performs an iterative calculation of a CD method;
A flag management unit that transmits a flag indicating that a train of training data corresponding to the component that has converged to 0 can be removed when some component of the parameter has converged to 0 during one iteration calculation. Data analysis device.

[Appendix 7]
The iterative calculation unit determines whether or not each component of the parameter has converged to 0 for each one iterative calculation, and if it is determined that there is a component that has converged to 0, the flag management unit The data analysis device according to appendix 6, which notifies the component that has converged to 0.

[Appendix 8]
The repetitive calculation unit further updates the component when the update difference of the updated component is larger than a predetermined threshold when at least one component included in the predetermined block is updated. Or the data analysis device according to 7;

[Appendix 9]
The queue management unit according to any one of appendices 6 to 8, wherein the queue management unit discards a block that becomes unnecessary as a result of the repeated calculation of the CD method from the queue and stores a newly required block in the queue. Data analysis equipment.

[Appendix 10]
The queue management unit identifies, among the plurality of blocks, a block in which the iterative calculation unit has not performed the CD method repeated calculation, and reads the identified block as the predetermined block. The data analysis device according to any one of claims.

[Appendix 11]
The flag management unit receives information on the component that has converged to 0 among the components of the parameter from the iterative calculation unit, and can remove a train of training data corresponding to the component that has converged to 0 The data analysis device according to any one of appendices 6 to 10, which transmits a flag to indicate.

[Appendix 12]
The flag management unit determines whether or not the number of components that have converged to 0 among the components of the parameter is equal to or greater than a predetermined number. The data analysis device according to any one of appendices 6 to 11, which requests reblocking.

[Appendix 13]
Among a plurality of blocks that are data obtained by dividing training data representing matrix data, a predetermined block is read and stored in a queue,
Read the predetermined block stored in the queue and perform repeated calculation of the CD method,
A data analysis method for transmitting a flag indicating that a train of training data corresponding to a component converged to 0 can be removed when some component of the parameter converges to 0 during one iterative calculation.

[Appendix 14]
Among a plurality of blocks that are data obtained by dividing training data representing matrix data, a process of reading a predetermined block and storing it in a queue;
A process of reading the predetermined block stored in the queue and performing an iterative calculation of a CD method;
When a part of the parameters of the parameter converges to 0 during one iteration, a process is executed to send to the computer a flag indicating that the train of training data corresponding to the component converged to 0 can be removed. Program to make.

[Appendix 15]
A blocking unit that divides training data representing matrix data into a plurality of blocks, and generates metadata indicating which column values of the original training data each block holds;
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. A data management device comprising:
A queue management unit that reads a predetermined block out of a plurality of blocks that are data obtained by dividing training data representing matrix data, and stores it in a queue;
An iterative calculation unit that reads the predetermined block stored in the queue and performs an iterative calculation of a CD method;
A flag management unit that transmits a flag indicating that a train of training data corresponding to the component that has converged to 0 can be removed when some component of the parameter has converged to 0 during one iteration calculation. And a data analysis system including the data analysis device.

[Appendix 16]
The reblocking unit connects adjacent blocks of the blocks while excluding a column corresponding to the component converged to 0 among columns included in the block, and re-blocks the blocks. The data analysis system according to Supplementary Note 15, which is configured.

[Appendix 17]
A metadata storage unit for storing the metadata;
The data analysis system according to appendix 16, wherein the reblocking unit generates metadata corresponding to the reconstructed block, and updates the metadata stored in the metadata storage unit.

[Appendix 18]
The iterative calculation unit determines whether or not each component of the parameter has converged to 0 for each one iterative calculation, and if it is determined that there is a component that has converged to 0, the flag management unit The data analysis system according to appendix 15, which notifies the component that has converged to zero.

[Appendix 19]
The repetitive calculation unit further updates the component when the update difference of the updated component is larger than a predetermined threshold when at least one component included in the predetermined block is updated. Or the data analysis system of 16.

[Appendix 20]
18. The queue management unit according to any one of appendices 15 to 17, wherein the queue management unit discards a block that is no longer necessary as a result of the repetition calculation of the CD method from the queue and stores a newly required block in the queue. Data analysis system.

[Appendix 21]
The queue management unit identifies a block in which the iterative calculation unit has not performed the repetition calculation of the CD method among the plurality of blocks, and reads the identified block as the predetermined block. The data analysis system according to any one of claims.

[Appendix 22]
The flag management unit receives information on a component that has converged to 0 out of each component of the parameter from the iterative calculation unit, and indicates that a train of training data corresponding to the component that has converged to 0 can be removed. 20. The data analysis system according to any one of appendices 15 to 19, which transmits a flag.

[Appendix 23]
The flag management unit determines whether or not the number of components that have converged to 0 among the components of the parameter is equal to or greater than a predetermined number. 21. The data analysis system according to any one of appendices 15 to 20, which requests reblocking.

[Appendix 24]
Divide the training data representing the matrix data into multiple blocks, generate metadata that indicates which column values in the original training data each block holds,
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. And
Among a plurality of blocks that are data obtained by dividing training data representing matrix data, a predetermined block is read and stored in a queue,
Read the predetermined block stored in the queue and perform repeated calculation of the CD method,
An analysis method of transmitting a flag indicating that a train of training data corresponding to a component converged to 0 can be removed when a part of the components of the parameter converges to 0 during one iteration calculation.

[Appendix 25]
Divide the training data representing matrix data into multiple blocks, generate metadata that shows which column values of each training block each block holds, and some of the parameters learned from the training data When the component of 収束 converges to 0, an old block including an unnecessary column in each block is replaced with a block from which the unnecessary column is removed, and a process for regenerating the metadata and training for representing matrix data A process of reading a predetermined block out of a plurality of blocks, which are data obtained by dividing the data, and storing it in a queue; a process of reading the predetermined block stored in the queue and performing an iterative calculation of a CD method; When some component of the parameter converges to 0 during one iteration, it is possible to remove the train of training data corresponding to the component converged to 0. Program for executing a process of transmitting, to the computer a flag indicating.

  This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2014-028454 for which it applied on February 18, 2014, and takes in those the indications of all here.

DESCRIPTION OF SYMBOLS 1 Data management apparatus 2 Blocking part 3 Metadata storage part 4 Reblocking part 5 Block storage part 6 Data analyzer 7 Parameter storage part 8 Queue 9 Queue management part 10 Flag management part 11 Iterative calculation part 12 Training data storage part 13 Network 20 Blocking unit 21 CPU
22 RAM
DESCRIPTION OF SYMBOLS 23 Memory | storage device 24 Communication interface 25 Input device 26 Output device 40 Reblocking part 90 Queue management part 100 Flag management part 101 Data management apparatus 102 Data analysis apparatus 103 Data analysis system 110 Iterative calculation part

Claims (10)

Blocking means for dividing training data representing matrix data into a plurality of blocks, and generating metadata indicating which column values of the original training data each block holds,
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. And a re-blocking means.   The reblocking means connects adjacent blocks of the blocks while excluding the column corresponding to the component converged to 0 among the columns included in the block, and reblocks the blocks. The data management apparatus according to claim 1, which is configured. Queue management means for reading a predetermined block out of a plurality of blocks, which is data obtained by dividing training data representing matrix data, and storing it in a queue;
Repetitive calculation means for reading the predetermined block stored in the queue and performing repetitive calculation of a CD method;
Flag management means for transmitting a flag indicating that a train of training data corresponding to the component that has converged to 0 can be removed when some component of the parameter converges to 0 during one iteration calculation. Data analysis device.   The iterative calculation means determines whether or not each component of the parameter has converged to 0 for each one of the iteration calculations. If it is determined that there is a component converged to 0, the flag management means The data analysis apparatus according to claim 3, wherein a component that has converged to 0 is notified. Blocking means for dividing training data representing matrix data into a plurality of blocks, and generating metadata indicating which column values of the original training data each block holds,
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. A data management device comprising:
Queue management means for reading a predetermined block out of a plurality of blocks, which is data obtained by dividing training data representing matrix data, and storing it in a queue;
Repetitive calculation means for reading the predetermined block stored in the queue and performing repetitive calculation of a CD method;
Flag management means for transmitting a flag indicating that a train of training data corresponding to the component that has converged to 0 can be removed when some component of the parameter converges to 0 during one iteration calculation. And a data analysis system including the data analysis device. A process of dividing the training data representing the matrix data into a plurality of blocks, and generating metadata indicating which column values of the original training data each block holds;
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. And a computer-readable recording medium storing a program for causing the computer to execute the process. Among a plurality of blocks that are data obtained by dividing training data representing matrix data, a process of reading a predetermined block and storing it in a queue;
A process of reading the predetermined block stored in the queue and performing an iterative calculation of a CD method;
When a part of the parameters of the parameter converges to 0 during one iteration, a process is executed to send to the computer a flag indicating that the train of training data corresponding to the component converged to 0 can be removed. A computer-readable recording medium storing a program to be executed. Divide the training data representing the matrix data into multiple blocks, generate metadata that indicates which column values in the original training data each block holds,
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. Data management method. Among a plurality of blocks that are data obtained by dividing training data representing matrix data, a predetermined block is read and stored in a queue,
Read the predetermined block stored in the queue and perform repeated calculation of the CD method,
A data analysis method for transmitting a flag indicating that a train of training data corresponding to a component converged to 0 can be removed when some component of the parameter converges to 0 during one iterative calculation. Divide the training data representing the matrix data into multiple blocks, generate metadata that indicates which column values in the original training data each block holds,
When some components of the parameters learned from the training data converge to 0, the old blocks including unnecessary columns in each block are replaced with blocks from which unnecessary columns are removed, and the metadata is regenerated. And
Among a plurality of blocks that are data obtained by dividing training data representing matrix data, a predetermined block is read and stored in a queue,
Read the predetermined block stored in the queue and perform repeated calculation of the CD method,
An analysis method of transmitting a flag indicating that a train of training data corresponding to a component converged to 0 can be removed when a part of the components of the parameter converges to 0 during one iteration calculation.

Images