CN118014098A - Machine learning training data scheduling method and equipment - Google Patents
Machine learning training data scheduling method and equipment Download PDFInfo
- Publication number
- CN118014098A CN118014098A CN202410155756.7A CN202410155756A CN118014098A CN 118014098 A CN118014098 A CN 118014098A CN 202410155756 A CN202410155756 A CN 202410155756A CN 118014098 A CN118014098 A CN 118014098A
- Authority
- CN
- China
- Prior art keywords
- module
- data
- machine learning
- adaptation
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000010801 machine learning Methods 0.000 title claims abstract description 143
- 238000012549 training Methods 0.000 title claims abstract description 111
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000006978 adaptation Effects 0.000 claims abstract description 77
- 238000006243 chemical reaction Methods 0.000 claims abstract description 33
- 238000012545 processing Methods 0.000 claims description 32
- 230000008569 process Effects 0.000 claims description 16
- 230000009466 transformation Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 4
- 230000003044 adaptive effect Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000001960 triggered effect Effects 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000015556 catabolic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000005477 standard model Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/11—File system administration, e.g. details of archiving or snapshots
- G06F16/116—Details of conversion of file system types or formats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application provides a machine learning training data scheduling method and equipment, wherein the method comprises the following steps: when a target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering a universal module layer to start through the grafting module; determining a target disk file according to the directed acyclic graph in the universal module layer; forming a machine learning training module subgraph according to the adaptation modules from the target disk file to all the directed paths of the grafting module; and converting the original training data in the target disk file into data required by a target machine learning algorithm based on the machine learning training module subgraph. According to the technical scheme, the training data conversion efficiency of the machine learning algorithm can be improved.
Description
Technical Field
The present application relates to the field of machine learning, and in particular, to a method and apparatus for scheduling training data for machine learning.
Background
With the development of the big data age, mining potentially useful knowledge from massive data has become a general need in various big industry fields. Machine learning is an artificial intelligence sub-field which is vigorously developed in the background of big data age, and the core problem to be solved is to find knowledge from data and store the learned knowledge into a predictive model or an explanatory model for subsequent application, wherein the predictive model and the explanatory model are collectively called as a model. The flow of machine learning is generally divided into two phases: the first stage is a learning stage, also called training stage, which inputs data and outputs a model; the second stage is an application stage (also called a predictive model for a predictive stage) that applies the model to new data to obtain an output of the model. The application stage of machine learning is relatively efficient, and generally, data can be processed piece by piece to obtain a prediction or interpretation result corresponding to each piece of data. The training phase of machine learning is a performance bottleneck of the total flow of machine learning, and a reliable and effective model can be obtained by considering all training data and through multiple non-sequential accesses of all data.
In order to ensure the effect of the training phase, most machine learning algorithms require that all training data be put into memory for processing. When the training data cannot be put into the memory as a whole, the machine learning algorithm can manage the data exceeding the memory capacity by using the disk of the operating system as a virtual memory of the memory. The method involves disk access, and requires the conversion of the original disk file storing all training data into data required by the machine learning algorithm, and the conversion process requires higher disk data processing technology of the computer equipment. However, the conversion process of the disk data is slow at present, even the phenomenon of operating system breakdown occasionally occurs in the processing process, and the conversion efficiency of the training data of machine learning is low.
Disclosure of Invention
The application provides a machine learning training data scheduling method and equipment, which are used for solving the technical problem that the data conversion efficiency of computer equipment is lower when training data conversion is performed by a machine learning algorithm.
In a first aspect, a machine learning training data scheduling method is provided, applied to a computer device, the method comprising: when a target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering a universal module layer to start through the grafting module; determining a target disk file according to the directed acyclic graph in the universal module layer; forming a machine learning training module subgraph according to the adaptation modules in all directed paths from the target disk file to the grafting module; and converting the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.
In a second aspect, there is provided a computer device comprising a memory, a processor connected to the processor for executing one or more computer programs stored in the memory, implementing the method of the first aspect.
The application can realize the following technical effects: when the target machine learning algorithm is detected to start training, a grafting module corresponding to the target machine learning algorithm is obtained, the grafting module is used for triggering the starting of the universal module layer, then a target disk file which needs to be used by the target machine learning algorithm is determined according to a directed graph in the universal module layer, a machine learning training module sub-graph is formed according to the target disk file and all the adaptation modules of the directed paths of the grafting module, and original training data in the target disk file are converted into data required by the target machine learning algorithm based on the machine learning training module sub-graph. According to the method, the training data conversion processing is realized through the preset universal module layer, so that a common computer can efficiently process massive training data stored in the disk file, the training stage of the machine learning algorithm can be implemented aiming at the massive training data, and the training data conversion efficiency of the machine learning is improved.
Drawings
FIG. 1 is a flow chart of a prior art training data conversion related to a machine learning algorithm;
fig. 2 is a flow chart of a machine learning training data scheduling method according to an embodiment of the present application;
FIG. 3 is an exemplary diagram of a generic module layer provided by an embodiment of the present application;
fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application.
The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be noted that, if not in conflict, the features of the embodiments of the present application may be combined with each other, which is within the protection scope of the present application. In addition, while functional block division is performed in a device diagram and logical order is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart. Furthermore, the words "first," "second," "third," and the like as used herein do not limit the order of data and execution, but merely distinguish between identical or similar items that have substantially the same function and effect.
To facilitate an understanding of the present application, a machine learning algorithm will first be described. Machine learning is an artificial intelligence sub-domain that is vigorously developed in the context of the big data age, and at the heart of "parse data using algorithms, learn from it, and then interpret or predict new data. That is, machine learning is to discover knowledge from data and save the learned knowledge to a model for later use. The flow of machine learning is generally divided into two phases: the first stage is a learning stage, also called training stage, which inputs data and outputs a model; the second stage is an application stage (also called a predictive model for a predictive stage) that applies the model to new data to obtain an output of the model. The application stage of machine learning is relatively efficient, and generally, data can be processed piece by piece to obtain a prediction or interpretation result corresponding to each piece of data. The training phase of machine learning is a performance bottleneck of the total flow of machine learning, and a reliable and effective model can be obtained by considering all training data and through multiple non-sequential accesses of all data. In order to ensure the effect of the training phase, most machine learning algorithms require that all training data be put into the memory for processing, so the machine learning algorithm has a large demand for the memory.
Aiming at the problem of overlarge memory requirement of the machine learning algorithm, two common schemes are adopted, one scheme is to upgrade hardware, and the high-efficiency implementation of the training stage of the machine learning algorithm is ensured by adding a memory bank and setting a larger available memory; the other is to distribute the training data to a plurality of machine nodes with limited memory capacity and unable to hold all the training data for processing through the deployment of the distributed system, and ensure that the machine learning algorithm trains as usual in a plurality of nodes through an efficient data communication mechanism. However, both the above solutions involve hardware enhancement, or adding memory, or adding computing devices and network communication devices, which cannot solve the problem on the premise of maintaining the original machine hardware facilities, and the use cost is high.
The inventor of the present application found in intensive studies of the prior art that there are two schemes in the related art that can realize conversion of machine learning training data without changing hardware facilities of computer devices. Specifically, one scheme is to upgrade through a machine learning algorithm, change the algorithm based on memory into a data stream algorithm supporting disk input, an incremental learning algorithm or an iterative learning algorithm, and the algorithm flows all need to sequentially access training data; the other scheme is based on approximate learning of random sampling, a training subset suitable for the current memory capacity is obtained by random sampling from all training data before training a model, learning is carried out based on the training subset, and finally an approximate model close to a standard model obtained through the learning of all training data is obtained. Both of these solutions are non-hardware dependent solutions, do not require changes to the machine hardware facilities, and are suitable for more application scenarios. Both of these solutions involve disk access, however, the performance bottleneck is the transfer flow from the original disk file storing all training data to the sequential access data stream required by the machine learning algorithm or no more than the current sample set of memory capacity, as shown in fig. 1. The data conversion flow involves view conversion of disk file data, such as adding or deleting certain columns or filtering certain rows, and requires efficient view conversion based on disks, and cannot load all data into memory for processing. Therefore, both of the above solutions require the introduction of efficient disk data processing techniques to implement the data conversion flow shown in fig. 1, so that the effectiveness and practicality of the solution as a whole can be ensured.
In view of the above, the present application provides a machine learning training data scheduling method and apparatus. The data conversion framework of the machine learning algorithm can be built through a preset universal module layer, the original training data file is efficiently converted into data required by the training stage of the machine learning algorithm, and the training data conversion efficiency of the machine learning is improved.
The basic idea of the application is to design the data conversion flow in fig. 1 as a generic module layer, which is formed by an adaptation module as a node, which delivers data objects via configured input/output ports, and from which the flow paths form a directed acyclic graph. When the machine learning algorithm to be called starts training, the grafting module connected with the machine learning algorithm initiates the execution of the universal module layer to form a machine learning training module subgraph. And for the adaptation module nodes in the machine learning training module subgraph, processing the adaptation modules layer by layer according to the original layering top-down, and executing all adaptation modules with disk files or memory data output ports in the layer simultaneously when the adaptation modules in each layer are processed. When the current layer adaptation module is executed, the upper layer adaptation modules connected with the file pointers are driven to be executed together, and the upper layer adaptation modules are further driven to sequentially access the input disk files. The scheme can effectively solve the problem of overlarge memory requirement of the machine learning algorithm, and can ensure that a common computer can effectively process massive training data stored in a disk file without adding or modifying computer hardware facilities, so that the training stage of the machine learning algorithm can be implemented aiming at the massive training data, and the training data conversion efficiency of computer equipment is improved.
The machine learning training data scheduling method shown in the present application will be specifically described below. Referring to fig. 2, a flow chart of a machine learning training data scheduling method according to the present application is shown, and the machine learning training data scheduling method can be applied to a computer device, and the method shown in fig. 2 includes:
And S201, when the target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering the universal module layer to start through the grafting module.
It should be noted that, the computer device may store training data required by a plurality of different machine learning algorithms, where the target machine learning algorithm is a machine learning algorithm that needs to initiate a training data conversion and is detected by the computer device.
It should be further noted that the generic module layer is a virtual module layer preset in advance for machine learning algorithm training data conversion. In one embodiment, the generic module layer includes at least two adaptation modules, wherein one of the adaptation modules is a grafting module corresponding to the target machine learning algorithm; the adaptation module is configured as nodes of the universal module layer according to a hierarchy and is used for transmitting data objects, and the flow paths of the data objects form a directed acyclic graph in the universal module layer, wherein the data objects comprise any one or more of disk files, file pointers and memory data.
In one embodiment, the adaptation module is provided with at least one input port and at least one output port, and the adaptation module receives the data object output by the node of the upper layer through the input port and transmits the converted data object to the node of the lower layer through the output port.
In one embodiment, the adaptation module includes any one or more types of data view generation module, data view column transformation module, data view row transformation module, data view batch processing module.
The data objects input by the data view generation module are disk files, and the data objects output by the data view generation module are file pointers for sequentially accessing the disk files. The data view column transformation module comprises a functional module for adding columns, deleting columns and changing column values in the data view; the data object input by the data view column conversion module is a file pointer, and the data object output by the data view column conversion module is a file pointer. The data view line transformation module comprises a functional module for randomly sampling lines in a data view, filtering the lines under a specified condition, filtering repeated lines and orderly combining the data lines in a plurality of data views; the data object input by the data view line transformation module is any one or more of a disk file, a file pointer and memory data, and the data object output by the data view line transformation module is any one or more of a disk file, a file pointer and memory data.
The data view batch processing module comprises batch processing with fixed line number and batch processing module with fixed sequence number, wherein the fixed sequence refers to a data line set with the same ID; the data objects input by the data view batch processing module are file pointers, the data objects output by the data view batch processing module are memory data of batches, the memory data of batches are used by an internal process, and the internal process is initiated by the target machine learning algorithm.
In one embodiment, the top node of the directed acyclic graph is the data view generation module, the bottom node of the directed acyclic graph is another type of module other than the data view generation module, and the middle node of the directed acyclic graph is another type of module other than the data view batch processing module.
As a possible embodiment, the computer system can be designed in advance as a generic module layer for training data conversion of the machine learning algorithm, which generic module layer is formed by the adaptation modules as nodes. Each of the adaptive module nodes may have a plurality of input ports, each of which receives data objects such as disk files, file pointers, or memory data, and each of the nodes may also have a plurality of output ports, each of which outputs data objects such as disk files, file pointers, or memory data. Wherein, at most one output port of the disk file is provided, at most one output port of the memory data is provided, and the disk file can be represented by a file path. The nodes of the adaptation module can pass data objects through input and output ports, and a directed acyclic graph (DAG graph) is formed by the flow routes of the data objects.
For example, referring to fig. 3, an exemplary diagram of a generic module layer is provided in the present application. The generic module layer shown in fig. 3 includes 3 to-be-tuned machine learning algorithms (it should be appreciated that in other embodiments, any other number of to-be-tuned machine learning algorithms may be included) and 7 adaptation modules. The 7 adaptation modules are configured in a layered manner, the adaptation module No.2 is positioned on the top layer, the adaptation modules No. 7 to No. 10 are positioned on the middle layer, and the adaptation modules No. 15 and No. 17 are positioned on the bottom layer; the No.2 adaptation module can be a data view generation module, the No. 8 adaptation module and the No. 10 adaptation module can be data view column conversion modules, the No. 7 adaptation module, the No. 9 adaptation module and the No. 15 adaptation module can be data view row conversion modules, the No. 17 adaptation module can be a data view batch processing module, the grafting module of the No. 16 machine learning algorithm is the No. 9 adaptation module, the grafting module of the No. 20 machine learning algorithm is the No. 15 adaptation module, and the grafting module of the No. 21 machine learning algorithm is the No. 17 adaptation module.
In one embodiment, the grafting module of the target machine learning algorithm is the data view batch processing module or the adapting module of the output data object as memory data.
For example, the batch processing module inputs the file pointers and outputs batches of memory data for use by internal processes initiated by the machine learning algorithm to be invoked, in which case the grafting module of the machine learning algorithm may be defined as the corresponding data view batch processing module. The output port of the memory data may be connected to a machine learning algorithm to be invoked, in which case the grafting module of the machine learning algorithm may be defined as an adaptation module that outputs the memory data.
For example, referring to fig. 3 as well, assuming that the 16 # machine learning algorithm to be tuned needs to start training, the computer device may take the 16 # machine learning algorithm to be tuned as the target machine learning algorithm to obtain a grafting module, that is, a 9 # adaptation module, connected to the 16 # machine learning algorithm, and trigger the start execution of the whole generic module layer through the grafting module.
S202, determining a target disk file according to the directed acyclic graph in the universal module layer.
It should be noted that, the target disk file refers to a disk file of the grafting module corresponding to the target machine learning algorithm, which can be reached through a directed path in the directed acyclic graph.
As a possible implementation manner, when the target machine learning algorithm starts training, the grafting module corresponding to the target machine learning algorithm may initiate execution of a generic module layer, where flow paths of various data objects in the generic module layer form the directed acyclic graph, and the computer device may determine, as a target disk file, a disk file capable of reaching the grafting module corresponding to the target machine learning algorithm through a directed path in the directed acyclic graph, where the target disk file may be one or multiple, and the application is not limited in any way.
In one embodiment, the target disk file is represented in a data view in the form of a two-dimensional table.
S203, forming a machine learning training module subgraph according to the adaptation modules in all the directed paths from the target disk file to the grafting module.
For example, when the 16 # machine learning algorithm is the target machine learning algorithm, the disk file involved is only 1 # so that the machine learning training module subgraph thereof is composed of {1,2,6,9,13,16} adaptation modules; similarly, when the No. 20 machine learning algorithm is the target machine learning algorithm, the related disk files have No. 1, no. 3, no. 4 and No. 11, so that the machine learning training module subgraph thereof is composed of {1,2,3,4,5,7,8,11,12,15,18} node sets.
S204, converting the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.
As a possible implementation, the conversion process from the disk file storing all training data to the input data required by the machine learning algorithm may all be performed in the machine learning training module subgraph of the target machine learning algorithm.
In one embodiment, the step S204 includes: and triggering and starting the adaptation module layer by layer from top to bottom by taking the adaptation module corresponding to the target disk file as a top node according to the level of the adaptation module in the machine learning training module subgraph so as to convert the original training data of the target disk file into the data required by the target machine learning algorithm.
In one embodiment, the method further includes, when the current layer of adaptation module is started, if the input data object of the started adaptation module is the file pointer, driving an upper layer adaptation module connected with the file pointer to be started; and concurrently executing the linkage adaptation module in the current layer, wherein the linkage adaptation module is an adaptation module of which the data object of the output port is a disk file or memory data.
For example, in the generic module layer example shown in fig. 3, the top-down hierarchical structure of the adaptation module in the machine learning training module subgraph of the machine learning algorithm No. 16 is {2} and {9}; similarly, the top-down hierarchies of adaptation modules in the machine learning training module subgraph of the machine learning algorithm No. 20 are {2}, {7,8} and {15}. When executing the subgraph of the machine learning training module, the No. 16 machine learning algorithm firstly processes {2}, and because the output of the No. 2 adapting module in the subgraph is not a disk file or memory data, the {2} is skipped to directly process {9}, the output of the No. 9 adapting module is memory data, and therefore the execution is triggered, and the upper-layer No. 2 adapting module connected with the file pointer is driven to execute together, the No. 1 disk file is sequentially accessed, and No. 13 memory data is generated. Similarly, when executing the subgraph of the machine learning training module, the 20 # machine learning algorithm firstly processes {2}, and because the output of the 2 # adaptive module in the subgraph is a disk file, the execution is triggered, the 1 # disk file is sequentially accessed, and the 4 # disk file is generated; then processing {7,8}, because the output of the No. 7 adaptation module in the subgraph is a disk file, and the output of the No. 8 adaptation module in the subgraph is not a disk file or memory data, the jump-out of the No. 8 adaptation module only triggers the execution of the No. 7 adaptation module; the No. 7 adapting module does not have an upper adapting module connected with a file pointer, so that the upper adapting module is not driven to operate, and only the No. 3 disk file and the No. 4 disk file are sequentially accessed to generate the No. 11 disk file; and finally, processing {15}, wherein the output of the No. 15 adapting module in the subgraph is memory data, so that the execution is triggered, and the upper layer No. 8 adapting module and the No. 2 adapting module connected with the file pointer are sequentially driven to be executed together, and sequentially access the No. 1 disk file and the No. 11 disk file to generate No. 18 memory data.
It should be noted that, in the worst case of the above general module layer, the adaptation module of each layer needs to trigger execution, and access all the disk files of the upper layer at most sequentially during execution, so that the general module layer can reduce the times of sequentially accessing the original training data file as much as possible, and through sub-drawing of the machine learning training module and concurrent execution of the adaptation modules of each layer, the original training data file is efficiently converted into the sequential access data stream required to be input in the training stage of the machine learning algorithm or the training sample set not exceeding the upper limit of the current memory capacity.
Therefore, the machine learning training data scheduling method designs the conversion process from the original disk file storing all training data to the input data required by the machine learning algorithm into a universal module layer so as to maximize the conversion efficiency on the premise of not changing the hardware facilities of the computer equipment and minimizing the memory occupation. When a target machine learning algorithm to be called starts training, an adaptation module grafted by the machine learning algorithm initiates execution of a universal module layer, and related disk files are disk files which can reach the grafting module through a directed path in a directed acyclic graph, and nodes from the disk files to all directed paths of the grafting module form a machine learning training module subgraph; in the process of data conversion, the adaptive module nodes in the subgraph of the machine learning training module implement the same-layer parallel computing flow, namely, the adaptive modules are processed layer by layer from top to bottom according to the original layering, all the adaptive modules with disk files or memory data output ports in the layers are concurrently executed when the adaptive modules in each layer are processed, the adaptive modules without outputting the disk files or memory data are not processed, the upper-layer adaptive modules connected with file pointers are driven to be executed together when the adaptive modules in the current layer are executed, the adaptive modules in the upper layers are further driven to sequentially access the input disk files, and the output disk files or memory data can be generated when all the disk files in the upper layers are sequentially accessed at most, so that the efficiency of training data conversion is effectively improved.
The method of the application is described above and the execution device of the application is described below.
Referring to fig. 4, fig. 4 is a schematic structural diagram of a computer device according to an embodiment of the present application, where the computer device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example by a bus, to the processor 401.
The processor 401 is configured to support the computer device 40 to perform the corresponding functions in the method embodiments described above. The processor 401 may be a central processing unit (central processing unit, CPU), a graphics processor (Graphics Processing Unit, GPU) or a hardware chip. The hardware chip may be an Application Specific Integrated Circuit (ASIC), a programmable logic device (programmable logic device, PLD), or a combination thereof. The PLD may be a complex programmable logic device (complex programmable logic device, CPLD), a field-programmable gate array (FPGA) GATE ARRAY, general-purpose array logic (GENERIC ARRAY logic, GAL), or any combination thereof.
The memory 402 is used for storing program codes and the like. Memory 402 may include Volatile Memory (VM), such as random access memory (random access memory, RAM); the memory 402 may also include a non-volatile memory (NVM), such as read-only memory (ROM), flash memory (flash memory), hard disk (HARD DISK DRIVE, HDD), or solid state disk (solid-STATE DRIVE, SSD); memory 402 may also include a combination of the above types of memory.
The processor 401 may call the program code to perform the following operations:
when a target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering a universal module layer to start through the grafting module;
determining a target disk file according to the directed acyclic graph in the universal module layer;
forming a machine learning training module subgraph according to the adaptation modules in all directed paths from the target disk file to the grafting module;
And converting the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.
The embodiment of the application also provides a computer device, which comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used for executing one or more computer programs stored in the memory, so as to realize the method according to the previous embodiment.
Those skilled in the art will appreciate that implementing all or part of the above-described methods in the embodiments may be accomplished by computer programs stored in a computer-readable storage medium, which when executed, may include the steps of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only memory (ROM), a random-access memory (Random Access memory, RAM), or the like.
The foregoing disclosure is illustrative of the present application and is not to be construed as limiting the scope of the application, which is defined by the appended claims.
Claims (10)
1. A machine learning training data scheduling method, for application to a computer device, the method comprising:
when a target machine learning algorithm starts training, acquiring a grafting module corresponding to the target machine learning algorithm, and triggering a universal module layer to start through the grafting module;
determining a target disk file according to the directed acyclic graph in the universal module layer;
forming a machine learning training module subgraph according to the adaptation modules in all directed paths from the target disk file to the grafting module;
And converting the original training data in the target disk file into data required by the target machine learning algorithm based on the machine learning training module subgraph.
2. The method of claim 1, wherein the generic module layer comprises at least two adaptation modules, wherein one adaptation module is a grafting module corresponding to the target machine learning algorithm;
the adaptation module is configured as nodes of the universal module layer according to a hierarchy and is used for transmitting data objects, and the flow paths of the data objects form a directed acyclic graph in the universal module layer, wherein the data objects comprise any one or more of disk files, file pointers and memory data.
3. The method of claim 2, wherein the adaptation module is provided with at least one input port and at least one output port, and wherein the adaptation module receives the data object output by the node of the previous layer through the input port and passes the converted data object to the node of the next layer through the output port.
4. The method of claim 1, further characterized in that the adaptation module comprises any one or more of a data view generation module, a data view column transformation module, a data view row transformation module, a data view batch processing module;
The data objects input by the data view generation module are disk files, and the data objects output by the data view generation module are file pointers for sequentially accessing the disk files;
The data view column transformation module comprises a functional module for adding columns, deleting columns and changing column values in the data view; the data object input by the data view column conversion module is a file pointer, and the data object output by the data view column conversion module is a file pointer;
The data view line transformation module comprises a functional module for randomly sampling lines in a data view, filtering the lines under a specified condition, filtering repeated lines and orderly combining the data lines in a plurality of data views; the data objects input by the data view line transformation module are any one or more of disk files, file pointers and memory data, and the data objects output by the data view line transformation module are any one or more of disk files, file pointers and memory data;
The data view batch processing module comprises batch processing with fixed line number and batch processing module with fixed sequence number, wherein the fixed sequence refers to a data line set with the same ID; the data objects input by the data view batch processing module are file pointers, the data objects output by the data view batch processing module are memory data of batches, the memory data of batches are used by an internal process, and the internal process is initiated by the target machine learning algorithm.
5. The method of claim 4, wherein a top level node of the directed acyclic graph is the data view generation module, a bottom level node of the directed acyclic graph is another type of module other than the data view generation module, and an intermediate node of the directed acyclic graph is another type of module other than the data view batch module.
6. The method of claim 1, further characterized in that the grafting module of the target machine learning algorithm is the data view batch processing module or an adaptation module of the output data object as memory data.
7. The method of claim 1, further characterized in that the target disk file is represented in a data view in the form of a two-dimensional table; the adaptation module in all directed paths from the target disk file to the grafting module forms a machine learning training module subgraph, which comprises:
The adaptation module of the target disk file in all the directed paths of the grafting module is obtained;
And constructing the machine learning training module subgraph according to the original hierarchical structure of the adaptation module in the universal module layer.
8. The method of claim 1, further characterized in that the converting the raw training data in the target disk file to data required by the target machine learning algorithm based on the machine learning training module subgraph comprises:
And triggering and starting the adaptation module layer by layer from top to bottom by taking the adaptation module corresponding to the target disk file as a top node according to the level of the adaptation module in the machine learning training module subgraph so as to convert the original training data of the target disk file into the data required by the target machine learning algorithm.
9. The method of claim 8, wherein the method further comprises:
When an adaptation module of a current layer is started, if an input data object of the started adaptation module is the file pointer, driving an upper layer adaptation module connected with the file pointer to be started;
and concurrently executing the linkage adaptation module in the current layer, wherein the linkage adaptation module is an adaptation module of which the data object of the output port is a disk file or memory data.
10. A computer device comprising a memory, a processor connected to the processor for executing one or more computer programs stored in the memory, which processor, when executing the one or more computer programs, causes the computer device to implement the method of any of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410155756.7A CN118014098B (en) | 2024-02-04 | 2024-02-04 | Machine learning training data scheduling method and equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410155756.7A CN118014098B (en) | 2024-02-04 | 2024-02-04 | Machine learning training data scheduling method and equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118014098A true CN118014098A (en) | 2024-05-10 |
| CN118014098B CN118014098B (en) | 2024-09-13 |
Family
ID=90942383
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410155756.7A Active CN118014098B (en) | 2024-02-04 | 2024-02-04 | Machine learning training data scheduling method and equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118014098B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807520A (en) * | 2020-01-08 | 2020-02-18 | 成都四方伟业软件股份有限公司 | Method and device for analyzing influence factors of neural nodes of convolutional neural network |
| JP2021026766A (en) * | 2019-07-31 | 2021-02-22 | 株式会社リコー | Processing method and device for deep learning task, and computer-readable recording medium |
| CN112966272A (en) * | 2021-03-31 | 2021-06-15 | 国网河南省电力公司电力科学研究院 | Internet of things Android malicious software detection method based on countermeasure network |
| CA3077006A1 (en) * | 2020-03-25 | 2021-09-25 | The Toronto-Dominion Bank | System and method for automatically managing storage resources of a big data platform |
| CN114169531A (en) * | 2021-11-12 | 2022-03-11 | 国电南瑞科技股份有限公司 | Prediction method and system for configuration machine learning modeling task description |
| CN114372579A (en) * | 2021-12-30 | 2022-04-19 | 胜斗士(上海)科技技术发展有限公司 | Method of training machine learning model, prediction method, computing device, and medium |
| CN116541155A (en) * | 2022-01-21 | 2023-08-04 | 中国石油化工股份有限公司 | Exploration and development cloud resource intelligent scheduling method based on machine learning |
| WO2023224430A1 (en) * | 2022-05-19 | 2023-11-23 | Samsung Electronics Co., Ltd. | Method and apparatus for on-device personalised analysis using a machine learning model |
| CN117251725A (en) * | 2023-06-27 | 2023-12-19 | 深圳市绿联科技股份有限公司 | Method and device for identifying data based on machine learning |
| CN117314680A (en) * | 2023-09-27 | 2023-12-29 | 北京邮电大学 | Power dispatching monitoring data anomaly detection method based on double-path self-encoder |
-
2024
- 2024-02-04 CN CN202410155756.7A patent/CN118014098B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2021026766A (en) * | 2019-07-31 | 2021-02-22 | 株式会社リコー | Processing method and device for deep learning task, and computer-readable recording medium |
| CN110807520A (en) * | 2020-01-08 | 2020-02-18 | 成都四方伟业软件股份有限公司 | Method and device for analyzing influence factors of neural nodes of convolutional neural network |
| CA3077006A1 (en) * | 2020-03-25 | 2021-09-25 | The Toronto-Dominion Bank | System and method for automatically managing storage resources of a big data platform |
| CN112966272A (en) * | 2021-03-31 | 2021-06-15 | 国网河南省电力公司电力科学研究院 | Internet of things Android malicious software detection method based on countermeasure network |
| CN114169531A (en) * | 2021-11-12 | 2022-03-11 | 国电南瑞科技股份有限公司 | Prediction method and system for configuration machine learning modeling task description |
| CN114372579A (en) * | 2021-12-30 | 2022-04-19 | 胜斗士(上海)科技技术发展有限公司 | Method of training machine learning model, prediction method, computing device, and medium |
| CN116541155A (en) * | 2022-01-21 | 2023-08-04 | 中国石油化工股份有限公司 | Exploration and development cloud resource intelligent scheduling method based on machine learning |
| WO2023224430A1 (en) * | 2022-05-19 | 2023-11-23 | Samsung Electronics Co., Ltd. | Method and apparatus for on-device personalised analysis using a machine learning model |
| CN117251725A (en) * | 2023-06-27 | 2023-12-19 | 深圳市绿联科技股份有限公司 | Method and device for identifying data based on machine learning |
| CN117314680A (en) * | 2023-09-27 | 2023-12-29 | 北京邮电大学 | Power dispatching monitoring data anomaly detection method based on double-path self-encoder |
Non-Patent Citations (2)
| Title |
|---|
| JINTING REN ET AL: "Archivist: A Machine Learning Assisted Data Placement Mechanism for Hybrid Storage Systems", 《 2019 IEEE 37TH INTERNATIONAL CONFERENCE ON COMPUTER DESIGN》, 10 February 2020 (2020-02-10), pages 676 - 679 * |
| 王冬青: "提升Web服务响应的Cache调度技术的研究与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, 15 March 2017 (2017-03-15), pages 137 - 259 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118014098B (en) | 2024-09-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7168772B2 (en) | Neural network search method, device, processor, electronic device, storage medium and computer program | |
| CN109993293B (en) | Deep learning accelerator suitable for heap hourglass network | |
| Yang et al. | AUTO-PRUNE: Automated DNN pruning and mapping for ReRAM-based accelerator | |
| US20200090051A1 (en) | Optimization problem operation method and apparatus | |
| JP2021072107A (en) | Storage system and method for hierarchical sort acceleration near storage | |
| JP2021507345A (en) | Fusion of sparse kernels to approximate the complete kernel of convolutional neural networks | |
| Roy et al. | Algorithmic techniques for solving graph problems on the automata processor | |
| CN113283575A (en) | Processor for reconstructing artificial neural network, operation method thereof and electrical equipment | |
| US11429299B2 (en) | System and method for managing conversion of low-locality data into high-locality data | |
| CN118014098B (en) | Machine learning training data scheduling method and equipment | |
| KR102674475B1 (en) | Electronic device for executing a pluraliry of neural networks | |
| CN113128688B (en) | General AI parallel reasoning acceleration structure and reasoning equipment | |
| EP2439652B1 (en) | Reordering binary decision diagrams for circuit design | |
| EP2439666B1 (en) | Parallel sifting algorithm | |
| CN113095024A (en) | Regional parallel loading device and method for tensor data | |
| US11442643B2 (en) | System and method for efficiently converting low-locality data into high-locality data | |
| US8374991B2 (en) | Determining optimum variable orders for BDDs using recursion | |
| US8370289B2 (en) | BDD variable reordering using parallel permutation | |
| CN110766150A (en) | Regional parallel data loading device and method in deep convolutional neural network hardware accelerator | |
| CN115129233B (en) | Data processing device, method and related product | |
| CN118171710B (en) | NPU acceleration method for sparse matrix multiplication | |
| US8583590B2 (en) | Parallel window algorithm | |
| US8595179B2 (en) | Determining optimum variable orders for BDDs using pair-wise variable grouping | |
| CN116700995B (en) | Concurrent access method, device, equipment and storage medium for heterogeneous memory pool | |
| Ahmad et al. | Low-Depth Parallel Algorithms for the Binary-Forking Model without Atomics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |