CN117573328B - Parallel task rapid processing method and system based on multi-model driving - Google Patents
Parallel task rapid processing method and system based on multi-model driving Download PDFInfo
- Publication number
- CN117573328B CN117573328B CN202410055306.0A CN202410055306A CN117573328B CN 117573328 B CN117573328 B CN 117573328B CN 202410055306 A CN202410055306 A CN 202410055306A CN 117573328 B CN117573328 B CN 117573328B
- Authority
- CN
- China
- Prior art keywords
- driving
- module
- task
- data
- parallel
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 82
- 238000000034 method Methods 0.000 claims abstract description 78
- 238000012546 transfer Methods 0.000 claims abstract description 63
- 230000002159 abnormal effect Effects 0.000 claims abstract description 52
- 230000005540 biological transmission Effects 0.000 claims abstract description 22
- 230000008569 process Effects 0.000 claims description 49
- 238000004364 calculation method Methods 0.000 claims description 48
- 230000005764 inhibitory process Effects 0.000 claims description 37
- 230000035772 mutation Effects 0.000 claims description 22
- 238000013507 mapping Methods 0.000 claims description 21
- 238000004806 packaging method and process Methods 0.000 claims description 20
- 238000012360 testing method Methods 0.000 claims description 13
- 230000004927 fusion Effects 0.000 claims description 12
- 238000000354 decomposition reaction Methods 0.000 claims description 10
- 238000004891 communication Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 9
- 230000006978 adaptation Effects 0.000 claims description 2
- 230000006870 function Effects 0.000 description 19
- 230000001629 suppression Effects 0.000 description 16
- 238000012549 training Methods 0.000 description 13
- 238000006243 chemical reaction Methods 0.000 description 7
- 238000013528 artificial neural network Methods 0.000 description 6
- 230000004044 response Effects 0.000 description 5
- 238000012858 packaging process Methods 0.000 description 4
- 230000005856 abnormality Effects 0.000 description 3
- 238000013499 data model Methods 0.000 description 3
- 230000001934 delay Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 238000005056 compaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000009193 crawling Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000007619 statistical method Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/505—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the load
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/481—Exception handling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/48—Indexing scheme relating to G06F9/48
- G06F2209/483—Multiproc
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/508—Monitor
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multi Processors (AREA)
Abstract
The invention provides a parallel task rapid processing method and a system based on multi-model driving, which relate to the technical field of data processing, and the method comprises the following steps: according to the parallel task set, a plurality of data sources output by the data source module for receiving the data of each data source are respectively packaged with a plurality of driving modules according to a plurality of driving models to obtain a plurality of driving modules for data source matching, a plurality of driving load indexes are output, an abnormal driving module is obtained to conduct fitness optimizing on the plurality of driving modules, an optimizing driving module is output, a data transmission channel of the optimizing driving module and the abnormal driving module is established to obtain a transfer task queue, the transfer task queue is processed based on the optimizing driving module, a processing return result of the transfer task queue is obtained and transmitted to the abnormal driving module, the technical problem that in the prior art, management and control on multi-model driving parallel task processing is lacked, the parallel task processing efficiency is low is solved, reasonable and accurate management and control parallel task processing is achieved, and the parallel task processing efficiency is improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a parallel task rapid processing method and system based on multi-model driving.
Background
With the development of scientific technology, particularly the development of the parallel task processing field, in the artificial intelligence field, the use of parallel computing technology can greatly improve the computing efficiency, so that the training and reasoning speed and the accuracy of the model are faster, the accuracy is higher, and the dynamic allocation of computing resources and task scheduling are the keys for realizing efficient utilization of resources. However, in the prior art, there is a technical problem that the parallel task processing efficiency is low due to lack of control over multi-model driving parallel task processing.
Disclosure of Invention
The application provides a method and a system for rapidly processing parallel tasks based on multi-model driving, which are used for solving the technical problem of low processing efficiency of the parallel tasks caused by lack of management and control on processing of the multi-model driving parallel tasks in the prior art.
In view of the above problems, the present application provides a method and a system for fast processing parallel tasks based on multi-model driving.
In a first aspect, the present application provides a method for fast processing parallel tasks based on multi-model driving, the method comprising: packaging according to the driving models respectively to obtain a plurality of driving modules, wherein the driving modules are in communication connection with the data source module; receiving data of each data source according to the data source module, and outputting a plurality of data sources; receiving a parallel task set, carrying out data source matching on the plurality of data sources and the plurality of driving modules according to the parallel task set to obtain corresponding driving load indexes of each driving module under task driving based on the plurality of matched data sources, and outputting a plurality of driving load indexes; acquiring an abnormal driving module according to the plurality of driving load indexes; performing fitness optimization on the plurality of driving modules according to the abnormal driving module, and outputting an optimized driving module; establishing a data transmission channel of the optimizing driving module and the abnormal driving module, and acquiring a transfer task queue according to the data transmission channel; processing the transfer task queue based on the optimizing driving module, and obtaining a processing return result of the transfer task queue; and transmitting a processing return result of the transfer task queue to the abnormal driving module.
In a second aspect, the present application provides a multi-model driven based parallel task fast processing system, the system comprising: the packaging module is used for respectively packaging according to the plurality of driving models to obtain a plurality of driving modules, wherein the plurality of driving modules are in communication connection with the data source module; the first receiving module is used for receiving the data of each data source according to the data source module and outputting a plurality of data sources; the first matching module is used for receiving a parallel task set, matching the data sources with the driving modules according to the parallel task set to obtain driving load indexes corresponding to each driving module under task driving based on the matched data sources, and outputting the driving load indexes; the abnormal module is used for acquiring an abnormal driving module according to the plurality of driving load indexes; the first optimizing module is used for optimizing the adaptation degree of the driving modules according to the abnormal driving module and outputting an optimizing driving module; the channel establishment module is used for establishing a data transmission channel of the optimizing driving module and the abnormal driving module and acquiring a transfer task queue according to the data transmission channel; the result acquisition module is used for processing the transfer task queue based on the optimizing driving module and acquiring a processing return result of the transfer task queue; and the first transmission module is used for transmitting the processing return result of the transfer task queue to the abnormal driving module.
One or more technical solutions provided in the present application have at least the following technical effects or advantages: the multi-model-drive-based parallel task rapid processing method and system relate to the technical field of data processing, solve the technical problem that the parallel task processing efficiency is low due to the lack of control over multi-model-drive parallel task processing in the prior art, realize reasonable and accurate control over parallel task processing, and improve the parallel task processing efficiency.
Drawings
FIG. 1 is a schematic flow diagram of a method for rapidly processing parallel tasks based on multi-model driving;
fig. 2 is a schematic structural diagram of a parallel task fast processing system based on multi-model driving.
Reference numerals illustrate: the system comprises a packaging module 1, a first receiving module 2, a first matching module 3, an abnormality module 4, a first optimizing module 5, a channel establishing module 6, a result obtaining module 7 and a first transmission module 8.
Detailed Description
The method and the system for rapidly processing the parallel tasks based on the multi-model drive are used for solving the technical problem that the processing efficiency of the parallel tasks is low due to the lack of control over the processing of the multi-model drive parallel tasks in the prior art.
Example 1
As shown in fig. 1, an embodiment of the present application provides a method for fast processing parallel tasks based on multi-model driving, where the method includes:
step A100: packaging according to the driving models respectively to obtain a plurality of driving modules, wherein the driving modules are in communication connection with the data source module;
in the application, the method for rapidly processing the parallel task based on the multi-model driver is applied to a rapid processing system of the parallel task based on the multi-model driver, in order to better rapidly process the parallel task in the later stage, data are required to be respectively packaged based on a plurality of driving modules, namely, the plurality of driving modules are in data transmission through entity classes, namely, the entity classes serve as data models, the attributes of the entity class package and the attributes of the form have a one-to-one correspondence relationship, the form data can be directly packaged into the entity class object by using the plurality of driving models to package, the specific steps can be that firstly a ModelDrive interface is realized through Action, secondly, the entity class object, namely, the data model object is created in the Action, finally, the getModel method of the interface is rewritten, the data model object is returned, a plurality of driving modules are acquired on the basis of the basis, the plurality of driving modules are in communication connection with a data source module, and the multi-model module is used for providing data required by executing the task to each driving module in the plurality of driving modules, and the rapid processing based on the model is realized in the later stage, and the rapid processing is based on important.
Step A200: receiving data of each data source according to the data source module, and outputting a plurality of data sources;
in the application, the data in each data source is received through the data source module in communication connection with the plurality of driving modules, the data in each data source has a one-to-one correspondence with the data required to be subjected to task processing in each driving module, and the data sources in the data source module are subjected to clustering analysis according to the data type required by each driving module, namely the data sources in the data source module are divided into a plurality of classes consisting of similar objects, and the classes are output as a plurality of data sources corresponding to the plurality of data, so that the purpose of performing parallel task rapid processing based on multi-model driving is ensured.
Step A300: receiving a parallel task set, carrying out data source matching on the plurality of data sources and the plurality of driving modules according to the parallel task set to obtain corresponding driving load indexes of each driving module under task driving based on the plurality of matched data sources, and outputting a plurality of driving load indexes;
further, step a300 of the present application further includes:
step a310: acquiring a task process queue of each driving model;
step A320: performing calculation complexity identification according to the task process queue to obtain module calculation complexity;
step a330: acquiring the equalization calculation complexity corresponding to each driving model, wherein the equalization calculation complexity is based on the calculation complexity of the driving module during load equalization;
step A340: and outputting the driving load index by utilizing the index of the module calculation complexity accounting for the balance calculation complexity.
Further, step a300 of the present application further includes:
step A350: receiving the parallel task set, and calculating the inhibition factors corresponding to the plurality of driving modules according to the parallel task set, wherein the inhibition factors corresponding to each driving module comprise the inhibition factors of the data dependency, the inhibition factors of the inherent delay of task decomposition and the inhibition factors of the encapsulated model in the module for use case test;
step A360: acquiring a plurality of task inhibition indexes of the plurality of driving modules according to the inhibition factors corresponding to the plurality of driving modules;
step a370: carrying out time sequence fusion on the task inhibition indexes to obtain parallel inhibition indexes;
step A380: and obtaining a parallel task decomposition result by taking the parallel inhibition index minimization as a target.
In the application, in order to improve the processing efficiency of parallel tasks in a multi-model driver, firstly, processing tasks in each driving module in the multi-model driver are received, a parallel task set is generated, and a plurality of data sources are matched with a plurality of driving modules according to the parallel task set, which means that task process queues of each driving model are recorded and acquired first, and task process queues are waiting queues for executing tasks, namely queues for storing tasks. A task process queue is a special linear table, and can use FIFO, i.e. first-in first-out, principle, in which new tasks are always inserted at the end of the queue, and when tasks are read, the tasks are always read from the head of the queue. And releasing a task from the queue after reading one task, and further, performing calculation complexity identification according to the expression of the task process queue through calculation complexity identification, wherein the expression of the calculation complexity identification is as follows:
；
wherein t (n) is the computational complexity, n is the total number of processes in the corresponding task process queue, k is the number of processes in the corresponding task process queue for single parallel processing, [0, k ]]Is processed in parallel by a driving model connected with a corresponding task process queue, [ k+1, n ]]Requiring a sequential processing of the driving model,for the computational complexity of each process,is the number of processes sequentially executed in the task process queue.
When executing tasks according to the task process queue, the task processing queue can be divided into synchronous execution or asynchronous execution, wherein synchronous execution refers to adding tasks to a designated queue synchronously, waiting until the execution of the added tasks is finished, and continuing to execute until the tasks in the queue are finished, namely blocking threads. The task can only be executed in the current thread, and the current thread is not necessarily the main thread, the capacity of starting a new thread is not provided, and asynchronous execution means that the thread can return immediately, and the following task can be continuously executed without waiting, so that the current thread is not blocked. The method can execute tasks in a new thread, has the capability of starting the new thread (the new thread is not necessarily started), executes tasks in sub-threads asynchronously if the new thread is not added to a main queue, acquires the computation complexity of each module in a plurality of driving modules through an expression of computation complexity identification based on synchronous execution tasks or asynchronous execution tasks, acquires the equilibrium computation complexity corresponding to each driving model on the basis, wherein the equilibrium computation complexity is the computation complexity based on the driving modules in load balancing, the load balancing of the driving modules refers to the fact that the tasks in the driving modules can be shared on a plurality of operation units for execution, such as a Web server, an FTP server, an enterprise key application server, other key task servers and the like, so that the work tasks are completed jointly.
Further, by receiving a parallel task set, calculating suppression factors corresponding to a plurality of driving modules according to the parallel task set, wherein the suppression factors refer to reasons for poor parallel efficiency caused by the fact that the parallel task is not efficient in the plurality of driving modules, the suppression factors corresponding to each driving module comprise suppression factors of data dependency relationships and suppression factors of inherent delays of task decomposition, and suppression factors of a model packaged in the modules for use case tests, the suppression factors of the data dependency relationships refer to reasons for low parallel task efficiency caused by the fact that the generation or the use of some data items in one data set depend on the relation of other data items, the suppression factors of inherent delays of task decomposition refer to reasons for delays existing when the tasks are decomposed according to inherent structures and sequences of the suppression factors for controlling cost and coordinating scheduling progress, the suppression factors of the model packaged in the modules for use case tests refer to a group of documents designed and written for the model test packaged in the execution module at first, and the suppression factors mainly comprise contents such as test input, execution conditions, expected results and the like. The test case is an important basis for executing the test, has the characteristics of effectiveness, repeatability, easiness in organization, clearness, simplicity, maintainability and the like, and is used for carrying out descending sequential processing on the parallel inhibition indexes according to the inhibition factors corresponding to the driving modules and the point position marks for carrying out task inhibition on the tasks to be executed, and is further used as a plurality of task inhibition indexes of the driving modules, and simultaneously, the task inhibition indexes are subjected to sequential fusion according to inhibition time information corresponding to the task inhibition indexes, so that future trends of task inhibition are well predicted to obtain the parallel inhibition indexes, finally, the parallel inhibition indexes in the last order are subjected to descending sequential processing, the parallel inhibition indexes in the last order are used as the minimization of the parallel inhibition indexes, the parallel task decomposition results in the driving modules are obtained as targets, and a quick processing foundation for carrying out parallel task compaction based on multi-model driving is realized.
Step A400: acquiring an abnormal driving module according to the plurality of driving load indexes;
further, step a400 of the present application further includes:
step A410: identifying the data of the data sources according to the data source module, and obtaining a plurality of mutation probabilities corresponding to the data sources, wherein the mutation probabilities are mutation probabilities of the received data amount in each data source in unit time;
step a420: matching the plurality of driving modules according to the plurality of variation probabilities to obtain N variation probabilities corresponding to each driving module;
step a430: fusing the N variation probabilities corresponding to each driving module, and outputting the fused variation probability corresponding to each driving module;
step a440: and optimizing the plurality of driving load indexes according to the fusion variation probability corresponding to each driving module, and outputting the optimized plurality of driving load indexes.
In the application, load evaluation is sequentially performed on the loads of the plurality of driving modules in the output plurality of driving load indexes, the driving load indexes are correspondingly matched with the driving modules, meanwhile, whether the actual loads in the driving modules are larger than or equal to the matched driving load indexes is judged, if the actual loads are smaller than the matched driving load indexes, the loads in the current driving module are regarded as balanced states, if the loads are larger than or equal to the balanced states, the loads in the current driving module are regarded as overlarge, and the loads are marked as abnormal driving modules for output.
Further, because the situation that the data source suddenly increases in the parallel task processing process, the corresponding data amount increases, on the basis of the situation, the mutation probability of the data source needs to be calculated, the mutation probability is that the mutation probability of the driving load index is automatically adjusted, firstly, the data in the data sources are identified according to the data source modules which are in communication connection with the driving modules, namely, the mutation probability corresponding to the data sources is formed after the difference or fluctuation between each data value and the average value in each group of data in the data sources is recorded and stored, the mutation probability is the mutation probability corresponding to the received data amount in unit time in each data source, further, the mutation probability is used as the basic reference data and the mutation probability corresponding to the driving modules, N mutation probabilities corresponding to each driving module are determined, the mutation probabilities corresponding to the driving modules are fused according to the mutation probability corresponding to the driving modules, the mutation probability is integrated together, the mutation probability corresponding to the driving modules is used as the fusion probability corresponding to each driving module, the fusion probability is output according to the driving module, the load is improved by the load of more than 10% after the driving modules are compared with the driving index, the load is optimized, the load is not optimized, the load is more than 80% is achieved, the realization of the rapid processing of parallel tasks based on multi-model driving has a limited effect.
Step A500: performing fitness optimization on the plurality of driving modules according to the abnormal driving module, and outputting an optimized driving module;
further, step a500 of the present application further includes:
step A510: acquiring the packaging environment configuration information of the abnormal driving module, the module running processor parameters, the module assembly function similarity and the module receiving data source type;
step A520: performing fitness optimization on the plurality of driving modules respectively according to the packaging environment configuration information, module operation processor parameters, module assembly function similarity and module receiving data source types to obtain a plurality of driving fitness;
step a530: and outputting the optimizing driving module according to the driving fitness, wherein the driving fitness of the optimizing driving module is highest in the rest driving modules.
Further, step a530 of the present application includes:
step A531: acquiring real-time driving load indexes corresponding to each driving module in the plurality of driving modules, and positioning the driving load indexes of the abnormal driving modules;
step a532: establishing a transfer mapping network according to the driving load index of the abnormal driving module and the driving load index of the residual driving module;
step A533: performing fitness optimization in the residual driving modules according to the transfer mapping network to obtain a plurality of transfer fitness;
step A534: and carrying out weighted calculation according to the plurality of transfer fitness and the plurality of driving fitness, and outputting the optimizing driving module.
In the application, in order to more accurately remove the abnormal driving module, so that the driving modules are required to be adaptively optimized according to the driving load in each abnormal driving module, the packaging environment configuration information of the abnormal driving module, the module operation processor parameters, the module assembly function similarity and the module receiving data source type are firstly obtained, the packaging environment configuration information is the data environment configured in the packaging process of the driving modules, the module operation processor parameters are the task processing parameters of the driving modules in the operation process, the module assembly function similarity is the data matching degree between the driving modules in the packaging process of the task data, the module receiving data source type is set according to the data type, and can comprise integer type, floating point type, character type, boolean type and the like, further, the packaging environment configuration information, the module operation processor parameters, the module assembly function similarity and the module receiving data source type are respectively adaptively optimized for the driving modules, the method comprises the steps of firstly carrying out the packaging environment configuration information, the module operation processor parameters, the module assembly function similarity and the module receiving data source type to the task processing parameters in the packaging process of the driving modules, the module assembly function similarity is the module function similarity and the receiving data source type, the module receiving data source type is correspondingly matched with the data between the packaging process of the task data, the driving modules can be greatly improved, the self-adaptive to the driving module receiving data can be adaptively evaluated according to the principle of the data type, the self-adaptive to the driving module receiving data is greatly obtained, the self-adaptive to the driving module, and the self-adaptive performance is greatly improved, and the adaptive to the self-adaptive performance is greatly improved, acquiring real-time driving load indexes corresponding to each driving module in the driving modules, wherein the real-time driving load indexes are used for representing the driving load condition of the corresponding driving module at each moment, and positioning the driving load indexes of the abnormal driving module at the same time, so that a transfer mapping network is established according to the driving load indexes of the abnormal driving module and the driving load indexes of the rest driving modules
The establishment flow of the transfer mapping network comprises the following steps:
the method comprises the steps of constructing a fully-connected neural network, training a driving load index of an abnormal driving module and a driving load index of a residual driving module by using the neural network, wherein the fully-connected neural network is a neural network of a multi-layer perception structure, further constructing a transfer mapping network, each node of each layer of the fully-connected neural network is all connected with nodes of an upper layer and a lower layer, the transfer mapping network comprises an input layer, a hidden layer and an output layer, the input layer is a layer for data input, the hidden layer is used for better separation of data characteristics, the output layer is a layer for result output, the transfer mapping network is trained through a training data set and a supervision data set, each group of training data in the training data set comprises the driving load index of the abnormal driving module, and the driving load index of the residual driving module, and the supervision data set is supervision data corresponding to the training data set one by one.
Further, each group of training data in the training data set is input into the transfer mapping network, the output supervision adjustment of the transfer mapping network is carried out through the supervision data corresponding to the group of training data, when the output result of the transfer mapping network is consistent with the supervision data, the current group of training is finished, all the training data in the training data set are trained, and then the training of the fully connected neural network is finished.
In order to ensure the convergence and accuracy of the transfer mapping network, the convergence process may be that when the output data in the transfer mapping network is converged to one point, the convergence is performed when the output data is close to a certain value, the accuracy may be tested by the test data set for the transfer mapping network, for example, the test accuracy may be set to 80%, and when the test accuracy of the test data set meets 80%, the construction of the transfer mapping network is completed.
And finally, carrying out fitness optimization in the residual driving modules according to the transfer mapping network, wherein the fitness calculation means that each module in the residual driving modules is evaluated and the fitness value of each module in the residual driving modules in the problem space is determined. The higher the fitness value, the stronger the task transfer process of the module in the transfer mapping network and the greater the probability of being selected. And the adaptability calculation method is various, and can comprise an objective function method, a constraint function method, an analog simulation method and the like. And the fitness of each of the remaining driving modules can be determined by calculating the objective function value of each module by an objective function method, and the method can be divided into two steps: an objective function value and a conversion fitness value are calculated. Calculating the objective function value refers to calculating each module in the remaining driving modules according to the conversion mapping requirement to obtain the objective function value. The conversion fitness value refers to converting the objective function value into a fitness value, and may be converted by using a linear or exponential function, so as to correspondingly obtain a plurality of transfer fitness corresponding to each of the remaining driving modules, and further, performing weighted calculation according to the plurality of transfer fitness and the plurality of driving fitness, where the weighted calculation needs to be performed for the purpose after summarizing a large amount of data and accurately determining weights, and an exemplary weight ratio of the plurality of transfer fitness and the plurality of driving fitness may be a first influence coefficient: and if the second influence coefficient is 4:6, the influence parameters after the weighted calculation process are respectively 0.4 of the first influence parameter and 0.6 of the second influence parameter, the optimizing driving module is obtained according to the weighted calculation result to output, and the driving fitness of the optimizing driving module is highest in the remaining driving modules so as to be used as reference data when the parallel tasks are rapidly processed based on multi-model driving in the later period.
Step A600: establishing a data transmission channel of the optimizing driving module and the abnormal driving module, and acquiring a transfer task queue according to the data transmission channel;
in the application, in order to ensure the processing efficiency of the parallel task, a data transmission channel is required to be correspondingly established between the optimizing driving module and the abnormal driving module, the process of establishing the data transmission channel can be that the receiving end of the optimizing driving module directly feeds back the flow identifier associated with the data channel in the response message, further negotiation of the flow identifier in the subsequent process by the receiving end of the optimizing driving module and the transmitting end of the abnormal driving module can be avoided, so that after the receiving end of the abnormal driving module receives the response message, the data channel can be established directly according to the response attribute row carried in the response message and the receiving end of the optimizing driving module in an SDP mode, finally the task data with the response attribute transmitted by the optimizing driving module is extracted by taking the established data transmission channel as reference basic data, the extracted task data is sequentially processed according to the sequence of the task data, and finally a task transferring queue is obtained according to the sequence of the task data, and the task processing data contained in the abnormal driving module is transferred to the queue processed in the optimizing driving module, and the accuracy of the task processing based on the multi-model driving module is improved.
Step A700: processing the transfer task queue based on the optimizing driving module, and obtaining a processing return result of the transfer task queue;
in the application, in order to better process a plurality of tasks in parallel based on multi-model driving, under the condition that a optimizing driving module obtains a transfer task queue, task data in the transfer task queue is required to be rapidly processed through driving load indexes in the optimizing driving module according to the relevance of a program, the relevance of the program can comprise data relevance, control relevance, resource relevance and the like, meanwhile, task data in the module is rapidly processed in other driving modules according to driving load indexes corresponding to the module, and the task data processing process can comprise links such as data acquisition, cleaning, conversion, analysis and visualization, so that a processing return result after the optimizing driving module processes the transfer task queue is determined and exported, and further, the fast processing of the parallel tasks based on the multi-model driving in the later period is ensured.
Step A800: and transmitting a processing return result of the transfer task queue to the abnormal driving module.
In the application, the data of the transfer task queue can be obtained first in the process of transmitting the processing return result of the transfer task queue into the abnormal driving module, and the data can comprise raw data from a plurality of data sources, such as a sensor, a database, a log file and the like. Data collection may require data crawling, API calls, file uploading, and the like. The collected data is then purged, and the raw data contained in the multiple data sources typically contains errors, missing values, duplicate items, and inconsistencies. These problems are identified and remedied by data cleansing to ensure quality and consistency of the data. Second, the task data is subjected to data conversion, and during the data conversion stage, the data can be normalized, reconstructed or summarized, which involves data format conversion, merging of data sets, feature engineering, and the like. And then analyzing the task data, which can be technologies such as statistical analysis, machine learning, data mining and the like, so as to find modes, associations and trends in the data, wherein the task data analysis result can be used for making decisions and solving problems, and finally, the task data is required to be visualized, and the data visualization presents the processing return result of the transfer task queue in an understandable form through a chart, a graph and a dashboard, so that the data insight and the support decision process can be conveyed, and the task data processing return result in the abnormal driving module can be presented better, so that the high efficiency when the parallel task rapid processing is performed based on the multi-model driving is ensured.
In summary, the multi-model driving-based parallel task rapid processing method provided by the embodiment of the application at least comprises the following technical effects, so that reasonable and precise management and control on parallel task processing are realized, and the parallel task processing efficiency is improved.
Example two
Based on the same inventive concept as the parallel task quick processing method based on the multi-model driving in the foregoing embodiment, as shown in fig. 2, the present application provides a parallel task quick processing system based on the multi-model driving, where the system includes:
the packaging module 1 is used for respectively packaging according to a plurality of driving models to obtain a plurality of driving modules, wherein the driving modules are in communication connection with the data source module;
the first receiving module 2 is used for receiving the data of each data source according to the data source module and outputting a plurality of data sources;
the first matching module 3 is configured to receive a parallel task set, match the multiple data sources with the multiple driving modules according to the parallel task set, obtain driving load indexes corresponding to each driving module under task driving based on multiple matching data sources, and output multiple driving load indexes;
the abnormality module 4 is used for acquiring an abnormality driving module according to the plurality of driving load indexes;
the first optimizing module 5 is used for carrying out fitness optimization on the plurality of driving modules according to the abnormal driving module and outputting an optimizing driving module;
the channel establishment module 6 is used for establishing a data transmission channel of the optimizing driving module and the abnormal driving module, and acquiring a transfer task queue according to the data transmission channel;
the result acquisition module 7 is used for processing the transfer task queue based on the optimizing driving module and acquiring a processing return result of the transfer task queue;
the first transmission module 8 is configured to transmit a processing return result of the transfer task queue to the abnormal driving module by using the first transmission module 8.
Further, the system further comprises:
the first identification module is used for identifying the data of the data sources according to the data source module, and obtaining a plurality of mutation probabilities corresponding to the data sources, wherein the mutation probabilities are mutation probabilities of the received data quantity in each data source in unit time;
the second matching module is used for matching the plurality of driving modules according to the plurality of variation probabilities, and acquiring N variation probabilities corresponding to each driving module;
the fusion module is used for fusing the N variation probabilities corresponding to each driving module and outputting the fusion variation probability corresponding to each driving module;
and the optimizing module is used for optimizing the plurality of driving load indexes according to the fusion variation probability corresponding to each driving module and outputting the optimized plurality of driving load indexes.
Further, the system further comprises:
the data acquisition module is used for acquiring the packaging environment configuration information of the abnormal driving module, the module running processor parameters, the module component function similarity and the module receiving data source type;
the second optimizing module is used for optimizing the fitness of the plurality of driving modules according to the packaging environment configuration information, module operation processor parameters, module assembly function similarity and module receiving data source types to obtain a plurality of driving fitness;
and the first output module is used for outputting the optimizing driving module according to the driving fitness, wherein the driving fitness of the optimizing driving module is highest in the remaining driving modules.
Further, the system further comprises:
the index positioning module is used for acquiring real-time driving load indexes corresponding to each driving module in the plurality of driving modules and positioning driving load indexes of the abnormal driving modules;
the network establishment module is used for establishing a transfer mapping network according to the driving load index of the abnormal driving module and the driving load index of the residual driving module;
the third optimizing module is used for optimizing the fitness in the residual driving module according to the transfer mapping network to obtain a plurality of transfer fitness;
and the first calculation module is used for carrying out weighted calculation according to the plurality of transfer fitness and the plurality of driving fitness and outputting the optimizing driving module.
Further, the system further comprises:
the queue acquisition module is used for acquiring a task process queue of each driving model;
the second calculation module is used for carrying out calculation complexity identification according to the task process queue to obtain module calculation complexity;
the third calculation module is used for acquiring the equalization calculation complexity corresponding to each driving model, wherein the equalization calculation complexity is based on the calculation complexity of the driving module in load equalization;
and the fourth calculation module is used for outputting the driving load index by utilizing the index of the balance calculation complexity of the module calculation complexity.
Further, the system further comprises:
a fifth calculation module, configured to receive the parallel task set, calculate, according to the parallel task set, suppression factors corresponding to the plurality of driving modules, where the suppression factors corresponding to each driving module include a suppression factor of a data dependency relationship, a suppression factor of a task decomposition inherent delay, and a suppression factor of a model encapsulated in the module for use case testing;
the index acquisition module is used for acquiring a plurality of task inhibition indexes of the plurality of driving modules according to the inhibition factors corresponding to the plurality of driving modules;
the time sequence fusion module is used for time sequence fusion of the task inhibition indexes to obtain parallel inhibition indexes;
and the decomposition result acquisition module is used for acquiring the parallel task decomposition result with the parallel inhibition index minimized as a target.
The foregoing detailed description of the parallel task fast processing method based on the multi-model driver will be clear to those skilled in the art, and the parallel task fast processing system based on the multi-model driver in this embodiment is relatively simple for the device disclosed in the embodiment, and the relevant points refer to the method part for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (6)
1. The parallel task rapid processing method based on the multi-model drive is characterized by comprising the following steps of:
packaging according to the driving models respectively to obtain a plurality of driving modules, wherein the driving modules are in communication connection with the data source module;
receiving data of each data source according to the data source module, and outputting a plurality of data sources;
receiving a parallel task set, carrying out data source matching on the plurality of data sources and the plurality of driving modules according to the parallel task set to obtain corresponding driving load indexes of each driving module under task driving based on the plurality of matched data sources, and outputting a plurality of driving load indexes;
acquiring an abnormal driving module according to the plurality of driving load indexes;
performing fitness optimization on the plurality of driving modules according to the abnormal driving module, and outputting an optimized driving module;
establishing a data transmission channel of the optimizing driving module and the abnormal driving module, and acquiring a transfer task queue according to the data transmission channel;
processing the transfer task queue based on the optimizing driving module, and obtaining a processing return result of the transfer task queue;
transmitting a processing return result of the transfer task queue to the abnormal driving module;
the method for obtaining the driving load index corresponding to each driving model under task driving based on a plurality of matched data sources comprises the following steps:
acquiring a task process queue of each driving model;
performing calculation complexity identification according to the task process queue to obtain module calculation complexity;
acquiring the equalization calculation complexity corresponding to each driving model, wherein the equalization calculation complexity is based on the calculation complexity of the driving module during load equalization;
outputting the driving load index by utilizing the index of the module calculation complexity accounting for the balance calculation complexity;
the expression of the computational complexity recognition is as follows:
；
wherein t (n) is the computational complexity, n is the total number of processes in the corresponding task process queue, k is the number of processes in the corresponding task process queue for single parallel processing, [0, k ]]Is processed in parallel by a driving model connected with a corresponding task process queue, [ k+1, n ]]Requiring a sequential processing of the driving model,for the computational complexity of each process,for task progressThe number of processes sequentially executing in the queue.
2. The method of claim 1, wherein the method further comprises:
identifying the data of the data sources according to the data source module, and obtaining a plurality of mutation probabilities corresponding to the data sources, wherein the mutation probabilities are mutation probabilities of the received data amount in each data source in unit time;
matching the plurality of driving modules according to the plurality of variation probabilities to obtain N variation probabilities corresponding to each driving module;
fusing the N variation probabilities corresponding to each driving module, and outputting the fused variation probability corresponding to each driving module;
and optimizing the plurality of driving load indexes according to the fusion variation probability corresponding to each driving module, and outputting the optimized plurality of driving load indexes.
3. The method of claim 1, wherein the plurality of drive modules are adaptively optimized based on the abnormal drive module, and the optimized drive module is output, the method comprising:
acquiring the packaging environment configuration information of the abnormal driving module, the module running processor parameters, the module assembly function similarity and the module receiving data source type;
performing fitness optimization on the plurality of driving modules respectively according to the packaging environment configuration information, module operation processor parameters, module assembly function similarity and module receiving data source types to obtain a plurality of driving fitness;
and outputting the optimizing driving module according to the driving fitness, wherein the driving fitness of the optimizing driving module is highest in the rest driving modules.
4. A method as claimed in claim 3, wherein the method further comprises:
acquiring real-time driving load indexes corresponding to each driving module in the plurality of driving modules, and positioning the driving load indexes of the abnormal driving modules;
establishing a transfer mapping network according to the driving load index of the abnormal driving module and the driving load index of the residual driving module;
performing fitness optimization in the residual driving modules according to the transfer mapping network to obtain a plurality of transfer fitness;
and carrying out weighted calculation according to the plurality of transfer fitness and the plurality of driving fitness, and outputting the optimizing driving module.
5. The method of claim 1, wherein the method further comprises:
receiving the parallel task set, and calculating the inhibition factors corresponding to the plurality of driving modules according to the parallel task set, wherein the inhibition factors corresponding to each driving module comprise the inhibition factors of the data dependency, the inhibition factors of the inherent delay of task decomposition and the inhibition factors of the encapsulated model in the module for use case test;
acquiring a plurality of task inhibition indexes of the plurality of driving modules according to the inhibition factors corresponding to the plurality of driving modules;
carrying out time sequence fusion on the task inhibition indexes to obtain parallel inhibition indexes;
and obtaining a parallel task decomposition result by taking the parallel inhibition index minimization as a target.
6. A multi-model driven parallel task fast processing system, the system comprising:
the packaging module is used for respectively packaging according to the plurality of driving models to obtain a plurality of driving modules, wherein the plurality of driving modules are in communication connection with the data source module;
the first receiving module is used for receiving the data of each data source according to the data source module and outputting a plurality of data sources;
the first matching module is used for receiving a parallel task set, matching the data sources with the driving modules according to the parallel task set to obtain driving load indexes corresponding to each driving module under task driving based on the matched data sources, and outputting the driving load indexes;
the abnormal module is used for acquiring an abnormal driving module according to the plurality of driving load indexes;
the first optimizing module is used for optimizing the adaptation degree of the driving modules according to the abnormal driving module and outputting an optimizing driving module;
the channel establishment module is used for establishing a data transmission channel of the optimizing driving module and the abnormal driving module and acquiring a transfer task queue according to the data transmission channel;
the result acquisition module is used for processing the transfer task queue based on the optimizing driving module and acquiring a processing return result of the transfer task queue;
the first transmission module is used for transmitting the processing return result of the transfer task queue to the abnormal driving module;
the first matching module further includes:
the queue acquisition module is used for acquiring a task process queue of each driving model;
the second calculation module is used for carrying out calculation complexity identification according to the task process queue to obtain module calculation complexity;
the third calculation module is used for acquiring the equalization calculation complexity corresponding to each driving model, wherein the equalization calculation complexity is based on the calculation complexity of the driving module in load equalization;
the fourth calculation module is used for outputting the driving load index by utilizing the index of the balance calculation complexity occupied by the calculation complexity of the module;
the expression of the computational complexity recognition is as follows:
；
wherein t (n) is the computational complexity, n is the total number of processes in the corresponding task process queue, k is the number of processes in the corresponding task process queue for single parallel processing, [0, k ]]Is processed in parallel by a driving model connected with a corresponding task process queue, [ k+1, n ]]Requiring a sequential processing of the driving model,for the computational complexity of each process,is the number of processes sequentially executed in the task process queue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410055306.0A CN117573328B (en) | 2024-01-15 | 2024-01-15 | Parallel task rapid processing method and system based on multi-model driving |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410055306.0A CN117573328B (en) | 2024-01-15 | 2024-01-15 | Parallel task rapid processing method and system based on multi-model driving |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117573328A CN117573328A (en) | 2024-02-20 |
| CN117573328B true CN117573328B (en) | 2024-03-29 |
Family
ID=89884791
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410055306.0A Active CN117573328B (en) | 2024-01-15 | 2024-01-15 | Parallel task rapid processing method and system based on multi-model driving |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117573328B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118034939A (en) * | 2024-04-12 | 2024-05-14 | 无锡九方科技有限公司 | Parallel task scheduling method based on dynamic I/O load sensing |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106055401A (en) * | 2016-06-13 | 2016-10-26 | 北京唯智佳辰科技发展有限责任公司 | Automatic starting-stopping and computation task dynamic allocation method for mass parallel coarse particle computation |
| CN113127204A (en) * | 2021-04-29 | 2021-07-16 | 四川虹美智能科技有限公司 | Method and server for processing concurrent services based on reactor network model |
| CN113157440A (en) * | 2021-03-23 | 2021-07-23 | 北京云上曲率科技有限公司 | Self-adaptive load balancing and high availability guaranteeing method applied to mobile terminal |
| CN113781002A (en) * | 2021-09-18 | 2021-12-10 | 北京航空航天大学 | Low-cost workflow application migration method based on agent model and multi-population optimization in cloud edge cooperative network |
| CN116933010A (en) * | 2023-07-19 | 2023-10-24 | 国网上海市电力公司 | Load rate analysis and evaluation method and system based on multi-source data fusion and deep learning |
| CN117135060A (en) * | 2023-10-27 | 2023-11-28 | 北京云科领创信息技术有限公司 | Business data processing method and system based on edge calculation |
-
2024
- 2024-01-15 CN CN202410055306.0A patent/CN117573328B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106055401A (en) * | 2016-06-13 | 2016-10-26 | 北京唯智佳辰科技发展有限责任公司 | Automatic starting-stopping and computation task dynamic allocation method for mass parallel coarse particle computation |
| CN113157440A (en) * | 2021-03-23 | 2021-07-23 | 北京云上曲率科技有限公司 | Self-adaptive load balancing and high availability guaranteeing method applied to mobile terminal |
| CN113127204A (en) * | 2021-04-29 | 2021-07-16 | 四川虹美智能科技有限公司 | Method and server for processing concurrent services based on reactor network model |
| CN113781002A (en) * | 2021-09-18 | 2021-12-10 | 北京航空航天大学 | Low-cost workflow application migration method based on agent model and multi-population optimization in cloud edge cooperative network |
| CN116933010A (en) * | 2023-07-19 | 2023-10-24 | 国网上海市电力公司 | Load rate analysis and evaluation method and system based on multi-source data fusion and deep learning |
| CN117135060A (en) * | 2023-10-27 | 2023-11-28 | 北京云科领创信息技术有限公司 | Business data processing method and system based on edge calculation |
Non-Patent Citations (3)
| Title |
|---|
| 云数据中心中负载均衡的虚拟机调度方法;栾志坤;牛超;;计算机与现代化;20170515(第05期);全文 * |
| 基于事件驱动的云端动态任务分解模式优化方法;王艳;程丽军;;系统仿真学报;20181108(11);全文 * |
| 基于多源数据驱动的道路网络交通状态指数研究;舒芹;《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》;20220115;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117573328A (en) | 2024-02-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN117573328B (en) | Parallel task rapid processing method and system based on multi-model driving | |
| CN107817787B (en) | Intelligent production line manipulator fault diagnosis method based on machine learning | |
| US11650968B2 (en) | Systems and methods for predictive early stopping in neural network training | |
| CN116760772B (en) | Control system and method for converging flow divider | |
| CN115118602B (en) | Container resource dynamic scheduling method and system based on usage prediction | |
| CN110740079A (en) | full link benchmark test system for distributed scheduling system | |
| CN112580784A (en) | Intelligent early warning method for equipment based on multi-input multi-output convolutional neural network | |
| CN116340006A (en) | Computing power resource idle prediction method based on deep learning and storage medium | |
| Nagahara et al. | Toward data-driven production simulation modeling: dispatching rule identification by machine learning techniques | |
| CN116771576A (en) | Comprehensive fault diagnosis method for hydroelectric generating set | |
| Ma | copent: Estimating copula entropy and transfer entropy in R | |
| CN113888136A (en) | Workflow scheduling method based on DQN algorithm principle | |
| Wen et al. | MapReduce-based BP neural network classification of aquaculture water quality | |
| CN113886454A (en) | Cloud resource prediction method based on LSTM-RBF | |
| CN108921287A (en) | A kind of optimization method and system of neural network model | |
| CN116957304A (en) | Unmanned aerial vehicle group collaborative task allocation method and system | |
| Yang et al. | Trust-based scheduling strategy for cloud workflow applications | |
| CN112580798A (en) | Intelligent early warning method for equipment based on multi-input multi-output ResNet | |
| CN115794385A (en) | Container automatic arrangement method for deep learning model distributed training | |
| WO2022249518A1 (en) | Information processing device, information processing method, information processing program, and learning model generation device | |
| CN115062791A (en) | Artificial intelligence interpretation method, device, equipment and storage medium | |
| CN113992542A (en) | Online network flow prediction method and system based on newly-added flow number characteristics | |
| CN113505879A (en) | Prediction method and device based on multi-attention feature memory model | |
| CN112667591A (en) | Data center task interference prediction method based on mass logs | |
| Du et al. | OctopusKing: A TCT-aware task scheduling on spark platform |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |