CN115688054B - Task classification processing method based on big data processing frame - Google Patents
Task classification processing method based on big data processing frame Download PDFInfo
- Publication number
- CN115688054B CN115688054B CN202310005310.1A CN202310005310A CN115688054B CN 115688054 B CN115688054 B CN 115688054B CN 202310005310 A CN202310005310 A CN 202310005310A CN 115688054 B CN115688054 B CN 115688054B
- Authority
- CN
- China
- Prior art keywords
- task
- classification
- line task
- big data
- main line
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a task classification processing method based on a big data processing frame, which comprises the following steps: the big data processing frame receives a target task; classifying and identifying the target task according to a pre-established task classification model to obtain a class label of the target task; and the big data processing frame invokes the corresponding class processing module according to the class label of the target task to perform directional processing on the target task. According to the method, the information of the auxiliary line task is introduced into the classification of the main line task, and the main line task information and the auxiliary line task information are introduced into the classification branches through the neural network model with reasonable design, so that the discrimination capability of the task classification model is improved, the task categories with higher similarity are better distinguished, the limitation on information quantity is reduced, the identification accuracy of the target task category is improved, and the classification accuracy is improved.
Description
Technical Field
The invention relates to the technical field of big data processing, in particular to a task classification processing method based on a big data processing frame.
Background
The big data processing framework is responsible for the framework system that computes (manages and processes) the data in the big data system. The data includes data read from persistent storage or data accessed into the system by means of a message queue or the like, and the calculation is a process of extracting information from the data. Now, each big data computing framework supports multiple big data computing types, for example, SQL operation supporting Query is basically supported by each big data computing framework, or the big data computing framework is not directly supported and can be realized in an indirect mode. That means that a type of big data calculation request can be sent to multiple big data processing frameworks for execution, but the efficiency and manner of execution of each framework are different, and this is basically selected by full-scale consideration of input, output, performance, resource use and the like. Other types of big data processing are also such as OLAP, stream computation, search queries, graph computation, machine learning, etc. Each big data processing frame on the big data platform is often provided with a default designated big data storage frame with better compatibility, so that different types of big data processing operations are realized on the big data platform, tasks are submitted to a certain frame, a part of data sources which are often required come from the big data storage frames which are permanently stored or flow data accessed in a mode of other storage frames or message queues, and other data sources come to the big data processing frame because the big data processing frame is provided with the storage frames at the back. As mentioned before which framework is used specifically for each task is preferred by overall consideration of input, output, performance, use of resources, etc. If different processing frameworks can be used for the same type of task in an application program system, the tasks need to be implemented one by one for each processing framework, and a large amount of information about the execution framework needs to be involved when the tasks are submitted, if support for each big data processing framework is needed, the client is huge and heavy, and therefore, the task classification processing method can be used for solving the problems.
In the prior art, a machine learning model is used for identifying task types of target tasks, but the machine learning model for identifying the target task types is limited by the information quantity of the target tasks, and the problems of inaccurate identification and lower classification precision exist.
Disclosure of Invention
The invention aims to provide a task classification processing method based on a big data processing frame, which aims to solve the technical problems that a machine learning model for identifying target task types in the prior art is limited by target task information amount, identification is inaccurate and classification accuracy is low.
In order to solve the technical problems, the invention specifically provides the following technical scheme:
a task classification processing method based on a big data processing frame comprises the following steps:
the big data processing frame receives a target task;
classifying and identifying the target task according to a pre-established task classification model to obtain a class label of the target task;
the big data processing frame invokes the corresponding class processing module according to the class label of the target task to perform directional processing on the target task;
the class processing module monitors the execution process of the target task and feeds back the execution process to the big data processing frame in real time until the target task is processed.
As a preferred scheme of the present invention, the construction of the task classification model includes:
acquiring a plurality of processed historical tasks in a big data frame, marking a main line task category and a secondary line task category in the historical tasks, and carrying out convolution feature extraction on each historical task to obtain a convolution feature map of the historical tasks;
extracting global features of main line task classification and global features of auxiliary line task classification respectively from the convolution feature map of the historical task to obtain global vectors suitable for the main line task classification and global vectors suitable for the auxiliary line task classification;
taking the global vector of the main line task classification and the global vector of the auxiliary line task classification as input items of a CNN neural network, taking the main line task class and the auxiliary line task class as output items of the CNN neural network, and carrying out network training on the input items of the CNN neural network and the output items of the CNN neural network by utilizing the CNN neural network to obtain a task classification model;
the model expression of the task classification model is as follows:
[labelA,labelB]=CNN(S_A,S_B)；
in the formula, labelA is a main line task category, labelB is a secondary line task category, S_A is a global vector of main line task classification, S_B is a global vector of secondary line task classification, and CNN is a CNN neural network.
As a preferred embodiment of the present invention, the loss of the task classification model includes a classification loss of a primary task and a classification loss of a secondary task, wherein,
the classification loss of the mainline task is as follows:
lossA=|CNN_labelA-real_labelA| 2 ；
the classification loss of the secondary line task is as follows:
lossB=|CNN_labelB-real_labelB| 2 ；
in the formula, lossA and LossB are respectively the classification loss of the main line task and the classification loss of the auxiliary line task, cnn_labela is the main line task category output by the task classification model, real_labela is the main line task category in the historical task, cnn_labelb is the auxiliary line task category output by the task classification model, real_labela is the auxiliary line task category in the historical task, |cnn_labela-real_labela| 2 Is the two norms of the difference value of CNN_labelA and real_labelA, |CNN_labelB-real_labelB| 2 Is the two norms of the difference between CNN_labelB and real_labelB.
As a preferred scheme of the present invention, the construction of the task classification model further includes:
sequentially extracting principal components of the global vectors of the principal line task classification to obtain principal component vectors of principal line task classification features, extracting principal components of the global vectors of the auxiliary line task classification to obtain principal component vectors of auxiliary line task classification features, and taking the similarity between the principal component vectors of the principal line task classification features and the principal component vectors of the auxiliary line task classification features as task association loss;
and optimizing the task classification model based on the classification loss of the main line task classification, the classification loss of the auxiliary line task classification and the task association loss by using a gradient descent method so as to improve the classification accuracy of the model.
As a preferable scheme of the invention, euclidean distance is utilized to measure the similarity between the principal component vector of the principal line task classification characteristic and the principal component vector of the auxiliary line task classification characteristic.
As a preferable scheme of the invention, the convolution feature map of each history task is subjected to convolution feature map normalization processing before the extraction of the main line task classification global features and the extraction of the auxiliary line task classification global features.
As a preferred solution of the present invention, the category processing module monitors the execution process of the target task, including:
and the class processing module feeds back the task part which cannot be matched with the target task by the class processing module to the big data processing frame, and the big data processing frame carries out task classification processing again until the target task processing is completed.
As a preferable scheme of the invention, the class processing module comprises a main line task processing module and a subsidiary line task processing module, and the main line task processing module and the subsidiary line task processing module are respectively in one-to-one correspondence with the main line task class and the subsidiary line task class.
As a preferred embodiment of the present invention, the target task includes at least one of a main line task category and a sub line task category, and the main line task category includes: the auxiliary line task comprises a data cleaning task, a segmentation task and a caching task.
As a preferred embodiment of the present invention, the convolutional feature extraction network comprises AlexNet, googLeNet, VGGNet or res net.
Compared with the prior art, the invention has the following beneficial effects:
according to the method, the information of the auxiliary line task is introduced into the classification of the main line task, and the information of the main line task and the information of the auxiliary line task are introduced into the classification branches through the neural network model with reasonable design, so that the discrimination capability of the task classification model is improved, the task categories with higher similarity are better distinguished, the limitation on the information quantity is reduced, the identification accuracy of the target task categories is improved, the classification accuracy is improved, the deep features of the classification task and the auxiliary line task are respectively reduced in dimension, the reduced-dimension classification task features and the reduced-dimension auxiliary line task features are obtained, the association relation between the main line task and the auxiliary line task is learned in the training process, the information of the auxiliary line task is shared with the classification task, and a plurality of samples which are difficult to distinguish can be distinguished more easily.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It will be apparent to those of ordinary skill in the art that the drawings in the following description are exemplary only and that other implementations can be obtained from the extensions of the drawings provided without inventive effort.
Fig. 1 is a flowchart of a task classification processing method according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
As shown in fig. 1, the present invention provides a task classification processing method based on a big data processing framework, which includes the following steps:
the big data processing frame receives a target task;
classifying and identifying the target task according to a pre-established task classification model to obtain a class label of the target task;
the construction of the task classification model comprises the following steps:
acquiring a plurality of processed historical tasks in a big data frame, marking a main line task category and a secondary line task category in the historical tasks, and extracting convolution characteristics of each historical task to obtain a convolution characteristic diagram of the historical tasks;
extracting global features (full connection features) of main line task classification and global features (full connection features) of auxiliary line task classification respectively from the convolution feature graphs of the historical tasks to obtain global vectors suitable for the main line task classification and global vectors suitable for the auxiliary line task classification;
taking the global vector of the main line task classification and the global vector of the auxiliary line task classification as input items of a CNN neural network, taking the main line task class and the auxiliary line task class as output items of the CNN neural network, and carrying out network training on the input items of the CNN neural network and the output items of the CNN neural network by utilizing the CNN neural network to obtain a task classification model;
the model expression of the task classification model is:
[labelA,labelB]=CNN(S_A,S_B)；
in the formula, labelA is a main line task category, labelB is a secondary line task category, S_A is a global vector of main line task classification, S_B is a global vector of secondary line task classification, and CNN is a CNN neural network.
The penalty of the task classification model includes the classification penalty of the primary task and the classification penalty of the secondary task, wherein,
the classification loss of the main line task is as follows:
lossA=|CNN_labelA-real_labelA| 2 ；
the classification loss of the secondary line task is as follows:
lossB=|CNN_labelB-real_labelB| 2 ；
in the formula, lossA and LossB are respectively the classification loss of the main line task and the classification loss of the auxiliary line task, cnn_labela is the main line task category output by the task classification model, real_labela is the main line task category in the historical task, cnn_labelb is the auxiliary line task category output by the task classification model, real_labela is the auxiliary line task category in the historical task, |cnn_labela-real_labela| 2 Is the two norms of the difference value of CNN_labelA and real_labelA, |CNN_labelB-real_labelB| 2 Is the two norms of the difference between CNN_labelB and real_labelB.
The target task does not necessarily only contain a single task type, and the situation that more task types coexist is likely to occur, when the task classification model is constructed, information of a secondary line task is introduced into the classification of the main line task to serve as new information, and the new information is introduced into classification branches through a neural network frame with reasonable design, so that the discrimination capability of a task classification network is improved, and the category with higher similarity is better distinguished; therefore, the invention solves the problems of limitation in reducing information quantity, accurate classification identification and improved classification precision of the existing neural network framework for classifying target tasks in the big data processing framework.
The construction of the task classification model further comprises the following steps:
sequentially extracting principal components of the global vectors of the principal line task classification to obtain principal component vectors of principal line task classification characteristics, extracting principal components of the global vectors of the auxiliary line task classification to obtain principal component vectors of auxiliary line task classification characteristics, and taking the similarity between the principal component vectors of the principal line task classification characteristics and the principal component vectors of the auxiliary line task classification characteristics as task association loss;
and optimizing the task classification model based on the classification loss of the main line task classification, the classification loss of the auxiliary line task classification and the task association loss by using a gradient descent method so as to improve the classification accuracy of the model.
And measuring the similarity between the principal component vector of the principal line task classification characteristic and the principal component vector of the auxiliary line task classification characteristic by using the Euclidean distance.
The convolution feature map of each history task is subjected to convolution feature map normalization processing before the extraction of the main line task classification global features and the extraction of the auxiliary line task classification global features.
The method and the device construct task association loss, realize the relation study between the main line task and the auxiliary line task, and are used for sharing the deep features of the auxiliary line task and the main line task. Specifically, after the deep features of the main line task and the auxiliary line task are respectively reduced in dimension, the classified task features and the auxiliary line task features with reduced dimension are obtained, the difference value between the two features after the dimension reduction is minimized in the training process, and the effect is equivalent to the learning of the relationship between the two tasks; in the training process, the relation influences the weight distribution of the classification tasks, so that the information of the auxiliary tasks is shared to the classification of the main line tasks, and a plurality of indistinguishable samples can be more easily distinguished. The relationship learning between the primary line task and the secondary line task forms a feature sharing mechanism for further enhancing information sharing. The feature sharing mechanism of the present invention includes two parts: for shallow networks (convolutional layers), the convolutional layers of the classification task and the convolutional layers of the sub-line task use the same weight (equivalent to the same set of convolutional layers used for the classification task and the convolutional layers of the sub-line task); for a deep network (global layer), relationship learning is adopted to share information, so that the accuracy of task classification is improved.
The big data processing frame invokes the corresponding class processing module according to the class label of the target task to perform directional processing on the target task;
the class processing module monitors the execution process of the target task and feeds back the execution process to the big data processing frame in real time until the target task is processed.
The category processing module monitors the execution process of the target task, and comprises the following steps:
and the class processing module feeds back the task part which cannot be matched with the target task by the class processing module to the big data processing frame, and the big data processing frame carries out task classification processing again until the target task processing is completed.
The class processing module comprises a main line task processing module and an auxiliary line task processing module, and the main line task processing module and the auxiliary line task processing module are respectively in one-to-one correspondence with the main line task class and the auxiliary line task class.
The target task includes at least one of a primary task category and a secondary task category, the primary task category including: the auxiliary line task comprises a data cleaning task, a segmentation task and a caching task.
The extraction network for convolutional feature extraction includes AlexNet, googLeNet, VGGNet or ResNet.
According to the method, the information of the auxiliary line task is introduced into the classification of the main line task, and the information of the main line task and the information of the auxiliary line task are introduced into the classification branches through the neural network model with reasonable design, so that the discrimination capability of the task classification model is improved, the task categories with higher similarity are better distinguished, the limitation on the information quantity is reduced, the identification accuracy of the target task categories is improved, the classification accuracy is improved, the deep features of the classification task and the auxiliary line task are respectively reduced in dimension, the reduced-dimension classification task features and the reduced-dimension auxiliary line task features are obtained, the association relation between the main line task and the auxiliary line task is learned in the training process, the information of the auxiliary line task is shared with the classification task, and a plurality of samples which are difficult to distinguish can be distinguished more easily.
The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application, the scope of which is defined by the claims. Various modifications and equivalent arrangements may be made to the present application by those skilled in the art, which modifications and equivalents are also considered to be within the scope of the present application.
Claims (7)
1. A task classification processing method based on a big data processing frame is characterized by comprising the following steps of: the method comprises the following steps:
the big data processing frame receives a target task;
classifying and identifying the target task according to a pre-established task classification model to obtain a class label of the target task;
the big data processing frame invokes the corresponding class processing module according to the class label of the target task to perform directional processing on the target task;
the class processing module monitors the execution process of the target task and feeds back the execution process to the big data processing frame in real time until the target task is processed;
the construction of the task classification model comprises the following steps:
acquiring a plurality of processed historical tasks in a big data frame, marking a main line task category and a secondary line task category in the historical tasks, and carrying out convolution feature extraction on each historical task to obtain a convolution feature map of the historical tasks;
extracting global features of main line task classification and global features of auxiliary line task classification respectively from the convolution feature map of the historical task to obtain global vectors suitable for the main line task classification and global vectors suitable for the auxiliary line task classification;
taking the global vector of the main line task classification and the global vector of the auxiliary line task classification as input items of a CNN neural network, taking the main line task class and the auxiliary line task class as output items of the CNN neural network, and carrying out network training on the input items of the CNN neural network and the output items of the CNN neural network by utilizing the CNN neural network to obtain a task classification model;
the task classification model has the model expression of
[labelA,labelB]=CNN(S_A,S_B)；
In the formula, labelA is a main line task category, labelB is a secondary line task category, S_A is a global vector of main line task classification, S_B is a global vector of secondary line task classification, and CNN is a CNN neural network;
the loss of the task classification model comprises the classification loss of the main line task and the classification loss of the auxiliary line task, wherein,
the classification loss of the mainline task is as follows:
the classification loss of the secondary line task is as follows:
in the formula, lossA and LossB are the classification loss of a main line task and the classification loss of a secondary line task respectively, CNN_labelA is the main line task category output by a task classification model, real_labelA is the main line task category in a historical task, CNN_labelB is the secondary line task category output by a task classification model, real_labelA is the secondary line task category in the historical task,
the construction of the task classification model further comprises the following steps:
sequentially extracting principal components of the global vectors of the principal line task classification to obtain principal component vectors of principal line task classification features, extracting principal components of the global vectors of the auxiliary line task classification to obtain principal component vectors of auxiliary line task classification features, and taking the similarity between the principal component vectors of the principal line task classification features and the principal component vectors of the auxiliary line task classification features as task association loss;
and optimizing the task classification model based on the classification loss of the main line task classification, the classification loss of the auxiliary line task classification and the task association loss by using a gradient descent method so as to improve the classification accuracy of the model.
2. The task classification processing method based on the big data processing framework according to claim 1, wherein: and measuring the similarity between the principal component vector of the principal line task classification characteristic and the principal component vector of the auxiliary line task classification characteristic by using the Euclidean distance.
3. A task classification processing method based on a big data processing framework as claimed in claim 2, wherein: and carrying out convolution feature map normalization processing on the convolution feature maps of the historical tasks before carrying out extraction of the main line task classification global features and extraction of the auxiliary line task classification global features.
4. A task classification processing method based on a big data processing framework according to claim 3, wherein the class processing module monitors the execution process of the target task, and includes:
and the class processing module feeds back the task part which cannot be matched with the target task by the class processing module to the big data processing frame, and the big data processing frame carries out task classification processing again until the target task processing is completed.
5. The task classification processing method based on the big data processing framework of claim 4, wherein the class processing module includes a main line task processing module and a sub line task processing module, and the main line task processing module and the sub line task processing module are respectively in one-to-one correspondence with the main line task class and the sub line task class.
6. The task classification processing method based on the big data processing framework according to claim 5, wherein the target task includes at least one of a main line task category and a sub line task category, the main line task category including: the auxiliary line task comprises a data cleaning task, a segmentation task and a caching task.
7. The task classification processing method based on a big data processing framework according to claim 6, wherein the extraction network for convolutional feature extraction comprises AlexNet, googLeNet, VGGNet or ResNet.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310005310.1A CN115688054B (en) | 2023-01-04 | 2023-01-04 | Task classification processing method based on big data processing frame |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310005310.1A CN115688054B (en) | 2023-01-04 | 2023-01-04 | Task classification processing method based on big data processing frame |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115688054A CN115688054A (en) | 2023-02-03 |
| CN115688054B true CN115688054B (en) | 2023-07-11 |
Family
ID=85057001
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310005310.1A Active CN115688054B (en) | 2023-01-04 | 2023-01-04 | Task classification processing method based on big data processing frame |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115688054B (en) |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200348662A1 (en) * | 2016-05-09 | 2020-11-05 | Strong Force Iot Portfolio 2016, Llc | Platform for facilitating development of intelligence in an industrial internet of things system |
| CN108459844A (en) * | 2018-02-06 | 2018-08-28 | 福建星瑞格软件有限公司 | The classification of task processing method and computer equipment of frame are handled based on big data |
| CN111190704A (en) * | 2019-12-30 | 2020-05-22 | 盐城古卓科技有限公司 | Task classification processing method based on big data processing framework |
| CN113886378A (en) * | 2021-10-20 | 2022-01-04 | 苏州中科先进技术研究院有限公司 | Big data management system |
-
2023
- 2023-01-04 CN CN202310005310.1A patent/CN115688054B/en active Active
Non-Patent Citations (1)
| Title |
|---|
| Developing a knowledge discovery framework in highway projects using Industrial Foundation Class(IFC) extension;Shen Zhenhua;《The University of North Carolina at Charlotte ProQuest Dissertations Publishing》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115688054A (en) | 2023-02-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107291945B (en) | High-precision clothing image retrieval method and system based on visual attention model | |
| CN106294344B (en) | Video retrieval method and device | |
| CN109214410A (en) | A kind of method and system promoting multi-tag classification accuracy rate | |
| CN106919957B (en) | Method and device for processing data | |
| CN105608113B (en) | Judge the method and device of POI data in text | |
| CN113361645B (en) | Target detection model construction method and system based on meta learning and knowledge memory | |
| EP4209959A1 (en) | Target identification method and apparatus, and electronic device | |
| CN111950561A (en) | Semantic SLAM dynamic point removing method based on semantic segmentation | |
| CN110728313A (en) | Classification model training method and device for intention classification recognition | |
| CN113762508A (en) | Training method, device, equipment and medium for image classification network model | |
| CN105183857A (en) | Automatic picture training sample extracting method and system | |
| CN115115825A (en) | Method and device for detecting object in image, computer equipment and storage medium | |
| CN115688054B (en) | Task classification processing method based on big data processing frame | |
| US11093531B2 (en) | Systems and methods for recalling points of interest using a tagging model | |
| CN112784102B (en) | Video retrieval method and device and electronic equipment | |
| CN113918605A (en) | Data query method, device, equipment and computer storage medium | |
| CN109165283B (en) | Resource recommendation method, device, equipment and storage medium | |
| WO2024000822A1 (en) | Text classification annotation sample anomaly detection method and apparatus, device, and medium | |
| US20230153286A1 (en) | Method and system for hybrid query based on cloud analysis scene, and storage medium | |
| CN104615620A (en) | Map search type identification method and device and map search method and system | |
| WO2023134142A1 (en) | Multi-scale point cloud classification method and system | |
| CN113780532B (en) | Training method, device, equipment and storage medium of semantic segmentation network | |
| CN115438719A (en) | Data processing method, device, server and storage medium | |
| CN112948469A (en) | Data mining method and device, computer equipment and storage medium | |
| US11232135B2 (en) | Methods and system of using N-gram analysis to discover points of interest in a given geographic region |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |