CN112633514A - Multi-task function-to-function regression method - Google Patents
Multi-task function-to-function regression method Download PDFInfo
- Publication number
- CN112633514A CN112633514A CN202011156334.XA CN202011156334A CN112633514A CN 112633514 A CN112633514 A CN 112633514A CN 202011156334 A CN202011156334 A CN 202011156334A CN 112633514 A CN112633514 A CN 112633514A
- Authority
- CN
- China
- Prior art keywords
- function
- regression
- regularization
- task
- tasks
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000006870 function Effects 0.000 claims abstract description 129
- 239000011159 matrix material Substances 0.000 claims abstract description 21
- 230000001419 dependent effect Effects 0.000 claims abstract description 15
- 238000005516 engineering process Methods 0.000 claims abstract description 10
- 238000013507 mapping Methods 0.000 claims abstract description 6
- 238000005065 mining Methods 0.000 claims abstract description 6
- 238000005457 optimization Methods 0.000 claims description 5
- 238000010801 machine learning Methods 0.000 abstract description 11
- 230000008901 benefit Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 238000012905 input function Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000013508 migration Methods 0.000 description 2
- 230000005012 migration Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000002068 genetic effect Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Operations Research (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Computational Biology (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
Abstract
The invention relates to the field of machine learning, and discloses a multi-task function-to-function regression method, which comprises the following steps: s1, constructing a function-to-function regression model based on the double expansion of the basis function, and establishing a target function of the mapping from the independent variable function to the dependent variable function; s2, further constructing a multi-task function to an objective function of the function regression model, and improving the performance of each regression task by collaboratively grouping and mining the hidden structures among the tasks, wherein the objective function in the step contains a regression coefficient matrix; s3, applying constraint to the regression coefficient matrix in the step S2 by adopting different sparsity regularization technologies; s4, optimizing the problem that the objective function of the final function regression model is not smooth and independent; the multi-task function-to-function regression model based on structural sparsity has the advantages that the similarity of tasks and the clustering characteristics of basis functions can be mined simultaneously, and the clustering characteristics are used for improving the performance of a function regression system.
Description
Technical Field
The invention relates to the field of machine learning, in particular to a multi-task function-to-function regression method.
Background
Heterogeneous machine learning is used for researching how to mine heterogeneous data association relation among multiple tasks, fields, views and modalities so as to improve performance of a single system. Because the sources of real-world data are diverse, heterogeneity is a natural property of functional data. The development of heterogeneous machine learning has promoted the rapid development of functional data analysis, wherein the functional data considers continuous infinite-dimension data rather than discrete finite-dimension vectors. The heterogeneous machine learning fully excavates data heterogeneity, and the generalization performance of the machine learning system is effectively improved through knowledge sharing or complementary utilization among heterogeneous data.
Feature deconstruction is one of the key challenges of machine learning. In the field of traditional machine learning, researchers have proposed many sophisticated methods of feature analysis and feature deconstruction, such as: non-negative matrix factorization, principal component analysis, singular value decomposition, canonical correlation analysis, and the like. However, there is currently a lack of mature and effective feature deconstruction methods in the field of heterogeneous machine learning. On the other hand, data distribution difference is a core research problem of heterogeneous learning (such as migration learning, multitask learning, lifetime learning and the like), and is one of the main obstacles for knowledge sharing and migration.
Functional data is data defined over one or more continuous domains (e.g., time domain, spatial domain, spectral domain, and genetic location, etc.). Because the feature space is of infinite dimensions, functional data has greater data representation capabilities. The data (such as time series, images, audio, video and text) commonly used in the machine learning system can be represented by the new frame. The functional data contains complex and diverse associated information, which is divided into two types, one is functional, which refers to the characteristics (such as smoothness, periodicity, sparsity and the like) contained in the functional data collected from a single data source; the other type is heterogeneity, which refers to the correlation between functional data collected from different data sources (such as domain heterogeneity, task heterogeneity, etc.).
In the real world, many application problems in the field of machine learning can be summarized as a mapping problem between two functions. However, few studies have been made at present. It is necessary to provide a regression method based on multitask function to function, which solves the problems in the current research and helps to develop artificial intelligence.
Disclosure of Invention
The present invention is directed to overcoming the above-mentioned drawbacks of the prior art, and provides a multi-task function-to-function regression method, which improves the prediction performance of each functional regression task by mining the correlation between different tasks.
To achieve the above object, the present invention provides a multitask function-to-function regression method, comprising the steps of:
s1, constructing a function-to-function regression model based on the double expansion of the basis function, and establishing a target function of the mapping from the independent variable function to the dependent variable function;
s2, further constructing a multi-task function to an objective function of the function regression model, and mining a hidden structure among tasks through a cooperative grouping technology to improve the performance of each regression task, wherein the objective function in the step contains a regression coefficient matrix;
s3, applying constraint to the regression coefficient matrix in the step S2 by adopting different sparsity regularization technologies;
and S4, optimizing the problem that the objective function of the final function regression model is not smooth and independent.
Preferably, the function-to-function regression model in step S1 is as follows:
where x(s) and y (t) are the independent and dependent variable functions, respectively, [ epsilon ] (t) is the error function, W is the regression coefficient matrix,
and θ is a basis function.
Preferably, the objective function of the function-to-function regression model in step S1 is as follows:
wherein,
Ω (W) is a regularization term. (ii) a
Preferably, the objective function of the function-to-function regression model for multitasking in step S2 is as follows:
where T is the number of function regression tasks, xi(s) and yi(t) is the independent variable function and dependent variable function of the ith regression task, WiIs the regression coefficient matrix of the ith function regression task,
Ω({Wi}) is a regularization term.
Preferably, the sparsity regularization technique in step S3 is Lasso, L1Regularization, L2,1Regularization or Schatten techniques.
Preferably, step S3 includes structure sparsity regularization, which includes two regularization terms, the first regularization term is regularization of task clustering, and the second regularization term is regularization of basis function grouping.
Preferably, the optimization processing in step S4 is as follows: and converting the target function into a smooth function, and decoupling a plurality of tasks to form independent functions.
Compared with the prior art, the invention has the beneficial effects that:
1. the problem of the method research based on the multi-task function-to-function regression is a new learning problem of heterogeneous function type data, and is not considered by the prior art and the method;
2. this method based on multi-tasking function-to-function regression is also a novel approach. Because it models the correlations between tasks and between basis functions simultaneously under consideration of cooperative grouping and structural sparsity. This is not available with the prior art and methods;
3. the method can effectively solve the problems of non-smoothness and non-independence caused by the structural sparsity of the cooperative grouping, so that the method has excellent characteristics of separability, convex functionality, global convergence and the like, and can be widely applied to the related technology of structural sparsity regularization.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a first schematic diagram of a multitasking function-to-function regression method flow provided by the present invention;
FIG. 2 is a second schematic diagram of a multitasking function-to-function regression method flow provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1 and 2, the present invention provides a multitask function-to-function regression method, comprising the steps of:
s1, constructing a function-to-function regression model based on the double expansion of the basis function, and establishing a target function of the mapping from the independent variable function to the dependent variable function;
s2, further constructing a multi-task function to an objective function of the function regression model, and mining a hidden structure among tasks through a cooperative grouping technology to improve the performance of each regression task, wherein the objective function in the step contains a regression coefficient matrix;
s3, applying constraint to the regression coefficient matrix in the step S2 by adopting different sparsity regularization technologies;
and S4, optimizing the problem that the objective function of the final function regression model is not smooth and independent.
The important steps in the overall process are further described below.
One, two-fold spread basis function system
We first construct a function-to-function regression model based on basis function extensions, whose goal is to create a mapping from the independent variable function (input function) to the dependent variable function (output function).
Let x(s) and y (t) be the independent and dependent variable functions, respectively. Here, the independent and dependent variable functions may be defined over different continuous domains, i.e., S ∈ S, T ∈ T. Here, we do not require that the input function and the output function belong to the same domain to adapt to different application scenarios. The function-function regression model we used is as follows:
where ε (t) is the error function, W is the regression coefficient matrix,
and θ is a basis function.
The objective function of the function-to-function regression model is a regular term that minimizes the regression loss and the regression coefficient matrix for reconstructing the dependent variable function using the independent variable function:
wherein,
Ω (W) is a regularization term.
Two, multi-tasking function-to-function regression
Based on the function-to-function regression model, a multi-task function-to-function regression model is further provided, and the performance of each regression task is improved by mining the hidden structure among tasks through a sparse regularization technology.
Assume that there are T function regression tasks. x is the number ofi(s) and yi(t) are the independent variable function and the dependent variable function of the ith regression task, respectively. WiIs the regression coefficient matrix of the ith function regression task. We propose a multitask function to function regression model based on structural sparsity as follows. The objective function is to minimize the function regression loss and structural sparsity regularization term of each task:
wherein
Ω({Wi}) is a regularization term.
The purpose of this method is to model both the dependencies between tasks and the dependencies between basis functions, and therefore to consider both aspects. On the one hand, common basis functions are generally divided into several classes: fourier functions, spline functions, wavelet functions, functional principal components, etc., with different types of basis functions being suitable for processing functional data of different attributes. On the other hand, in many cases, we can get a priori knowledge of a set of tasks.
The assumption of cooperative grouping is that the regression coefficient weight matrix WiThe rows of (2) may be grouped according to the type of basis function; regression coefficient weight matrix WiMay be grouped according to soft clustering of tasks. In this way, clustering groupings of multiple regression tasks may overlap, e.g., each task may belong to one or more cluster groups.
Task and basis function sparse regularization
According to a specific application scenario, we use different sparsity regularization techniques to constrain model parameters, such as: lasso, L1Regularization, L2,1Regularization, Schatten regularization, and the like. In addition, implicit clustering structures exist in tasks, and different groups of basic functions (such as wavelet functions, Fourier functions, spline functions and the like) can exist. In this way, we can apply a structural sparsity constraint to the regression coefficient matrix. The multi-task function to function regression model based on structural sparsity has the advantage that the task soft clustering features and the basis function clustering features can be modeled simultaneously. It encourages similar tasks to choose similar basis functions or sets of basis functions. In addition to feature selection, sparsity regularization techniques may also enhance the interpretability of the multitask learning model.
The structural sparsity regularization of the method is as follows:
wherein, W(k)Is a block matrix which is the combination of regression coefficient matrices corresponding to all regression tasks in the kth task cluster.
Is W(k)The submatrices corresponding to the b-th set of basis functions. Assume that there are a total of G task clusters and B groups of basis functions.
The structural sparsity regularization technique includes two regularizations. The first regularization is regularization of task clustering, which can approximate the result of task selection basis functions in the same cluster, and ensure the specific sparsity of the task clustering by reducing irrelevant basis functions. The second regularization is regularization of the basis function groups, ensuring group sparsity of the basis functions by reducing the associated groups. Therefore, in this way we can model both the dependencies between tasks and the dependencies between basis functions.
We use the generalized Schatten paradigm to simultaneously model soft clustering of tasks and grouping of basis functions, with the goal of having an input function x for each taski(s) to an output function yi(t) the loss of the regression model is minimal. The generalized Schatten paradigm may encompass many models of sparse regularization, such as l2, the p regularization model, and the Schatten-p regularization model. l2, p regularization model, can make the results of task selection basis functions in the same cluster similar, reducing irrelevant basis functions. The Schatten-p regularization model can enable similar tasks to share similar low rank structures. Both may produce a weight matrix of sparse structures. Further, the generalized Schatten paradigm is a flexible mechanism for characterizing task dependencies. It can model soft clustering of regression tasks. For example, we can learn task-dependent correlations and task-independent correlations simultaneously through the multi-tasking function-function regression approach presented herein. Task-dependent dependencies may capture specific attributes of each different task, while task-independent dependencies may capture common attributes of all tasks.
Fourthly, optimization processing of method
Because the cooperative grouping is structurally sparse and multiple tasks are coupled together, the problems of non-smoothness and non-independence are generated, and optimization processing is needed. To solve this problem, the objective function can be first transformed into a smooth function, and then in this modified function, the tasks can be decoupled, i.e. independent functions. The two processes can be performed simultaneously. The result of this optimization process is convex optimized, convergent, fitting l2, the p regularization model and the Schatten-p regularization model.
In conclusion, the beneficial effects of the invention are as follows: firstly, the problem of the research of the method from function to function regression based on multiple tasks is a new learning problem of heterogeneous function type data, and is not considered by the prior art and the method; secondly, this method based on multi-tasking function-to-function regression is also a novel method. Because it models the correlations between tasks and between basis functions simultaneously under consideration of cooperative grouping and structural sparsity. This is not available with the prior art and methods; in addition, the method can effectively solve the problems of non-smoothness and non-independence caused by the structural sparsity of the cooperative grouping, so that the method has excellent characteristics of separability, convex functionality, global convergence and the like, and can be widely applied to the related technology of structural sparsity regularization.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (7)
1. A multitasking function-to-function regression method, comprising the steps of:
s1, constructing a function-to-function regression model based on the double expansion of the basis function, and establishing a target function of the mapping from the independent variable function to the dependent variable function;
s2, further constructing a multi-task function to an objective function of the function regression model, and mining a hidden structure among tasks through a cooperative grouping technology to improve the performance of each regression task, wherein the objective function in the step contains a regression coefficient matrix;
s3, applying constraint to the regression coefficient matrix in the step S2 by adopting different sparsity regularization technologies;
and S4, optimizing the problem that the objective function of the final function regression model is not smooth and independent.
2. The multi-tasking function-to-function regression method of claim 1, wherein the function-to-function regression model in step S1 is as follows:
where x(s) and y (t) are the independent and dependent variable functions, respectively, [ epsilon ] (t) is the error function, W is the regression coefficient matrix,
and θ is a basis function.
3. The multi-tasking function-to-function regression method of claim 2, wherein the objective function of the function-to-function regression model in step S1 is as follows:
wherein,
Ω (W) is a regularization term. (ii) a
4. The method of claim 3, wherein the objective function of the multi-tasking function-to-function regression model in step S2 is as follows:
where T is the number of function regression tasks, xi(s) and yi(t) are respectively the ith roundIndependent and dependent variable functions of the tasking, WiIs the regression coefficient matrix of the ith function regression task,
Ω({Wi}) is a regularization term.
5. The multi-tasking function-to-function regression method of claim 1, wherein the sparsity regularization technique in step S3 is Lasso, L1Regularization, L2,1Regularization or Schatten techniques.
6. The multi-tasking function-to-function regression method of claim 1, wherein step S3 comprises structure sparsity regularization, which comprises two regularization terms, the first regularization term is regularization of task clustering, and the second regularization term is regularization of basis function grouping.
7. The multi-tasking function-to-function regression method of claim 1, wherein the optimization in step S4 is performed by: and converting the target function into a smooth function, and decoupling a plurality of tasks to form independent functions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011156334.XA CN112633514A (en) | 2020-10-26 | 2020-10-26 | Multi-task function-to-function regression method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011156334.XA CN112633514A (en) | 2020-10-26 | 2020-10-26 | Multi-task function-to-function regression method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112633514A true CN112633514A (en) | 2021-04-09 |
Family
ID=75303106
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011156334.XA Pending CN112633514A (en) | 2020-10-26 | 2020-10-26 | Multi-task function-to-function regression method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112633514A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114696788A (en) * | 2022-04-12 | 2022-07-01 | 电子科技大学 | Multi-main-lobe interference resistant waveform and filter joint cognitive design method |
| CN115415851A (en) * | 2022-10-08 | 2022-12-02 | 清华大学 | Cutter health monitoring method based on functional data principal component analysis |
-
2020
- 2020-10-26 CN CN202011156334.XA patent/CN112633514A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114696788A (en) * | 2022-04-12 | 2022-07-01 | 电子科技大学 | Multi-main-lobe interference resistant waveform and filter joint cognitive design method |
| CN114696788B (en) * | 2022-04-12 | 2023-05-05 | 电子科技大学 | Wave form and filter combined cognition design method for resisting multi-main lobe interference |
| CN115415851A (en) * | 2022-10-08 | 2022-12-02 | 清华大学 | Cutter health monitoring method based on functional data principal component analysis |
| CN115415851B (en) * | 2022-10-08 | 2023-09-26 | 清华大学 | Cutter health monitoring method based on functional data principal component analysis |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Deng et al. | Model compression and hardware acceleration for neural networks: A comprehensive survey | |
| US11544539B2 (en) | Hardware neural network conversion method, computing device, compiling method and neural network software and hardware collaboration system | |
| CN110533183B (en) | Task placement method for heterogeneous network perception in pipeline distributed deep learning | |
| CN104951425B (en) | A kind of cloud service performance self-adapting type of action system of selection based on deep learning | |
| WO2024114399A1 (en) | Optimization method for distributed execution of deep learning task, and distributed system | |
| Han et al. | Signal processing and networking for big data applications | |
| CN112633514A (en) | Multi-task function-to-function regression method | |
| CN111858072A (en) | Resource management method and system for large-scale distributed deep learning | |
| CN114356540A (en) | Parameter updating method and device, electronic equipment and storage medium | |
| US11567778B2 (en) | Neural network operation reordering for parallel execution | |
| CN113538162A (en) | Planting strategy generation method and device, electronic equipment and storage medium | |
| CN113537365B (en) | Information entropy dynamic weighting-based multi-task learning self-adaptive balancing method | |
| Li et al. | Optimizing makespan and resource utilization for multi-DNN training in GPU cluster | |
| CN111753995A (en) | Local interpretable method based on gradient lifting tree | |
| CN113935489A (en) | Variational quantum model TFQ-VQA based on quantum neural network and two-stage optimization method thereof | |
| CN113792621A (en) | Target detection accelerator design method based on FPGA | |
| Dong et al. | Lambo: Large language model empowered edge intelligence | |
| CN113836174B (en) | Asynchronous SQL (structured query language) connection query optimization method based on reinforcement learning DQN (direct-to-inverse) algorithm | |
| CN112799603B (en) | Task behavior model for multi-data stream driven signal processing system | |
| Ma et al. | Accelerating deep neural network filter pruning with mask-aware convolutional computations on modern CPUs | |
| CN107766076B (en) | Software module clustering method for probability selection | |
| Du | In-machine-learning database: Reimagining deep learning with old-school SQL | |
| CN115796029A (en) | NL2SQL method based on explicit and implicit characteristic decoupling | |
| CN114120447A (en) | Behavior recognition method and system based on prototype comparison learning and storage medium | |
| Niu et al. | A Novel Distributed Duration-Aware LSTM for Large Scale Sequential Data Analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210409 |