CN115982586A - Semi-supervised continuous learning method for converting few-sample text into SQL task flow - Google Patents

Semi-supervised continuous learning method for converting few-sample text into SQL task flow Download PDF

Info

Publication number
CN115982586A
CN115982586A CN202310025951.3A CN202310025951A CN115982586A CN 115982586 A CN115982586 A CN 115982586A CN 202310025951 A CN202310025951 A CN 202310025951A CN 115982586 A CN115982586 A CN 115982586A
Authority
CN
China
Prior art keywords
task
semi
learning
sample
supervised
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310025951.3A
Other languages
Chinese (zh)
Inventor
陈永锐
郭心南
吴桐桐
漆桂林
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202310025951.3A priority Critical patent/CN115982586A/en
Publication of CN115982586A publication Critical patent/CN115982586A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The invention relates to a semi-supervised continuous learning method for converting a few-sample text into an SQL task flow, which comprises the following steps: step 1, for a new task, a model is trained by using semi-supervised learning in the task, step 2, for a past task in a task flow, the model keeps memory of the learned task by using continuous learning, and step 3, a semi-supervised learning process and a continuous learning process are respectively executed by using a teacher-student framework; and 4, strengthening and executing semi-supervised learning and continuous learning by utilizing double sampling, wherein the semi-supervised learning and the continuous learning respectively comprise prompt sampling and review sampling, and strengthening mutual promotion of two learning processes. The technical scheme applies a 'teacher-student' framework to isolate different optimization targets of semi-supervised learning and continuous learning, and adopts a double sampling strategy to strengthen the mutual promotion effect between the semi-supervised learning and the continuous learning.

Description

Semi-supervised continuous learning method for converting few-sample text into SQL task flow
Technical Field
The invention belongs to the field of natural language processing, and relates to a semi-supervised continuous learning method for converting a few-sample text into an SQL (structured query language) task flow.
Background
The power relational database stores a large amount of information at present, and provides a foundation for application of customer relation management, financial markets, medical records and the like. The text-to-SQL technique trains a parser to translate natural language questions into machine-readable SQL programs, providing an ideal way for non-technical users to easily interact with data stored in a relational database. Current research on text-to-SQL has covered single-table, multi-table, and dialogue scenarios, with the common assumption that the size of training and test data does not change over time. Unfortunately, in real-world applications, new databases are constantly emerging to adapt to changing environments (e.g., new diseases and adjusted financial policies) and to continually generate new tasks for the parser. While machine learning-based text-to-SQL approaches have achieved the most advanced performance, they suffer from the following two challenges when faced with a rapidly growing task:
1) Sample-less supervision data. For a new text-to-SQL task for unseen databases, it is often impractical to train by annotating enough SQL tags in a short time, resulting in easy overfitting of the parser. 2) Expensive full retraining. Considering a new task, an intuitive idea is to train the model from scratch on all the tasks seen. Unfortunately, due to the large scale of the pre-trained model, the computational cost of such retraining is prohibitive even in a small sample scenario.
Based on this, the present work proposes to integrate semi-supervised learning (semi-supervised learning) and continuous learning (continuous learning) to solve the sample-less text-to-SQL task flow. The parser applies self-training to predict pseudo-tag instances to improve generalization over current tasks while replaying a small portion of past instances stored in memory to mitigate forgetting over previous tasks. In this process, semi-supervised learning and continuous learning can be mutually facilitated. On the one hand, some examples in previous tasks may provide valuable information for semi-supervised learning to predict the pseudo-label of unlabeled examples. On the other hand, high quality pseudo-tag instances may also enrich the memory of past tasks. A "teacher-student" framework is applied to semi-supervised learning and continuous learning processes, respectively. The teacher model strives to achieve optimization of each single task through self-training, while the student model achieves optimization of the entire task flow by replaying the pseudo labels that learn all tasks predicted by the teacher model. To take advantage of the interactive facilitation of continuous learning and semi-supervised learning, past instances associated with a current task are used to prompt the semi-supervised learning process when training the teacher model; when training the student model, both tagged and untagged instances of previous tasks are sampled to ensure complete memory for replay.
Disclosure of Invention
In order to solve the defects in the prior art, the invention aims to provide a semi-supervised continuous learning method for converting a few-sample text into an SQL task flow.
The invention adopts the following technical scheme: a semi-supervised continuous learning method for a sample-less text-to-SQL task flow, the method comprising the steps of:
step 1, for a new task, a model is trained by using semi-supervised learning in the task,
for the ith new task D in the task stream i Using its supervision data A i ＝{a 1 ,a 2 …,a n And unsupervised data U i ＝{u 1 ,u 2 ,…,u m Self-training is carried out, and a semi-supervised learning process in the task is completed, wherein a j ＝(q j ,S j ,y j )，u j ＝(q j ,S j )，q j And S j Natural language question and database model for respectively representing inputFormula (ii) y j Representing a target SQL program, wherein n and m are respectively the number of supervised data and unsupervised data, and the process mainly comprises the following two stages:
and (3) hot start: model using coding-decoding architecture
Figure BDA0004044939470000021
Where θ is the model parameter and i represents the ith task. The encoder is a table pre-training language model and the decoder is a long-short term memory network (LSTM), first, the model is firstly at the ith task D i Supervision data A of i Training is carried out to obtain parameters of parameter initialization, wherein the loss L of each sample is calculated as the following formula,
Figure BDA0004044939470000022
wherein
Figure BDA0004044939470000023
For the actions predicted by the model at the decoding time t, z ＜t All actions before the moment t, Z is the whole action sequence, | Z | is the length of the whole action sequence, and when all actions are completed, an SQL program is generated;
self-renewal: subsequently, the air conditioner is operated to,
Figure BDA0004044939470000024
in supervision data A i And unsupervised data U i Performs a joint iterative training, in particular, the model ≥ at each round of the training process>
Figure BDA0004044939470000025
First for each unlabeled data u j ∈U i Predictive pseudo-tags, i.e., corresponding SQL programs +>
Figure BDA0004044939470000026
Constitute a pseudo-sample>
Figure BDA0004044939470000027
Randomly selecting k pseudo samples p to form an ith task D i Of the pseudo sample set P i ＝{p 1 ,p 2 ,…,p k Then further update theta by optimizing the penalty,
Figure BDA0004044939470000031
wherein, mu j ＝P(z t |q j ,S j θ) is a pseudo sample p j The confidence score of.
Step 2, for past tasks in the task flow, the model keeps the memory of the learned tasks by using continuous learning.
When a new task is encountered, the task is executed,
Figure BDA0004044939470000032
cannot be retrained in all the previous examples, i.e. < >>
Figure BDA0004044939470000033
Forget a past task after learning a new task and, therefore, based on the past task and the previous task, based on the previous task and the previous task>
Figure BDA0004044939470000034
Some of the examples in the past tasks need to be reviewed to ensure that their performance does not significantly degrade, and the process mainly includes the following two phases:
memory storage construction: in preparation, a fixed size memory M is constructed i ＝{m 1 ,m 2 ,…,m |M| With each task D i Is associated to store a small number of replay instances of the task, wherein
Figure BDA0004044939470000035
Is from supervisory data A i The sampling-in-process is carried out,
calculating the replay loss: whenever a
Figure BDA0004044939470000036
When self-training is performed, M1 to M are calculated i-1 All replay sample losses, added to the semi-supervised loss L ST Upper, replay loss L EMR By the following calculation formula,
Figure BDA0004044939470000037
at task D i After the training of (2), from A i In the random selection
Figure BDA0004044939470000038
A sample with a label and storing it in M i For replay in future tasks.
Step 3, respectively executing a semi-supervised learning process and a continuous learning process by utilizing a teacher-student framework;
since semi-supervised learning addresses the optimization of a single (current) task, while continuous learning focuses more on the overall performance of all tasks, the present method proposes to perform both processes separately using a "teacher-student" framework. The model is composed of two basic text-to-SQL models, namely a teacher for semi-supervised learning
Figure BDA0004044939470000039
And student for continuous learning>
Figure BDA00040449394700000310
In learning task D i During, is greater than or equal to>
Figure BDA00040449394700000311
And &>
Figure BDA00040449394700000312
Are all made by>
Figure BDA00040449394700000313
Initialized, but with a separate parameter update during each task, the parameter being recorded as ≥ er>
Figure BDA00040449394700000314
And &>
Figure BDA00040449394700000315
To facilitate the mutual promotion of semi-supervised learning and continuous learning, double sampling is used: involving two different strategies, respectively enhancing>
Figure BDA0004044939470000041
And &>
Figure BDA0004044939470000042
The training data of (a) is obtained,
a teacher model:
Figure BDA0004044939470000043
is to provide the current task D through the intra-task ST i Of interest to D only i Allowing forgetting and D i Unrelated past tasks; simultaneously with D i The relevant past instance may be emphasized to deepen @>
Figure BDA0004044939470000044
To D i To achieve this goal, during ST, the appropriate instance with the potential hint is replayed, in particular while observing each task D i In time, slave->
Figure BDA0004044939470000045
Extract and D i Associative tagged sample composition memory storage>
Figure BDA0004044939470000046
This step is called prompt sampling, after which the decision is taken to be based on the result of the determination>
Figure BDA0004044939470000047
Self-training is performed as in step 1, except that L will be lost during the self-refresh process ST Replacement by the following loss L tea ，
Figure BDA0004044939470000048
When the training has converged on the time the training is,
Figure BDA0004044939470000049
is considered to be close to D i At the cost of->
Figure BDA00040449394700000410
May forget D 1 ,…,D i-1 Some of the key information of (1).
A student model: after training
Figure BDA00040449394700000411
Can be considered as a proficiency task D i ，D1,…,D K Providing K experts, where K is the total task number->
Figure BDA00040449394700000412
By learning all +>
Figure BDA00040449394700000413
The pseudo label is provided, so that the overall performance of the whole task flow is optimized. For each task D j (1. Ltoreq. J. Ltoreq. I-1) except for the original D j In addition to this, it is also possible to select from trained->
Figure BDA00040449394700000414
Learning in the generated pseudo label example, and considering the training efficiency, the method is to do every D i Using cross-task sample replay, specifically, based on the evaluation of the results of the evaluation>
Figure BDA00040449394700000415
Comprises 1) D i Task loss of 2) D1, …, D i-1 Is lost.
Step 4, performing semi-supervised learning and continuous learning by utilizing double sampling reinforcement, wherein the double sampling reinforcement respectively comprises prompt sampling and review sampling, and mutual promotion of two learning processes is reinforced;
prompting sampling: to ensure there is study D i Should the sample instance satisfy two criteria, 1) their corresponding database schema S should be semantically related to D i The database schema presented in (1) is close, 2) the structure of their golden SQL programs should be diversified to improve
Figure BDA00040449394700000416
The generalization ability of the method to different structures,
the sampling process mainly comprises the following two stages, firstly, the sampling process is carried out
Figure BDA00040449394700000417
Select the first s and D i Most correlated samples, each sample x j And D i The correlation ω of (d) is calculated by the following formula:
Figure BDA0004044939470000051
Figure BDA0004044939470000052
wherein, d sch (x l ,x j ) Represents a sample x l And x j Distance over the database schema, Ψ represents a dictionary that encompasses all words present in the database schema, Ψ is each word,
Figure BDA0004044939470000053
is the ψ th bit of the unique heat vector corresponding to the bag of words, if ψ exists in the dictionary, then +>
Figure BDA0004044939470000054
Are all 1, then>
Figure BDA0004044939470000055
Is a group of a number of 0 s,
then, the obtained s samples are divided into N clusters by using a K-means clustering algorithm. Wherein the structure distance is defined as d str To satisfy the diversity of SQL constructs. d is a radical of str Similar in form to d sch But instead of Ψ (including ODRER BY, GROUP BY, LIMIT, etc.) a SQL keyword table Φ is used. Finally, the center sample of each cluster is selected to be composed
Figure BDA0004044939470000056
Reviewing and sampling: review sampling except for the labeled sample set A i In addition to sampling, a pseudo label instance P is also sampled i Sampling is carried out, wherein P i The label of each p in (1) is
Figure BDA0004044939470000057
Prediction during intra-task self-training, in such a way as to allow
Figure BDA0004044939470000058
Review task D more fully i To make a sampled instance represent D in terms of both SQL structure and database schema i First a combined distance d (x) is defined 1 ,x 2 )＝d sch (x 1 ,x 2 )*d str (x 1 ,x 2 ) To measure two samples x 1 And x 2 Then, using the distance A i ∪P i Division into M clusters, result +>
Figure BDA0004044939470000059
Composed of the center sample of each cluster as D i Representative sample of (a).
Compared with the prior art, the invention has the beneficial effects that:
1. a solution combining semi-supervised learning and continuous learning is provided to solve the problem of converting text into SQL task flow under the scene of few samples;
2. a teacher-student framework is provided to isolate different optimization targets of semi-supervised learning and continuous learning, so that the respective advantages of the semi-supervised learning and the continuous learning can be fully exerted in the training process;
3. a dual sampling strategy is proposed to strengthen the link between semi-supervised learning and continuous learning, facilitating the effect of both learning processes by reviewing different samples.
Drawings
FIG. 1 is a block flow diagram of a semi-supervised continuous learning text to SQL conversion
FIG. 2 is a schematic diagram of the text-to-SQL model of the present invention
Detailed Description
The present application is further described below with reference to the accompanying drawings. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and the protection scope of the present application is not limited thereby.
Example 1: the semi-supervised continuous learning method aiming at the conversion from the text with few samples to the SQL task flow of the invention has the overall flow as shown in figure 1,
the method comprises the following steps:
step 1, for a new task, the model is trained by using semi-supervised learning in the task.
For the ith new task D in the task stream i Using its supervision data A i ＝{a 1 ,a 2 …,a n And unsupervised data U i ＝{u 1 ,u 2 ,…,u m Self-training is carried out, and a semi-supervised learning process in the task is completed, wherein a j ＝(q j ,S j ,y j )，u j ＝(q j ,S j )，q j And S j Respectively representing input natural language questions and database schemas, y j Representing the target SQL program, m and, respectively, the number of supervised and unsupervised data, the process mainly comprises the following two phases:
and (3) hot start: model using coding-decoding architecture
Figure BDA0004044939470000061
Where θ is the model parameter and i represents the ith task. The encoder is a table pre-training language model and the decoder is a long-short memory network (LSTM), first, the model is firstly at the ith task D i Supervision data A of i Training is performed to obtain a parameter for which a parameter is initialized, wherein the loss L per sample is calculated as follows, and ` H `>
Figure BDA0004044939470000062
Wherein
Figure BDA0004044939470000063
For the actions predicted by the model at the decoding time t, z ＜t For all moves before time t, Z is the entire move sequence and | Z | is the length of the entire move sequence. />
Figure BDA0004044939470000064
Indicates that the time t is generating>
Figure BDA0004044939470000065
As shown in FIG. 2, the model iteratively generates an action z using the problem and database patterns as inputs t . When all actions are completed, the SQL program is generated;
self-renewal: subsequently, the process of the present invention,
Figure BDA0004044939470000066
in supervision data A i And unsupervised data U i Performs a joint iterative training, in particular, the model ≥ at each round of the training process>
Figure BDA0004044939470000071
First, for each non-label data u j ∈U i Predictive pseudo-tags, i.e., corresponding SQL programs +>
Figure BDA0004044939470000072
Constitute a dummy sample>
Figure BDA0004044939470000073
Randomly selecting k pseudo samples p to form an ith task D i Pseudo sample set P of i ＝{p 1 ,p 2 ,…,p k Then further update theta by optimizing the penalty,
Figure BDA0004044939470000074
wherein, mu j ＝P(z t |q j ,S j θ) is a dummy sample p j The confidence score of.
Step 2, for past tasks in the task flow, the model keeps the memory of the learned tasks by using continuous learning.
When a new task is encountered, the task is executed,
Figure BDA0004044939470000075
cannot be retrained in all the preceding examples, i.e. <' >>
Figure BDA0004044939470000076
Forgets the past task after learning a new task, and therefore, based on the past task>
Figure BDA0004044939470000077
Some examples in past tasks need to be reviewed to ensure that their performance does not significantly degrade, and the process mainly includes the following two stages:
memory storage construction: as a preparation, a fixed-size memory M is constructed i ＝{m 1 ,m 2 ,…,m |M| With each task D i Associated to store a small number of replay instances of the task, wherein
Figure BDA0004044939470000078
Is from supervisory data A i The sampling rate of the medium-frequency sampling is medium,
calculating the replay loss: whenever there is a need for
Figure BDA0004044939470000079
When self-training is performed, M1 to M are calculated i-1 All replay sample losses of (2) added to the semi-supervised loss L ST Upper, replay loss L EMR By the following calculation formula,
Figure BDA00040449394700000710
at task D i After the training of (A) is finished, the training is performed i In the random selection
Figure BDA00040449394700000711
A labeled sample is stored in M i For replay in future tasks.
Step 3, respectively executing a semi-supervised learning process and a continuous learning process by utilizing a teacher-student framework;
since semi-supervised learning addresses the optimization of a single (current) task, while continuous learning focuses more on the overall performance of all tasks, the present method proposes to perform both processes separately using a "teacher-student" framework. The model is composed of two basic text-to-SQL models, namely a teacher for semi-supervised learning
Figure BDA00040449394700000712
And student for continuous learning>
Figure BDA0004044939470000081
In learning task D i During, is greater than or equal to>
Figure BDA0004044939470000082
And &>
Figure BDA0004044939470000083
Are all determined by>
Figure BDA0004044939470000084
Initialized, but with individual updating of parameters during each task, with the parameters respectively noted as->
Figure BDA0004044939470000085
And &>
Figure BDA0004044939470000086
To facilitate the mutual promotion of semi-supervised learning and continuous learning, double sampling is used: involving two different strategies, respectively enhancing>
Figure BDA0004044939470000087
And &>
Figure BDA0004044939470000088
The training data of (2) is obtained,
a teacher model:
Figure BDA0004044939470000089
is to provide the current task D through the intra-task ST i Of interest to D only i Allowing forgetting and D i Unrelated past tasks; simultaneously with D i Related past instances may be emphasized to deepen @>
Figure BDA00040449394700000810
To D i To achieve this goal, during ST, the appropriate instance with the potential hint is replayed, in particular while observing each task D i Then, the slave->
Figure BDA00040449394700000811
Extract and D i The associated tagged sample constitutes a memory store->
Figure BDA00040449394700000812
This step is called a prompt sample, after which>
Figure BDA00040449394700000813
Self-training is performed as per step 1, except that L will be lost during self-refresh ST Replacement by the following loss L te0 ，
Figure BDA00040449394700000814
When the training has converged on the time the training is,
Figure BDA00040449394700000815
is considered to be close to D i In the optimal model of (4), at a cost of &>
Figure BDA00040449394700000816
May forget D 1 ,…,D i-1 Some of the key information of (2).
A student model: trained
Figure BDA00040449394700000817
Can be considered as a proficiency task D i ，D 1 ,…,D K Providing K experts, where K is the total task number, based on the total task number>
Figure BDA00040449394700000818
By learning all->
Figure BDA00040449394700000819
The pseudo label is provided, so that the overall performance optimization of the whole task flow is achieved. For each task D j (1. Ltoreq. J. Ltoreq. I-1) except for the original D j In addition to this, it is also possible to select from trained->
Figure BDA00040449394700000820
Learning in the generated pseudo label instance, as shown in FIG. 1, takes into account training efficiency by doing so at every D i Using cross-task sample replay, particularly>
Figure BDA00040449394700000821
Comprises 1) D i Task loss of and 2) D 1 ,…,D i-1 Is lost.
Step 4, performing semi-supervised learning and continuous learning by utilizing double sampling reinforcement, wherein the double sampling reinforcement respectively comprises prompt sampling and review sampling, and mutual promotion of two learning processes is reinforced;
prompting sampling: to ensure there is study D i Should meet two criteria, 1) their corresponding database schema S should be semantically related to D i The database schema presented in (1) is close, 2) the structure of their golden SQL program should be diversified to improve
Figure BDA0004044939470000091
For the generalization ability of different structures,
the sampling process mainly comprises the following two stages, firstly, the sampling process is carried out
Figure BDA0004044939470000092
Select the first s and D i Most correlated samples, each sample x j And D i The correlation ω of (d) is calculated by the following formula:
Figure BDA0004044939470000093
Figure BDA0004044939470000094
wherein, d sch (x l ,x j ) Representing a sample x l And x j Distance over the database schema, Ψ denotes a dictionary that covers all words present in the database schema, Ψ is each word,
Figure BDA0004044939470000095
is the ψ th bit of the unique heat vector corresponding to the bag of words, if ψ exists in the dictionary, then +>
Figure BDA0004044939470000096
1, then>
Figure BDA0004044939470000097
Is a group of a number of 0 s,
then, the obtained s samples are divided into N clusters by using a K-means clustering algorithm. Wherein the structure distance is defined as d str To satisfy the diversity of SQL constructs. d s(r Similar in form to d sch But instead of Ψ (including ODRER BY, GROUP BY, LIMIT, etc.) a SQL keyword table Φ is used. Finally, the center sample of each cluster is selected to be composed
Figure BDA0004044939470000098
Reviewing and sampling: review sampling except for labeled sample set A i In addition to sampling, pseudo tag instance P is also sampled i Sampling is carried out, wherein P i The label of each p in (1) is
Figure BDA0004044939470000099
Prediction during intra-task self-training, in such a way as to allow
Figure BDA00040449394700000910
Review task D more fully i To make a sample instance represent D in terms of both SQL constructs and database schema i First a combined distance d (x) is defined 1 ,x 2 )＝d sch (x 1 ,x 2 )*d str (x 1 ,x 2 ) To measure two samples x 1 And x 2 Then, using the distance A i ∪P i Divided into M clusters, result->
Figure BDA00040449394700000911
Composed of the center sample of each cluster as D i Representative sample of (a).
Experimental analysis proves that the semi-supervised continuous learning method for converting the few-sample text into the SQL task flow uses unsupervised data to supplement supervision signals under the condition of insufficient sample amount, and adopts a replay strategy to recall the past samples according to the characteristics of the task flow. In addition, the method also uses a 'teacher-student' framework to respectively process the continuous learning process and the semi-supervised learning process, and utilizes double sampling to mutually promote the two processes, thereby realizing an accurate and efficient task flow training method.
The present applicant has described and illustrated embodiments of the present invention in detail with reference to the accompanying drawings, but it should be understood by those skilled in the art that the above embodiments are merely preferred embodiments of the present invention, and the detailed description is only for the purpose of helping the reader to better understand the spirit of the present invention, and not for limiting the scope of the present invention, and on the contrary, any improvement or modification made based on the spirit of the present invention should fall within the scope of the present invention.

Claims (5)

1. The semi-supervised continuous learning method for the conversion from the few-sample text to the SQL task flow is characterized by comprising the following steps of:
step 1, for a new task, a model is trained by using semi-supervised learning in the task,
step 2, for the past tasks in the task flow, the model keeps the memory of the learned tasks by using continuous learning,
step 3, respectively executing a semi-supervised learning process and a continuous learning process by utilizing a teacher-student framework;
and 4, strengthening and executing semi-supervised learning and continuous learning by utilizing double sampling, wherein the semi-supervised learning and the continuous learning respectively comprise prompt sampling and review sampling, and strengthening mutual promotion of two learning processes.
2. The semi-supervised continuous learning method for the text-to-SQL task flow with few samples according to claim 1, wherein the step 1 is as follows: for the ith new task D in the task stream i Using its supervision data A i ＝{a 1 ，a 2 ...，a n And unsupervised data U i ＝{u 1 ，u 2 ，...，u m Self-training is carried out, and a semi-supervised learning process in the task is completed, wherein a j ＝(q j ，S j ，y j )，u j ＝(q j ，S j )，q j And S j Respectively representing input natural language questions and database schemas, y j Representing a target SQL program, wherein n and m are respectively the number of supervised data and unsupervised data, and the process mainly comprises the following two stages:
and (3) hot start: model employing an encoding-decoding architecture
Figure FDA0004044939460000011
Where θ is the model parameter, i represents the ith task, the encoder is the table pre-training language model, the decoder is the long-short memory network (LSTM), first, the model is first at the ith task D i Supervision data A of i Training is carried out to obtain parameters of parameter initialization, wherein the loss L of each sample is calculated as the following formula,
Figure FDA0004044939460000012
wherein
Figure FDA0004044939460000013
For the actions predicted by the model at the decoding time t, z ＜t All actions before the moment t, Z is the whole action sequence, | Z | is the length of the whole action sequence, and when all actions are completed, an SQL program is generated;
self-renewal: subsequently, the air conditioner is operated to,
Figure FDA0004044939460000014
in supervision data A i And unsupervised data U i Performs a joint iterative training, in particular, the model ≥ at each round of the training process>
Figure FDA0004044939460000015
First, for each non-label data u j ∈U i Predictive false tag, i.e., the corresponding SQL procedure >>
Figure FDA0004044939460000016
Constitute a pseudo-sample>
Figure FDA0004044939460000017
Randomly selecting k pseudo samples p to form an ith task D i Of the pseudo sample set P i ＝{p 1 ，p 2 ，...，p k H, then further update theta by optimizing for losses as follows,
Figure FDA0004044939460000021
wherein, mu j ＝P(z t |q j ，S j θ) is a dummy sample p j The confidence score of.
3. The semi-supervised continuous learning method for a text-to-SQL task flow with few samples according to claim 1, wherein the step 2 is specifically as follows: when a new task is encountered, the task is executed,
Figure FDA0004044939460000022
cannot be retrained in all the preceding examples, i.e. <' >>
Figure FDA0004044939460000023
Forget a past task after learning a new task and, therefore, based on the past task and the previous task, based on the previous task and the previous task>
Figure FDA0004044939460000024
Some examples in past tasks need to be reviewed to ensure that their performance does not significantly degrade, and the process mainly includes the following two stages:
memory storage construction: in preparation, a fixed size memory M is constructed i ＝{m 1 ，m 2 ，...，m |M| With each task D i Associated to store a small number of replay instances of the task, wherein
Figure FDA0004044939460000025
Is from supervisory data A i The sampling-in-process is carried out,
and (3) calculating the replay loss: whenever there is a need for
Figure FDA0004044939460000026
When self-training is performed, M is calculated 1 To M i-1 All replay sample losses of (2) added to the semi-supervised loss L ST Upper, replay loss L EMR By the following calculation formula,
Figure FDA0004044939460000027
at task D i After the training of (A) is finished, the training is performed i In the random selection
Figure FDA0004044939460000028
A sample with a label and storing it in M i For replay in future tasks.
4. The semi-supervised continuous learning method for the text-to-SQL task flow with few samples according to claim 1, wherein the step 3 is as follows: step 3, respectively executing a semi-supervised learning process and a continuous learning process by utilizing a teacher-student framework;
the model is composed of two basic text-to-SQL models, namely a teacher for semi-supervised learning
Figure FDA0004044939460000029
And students for continuous learning>
Figure FDA00040449394600000210
In learning task D i During, is greater than or equal to>
Figure FDA00040449394600000211
And &>
Figure FDA00040449394600000212
Are all made by>
Figure FDA00040449394600000213
Initialized, but with individual updating of parameters during each task, with the parameters respectively noted as->
Figure FDA00040449394600000214
And &>
Figure FDA00040449394600000215
To facilitate the mutual promotion of semi-supervised learning and continuous learning, double sampling is used: involving two different strategies, respectively enhancing>
Figure FDA00040449394600000216
And &>
Figure FDA00040449394600000217
The training data of (2) is obtained,
a teacher model:
Figure FDA0004044939460000031
is to provide the current task D through the intra-task ST i Of interest to D only i Allowing forgetting to associate with D i Unrelated past tasks; simultaneously with D i The relevant past instance may be emphasized to deepen @>
Figure FDA0004044939460000032
To D i To achieve this goal, during ST, the appropriate instance with the potential hint is replayed, in particular while observing each task D i Then, the slave->
Figure FDA0004044939460000033
Extract and D i The associated tagged sample constitutes a memory store->
Figure FDA0004044939460000034
This step is called prompt sampling, after which the decision is taken to be based on the result of the determination>
Figure FDA0004044939460000035
Self-training is performed as in step 1, except that L will be lost during the self-refresh process ST Is replaced by the following loss L tea ，
Figure FDA0004044939460000036
When the training has converged on the time the training is,
Figure FDA0004044939460000037
is considered to be close to D i At the cost of->
Figure FDA0004044939460000038
May forget D 1 ，...，D i-1 Is determined by the user, and some of the key information of (2),
a student model: assumed to be trained
Figure FDA0004044939460000039
Is a task of essence i Then the task flow could theoretically be D 1 ，...，D K Providing K experts, where K is the total number of tasks, a parser will be the global optimum, or "arm", for the entire task stream if it can inherit the capabilities of all of these experts>
Figure FDA00040449394600000310
Intended to be a parser for each task D j (1. Ltoreq. J. Ltoreq. I-1) except for the original D j In addition to this, it is also possible to select from trained->
Figure FDA00040449394600000311
Learning in the generated pseudo label example, and considering the training efficiency, the method is to do every D i Using cross-task sample replay, specifically, based on the evaluation of the results of the evaluation>
Figure FDA00040449394600000312
The loss of (1) D i Task loss of and 2) D 1 ，...，D i-1 Is lost.
5. The semi-supervised continuous learning method for the text-to-SQL task flow with few samples according to claim 1, wherein the step 4 is as follows:
prompting sampling: to ensure there is study D i Should the sample instance satisfy two criteria, 1) their corresponding database schema S should be semantically related to D i The database schema presented in (1) is close, 2) the structure of their golden SQL programs should be diversified to improve
Figure FDA00040449394600000313
For the generalization ability of different structures,
the sampling process mainly comprises the following two stages, firstly, the sampling process is carried out
Figure FDA00040449394600000314
Select the first s and D i Most correlated samples, each sample x j And D i The correlation ω of (d) is calculated by the following formula: />
Figure FDA0004044939460000041
Figure FDA0004044939460000042
Wherein d is sch (x l ，x j ) Represents a sample x l And x j Distance over the database schema, Ψ represents a dictionary that encompasses all words present in the database schema, Ψ is each word,
Figure FDA0004044939460000043
to correspond to the ψ th digit of the bag-of-words unique heat vector, if ψ exists in the dictionary
Figure FDA0004044939460000044
1, then>
Figure FDA0004044939460000045
Is a group of a number of 0 s,
then, dividing the obtained s samples into N clusters by using a K-means clustering algorithm, wherein the structure distance is defined as d str To satisfy the diversity of SQL structure, finally, the central sample of each cluster is selected to be composed
Figure FDA0004044939460000046
Reviewing and sampling: review sampling except for the labeled sample set A i In addition to sampling, a pseudo label instance P is also sampled i Sampling is carried out, wherein P i The label of each p in (1) is
Figure FDA0004044939460000047
Prediction during intra-task self-training, in such a way as to allow
Figure FDA0004044939460000048
Review task D more fully i To make a sample instance represent D in terms of both SQL constructs and database schema i First a combined distance d (x) is defined 1 ，x 2 )＝d sch (x 1 ，x 2 )*d str (x 1 ，x 2 ) To measure two samples x 1 And x 2 Then, using the distance A i ∪P i Divided into M clusters, result->
Figure FDA0004044939460000049
Composed of the center sample of each cluster as D i Representative sample of (a). />
CN202310025951.3A 2023-01-09 2023-01-09 Semi-supervised continuous learning method for converting few-sample text into SQL task flow Pending CN115982586A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310025951.3A CN115982586A (en) 2023-01-09 2023-01-09 Semi-supervised continuous learning method for converting few-sample text into SQL task flow

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310025951.3A CN115982586A (en) 2023-01-09 2023-01-09 Semi-supervised continuous learning method for converting few-sample text into SQL task flow

Publications (1)

Publication Number Publication Date
CN115982586A true CN115982586A (en) 2023-04-18

Family

ID=85972207

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310025951.3A Pending CN115982586A (en) 2023-01-09 2023-01-09 Semi-supervised continuous learning method for converting few-sample text into SQL task flow

Country Status (1)

Country Link
CN (1) CN115982586A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132003A (en) * 2023-10-26 2023-11-28 云南师范大学 Student academic performance early prediction method based on self-training semi-supervised learning

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117132003A (en) * 2023-10-26 2023-11-28 云南师范大学 Student academic performance early prediction method based on self-training semi-supervised learning
CN117132003B (en) * 2023-10-26 2024-02-06 云南师范大学 Early prediction method for student academic performance of online learning platform based on self-training and semi-supervised learning

Similar Documents

Publication Publication Date Title
CN110532377B (en) Semi-supervised text classification method based on confrontation training and confrontation learning network
CN109902145B (en) Attention mechanism-based entity relationship joint extraction method and system
CN111695779B (en) Knowledge tracking method, knowledge tracking device and storage medium
CN112733541A (en) Named entity identification method of BERT-BiGRU-IDCNN-CRF based on attention mechanism
Peng et al. Unsupervised cross-media retrieval using domain adaptation with scene graph
CN112487820B (en) Chinese medical named entity recognition method
Fang et al. A new sequential image prediction method based on LSTM and DCGAN
CN112527993B (en) Cross-media hierarchical deep video question-answer reasoning framework
CN114398976A (en) Machine reading understanding method based on BERT and gate control type attention enhancement network
CN115982586A (en) Semi-supervised continuous learning method for converting few-sample text into SQL task flow
CN113704437A (en) Knowledge base question-answering method integrating multi-head attention mechanism and relative position coding
Luo et al. Self-supervised learning for semi-supervised temporal language grounding
Zhu et al. Complementarity-aware space learning for video-text retrieval
CN113420552B (en) Biomedical multi-event extraction method based on reinforcement learning
Ji et al. Teachers cooperation: team-knowledge distillation for multiple cross-domain few-shot learning
Wei et al. MoCo4SRec: A momentum contrastive learning framework for sequential recommendation
Li et al. Progressive cross-domain knowledge distillation for efficient unsupervised domain adaptive object detection
CN112926323A (en) Chinese named entity identification method based on multi-stage residual convolution and attention mechanism
Yu et al. Cross-Domain Slot Filling as Machine Reading Comprehension.
CN114548090B (en) Fast relation extraction method based on convolutional neural network and improved cascade labeling
CN115545038A (en) Aspect emotion analysis method for optimizing grid label
CN116127954A (en) Dictionary-based new work specialized Chinese knowledge concept extraction method
CN115480585A (en) Unmanned equipment cluster collaborative exploration method carrying reinforcement learning agent
Chen et al. Learn from yesterday: A semi-supervised continual learning method for supervision-limited text-to-sql task streams
Song Distilling knowledge from user information for document level sentiment classification

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination