CN111680517B - Method, apparatus, device and storage medium for training model - Google Patents
Method, apparatus, device and storage medium for training model Download PDFInfo
- Publication number
- CN111680517B CN111680517B CN202010524463.3A CN202010524463A CN111680517B CN 111680517 B CN111680517 B CN 111680517B CN 202010524463 A CN202010524463 A CN 202010524463A CN 111680517 B CN111680517 B CN 111680517B
- Authority
- CN
- China
- Prior art keywords
- intention
- labeling
- training
- sample data
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
Abstract
The application discloses a method, a device, equipment and a storage medium for training a model, and relates to the technical fields of natural language processing and deep learning. The specific implementation scheme is as follows: acquiring a first training sample set and a second training sample set; predicting the first sample data by using a first intention recognition model, and determining the confidence coefficient corresponding to each prediction intention; determining a third training sample set according to the first labeling intention, the confidence coefficient corresponding to each prediction intention, the second sample data and the corresponding second labeling intention; and training a second intention recognition model according to the third training sample set. The implementation mode can fully utilize the historical data used for establishing the man-machine conversation robot, reduce the cost of the newly-built man-machine conversation robot, and promote the intention recognition effect of the newly-built man-machine conversation robot under the condition of a small number of labeling samples.
Description
Technical Field
The present application relates to the field of computer technology, and in particular, to the field of natural language processing and deep learning, and in particular, to a method, apparatus, device, and storage medium for training a model.
Background
Intent recognition is one of the core functions of man-machine conversation robots, typically implemented using an intent recognition model. While the effectiveness of the intent recognition model is heavily dependent on the amount and quality of the training data. The more the number of samples in the training data is, the higher the labeling quality is, and the better the obtained intention recognition model effect is. The manually marked intention dialogue sample has high data quality, but is difficult to form a scale due to high marking cost.
With the rise of man-machine conversation robots, various man-machine conversation robots are established for different applications, and the intention of the man-machine conversation robots is basically customized in depth on application scenes. Therefore, the man-machine conversation robots of different application scenes have the situations of different intention classification granularity, different intention names and the like. Thus, when a new application scene is faced, it is difficult to multiplex the data of the man-machine interaction robot before establishing.
Disclosure of Invention
A method, apparatus, device, and storage medium for training a model are provided.
According to a first aspect, there is provided a method for training a model, comprising: acquiring a first training sample set and a second training sample set, wherein the first training sample set comprises first sample data and corresponding first labeling intention, and the second training sample set comprises second sample data and corresponding second labeling intention; acquiring a trained first intention recognition model and a second intention recognition model to be trained; predicting the first sample data by using a first intention recognition model, and determining the confidence coefficient corresponding to each prediction intention; determining a third training sample set according to the first labeling intention, the confidence coefficient corresponding to each prediction intention, the second sample data and the corresponding second labeling intention; and training a second intention recognition model according to the third training sample set.
According to a second aspect, there is provided an apparatus for training a model, comprising: a first acquisition unit configured to acquire a first training sample set and a second training sample set, wherein the first training sample set includes first sample data and a corresponding first labeling intention, and the second training sample set includes second sample data and a corresponding second labeling intention; a second acquisition unit configured to acquire a trained first intention recognition model and a second intention recognition model to be trained; a confidence determining unit configured to predict the first sample data using the first intention recognition model, and determine a confidence corresponding to each predicted intention; the sample generation unit is configured to determine a third training sample set according to the first labeling intention, the confidence corresponding to each prediction intention, the second sample data and the corresponding second labeling intention; the first model training unit is configured to train the second intention recognition model according to the third training sample set.
According to a third aspect, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in the first aspect.
According to a fourth aspect, there is provided a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method as described in the first aspect.
According to the technology, the historical data used for building the man-machine conversation robot can be fully utilized, the cost of the newly built man-machine conversation robot is reduced, and the intention recognition effect of the newly built man-machine conversation robot is improved under the condition of a small number of labeling samples.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The drawings are for better understanding of the present solution and do not constitute a limitation of the present application. Wherein:
FIG. 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;
FIG. 2 is a flow chart of one embodiment of a method for training a model according to the present application;
FIG. 3 is a schematic illustration of one application scenario of a method for training a model according to the present application;
FIG. 4 is a flow chart of another embodiment of a method for training a model according to the present application;
FIG. 5 is a flow chart of yet another embodiment of a method for training a model according to the present application;
FIG. 6 is a schematic structural view of one embodiment of an apparatus for training a model according to the present application;
FIG. 7 is a block diagram of an electronic device for implementing a method for training a model in accordance with an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present application to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It should be noted that, in the case of no conflict, the embodiments and features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
FIG. 1 illustrates an exemplary system architecture 100 to which embodiments of the methods for training a model or apparatus for training a model of the present application may be applied.
As shown in fig. 1, a system architecture 100 may include intelligent terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is the medium used to provide communication links between the intelligent terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with the server 105 via the network 104 using the intelligent terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as a voice recognition class application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, etc., may be installed on the smart terminal devices 101, 102, 103.
The intelligent terminal equipment 101, 102, 103 may be hardware or software. When the smart terminal devices 101, 102, 103 are hardware, they may be various electronic devices with voice recognition functions, including but not limited to smart phones, smart speakers, smart robots, etc. When the intelligent terminal apparatuses 101, 102, 103 are software, they can be installed in the above-listed electronic apparatuses. Which may be implemented as multiple software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein.
The server 105 may be a server providing various services, such as a background server providing a man-machine conversation robot for the intelligent terminal devices 101, 102, 103. The background server may perform processing such as analysis on data such as voice received by the terminal devices 101, 102, 103, and feed back a processing result (e.g., response data) to the intelligent terminal devices 101, 102, 103.
The server 105 may be hardware or software. When the server 105 is hardware, it may be implemented as a distributed server cluster formed by a plurality of servers, or as a single server. When server 105 is software, it may be implemented as a plurality of software or software modules (e.g., to provide distributed services), or as a single software or software module. The present invention is not particularly limited herein.
It should be noted that the method for training a model provided in the embodiment of the present application is generally performed by the server 105. Accordingly, the means for training the model is typically provided in the server 105.
It should be understood that the number of intelligent terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of intelligent terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of one embodiment of a method for training a model according to the present application is shown. The method for training a model of the present embodiment includes the steps of:
In this embodiment, an executing body (e.g., the server 105 shown in fig. 1) of the method for training a model may acquire a first training sample set and a second training sample set. The first training sample set comprises first sample data and corresponding first labeling intention, and the second training sample set comprises second sample data and corresponding second labeling intention. Here, the sample data in the first training sample set may be sample data for a new field, and the number thereof may be relatively small. Because the manual annotation data has long consumption period and high cost, only a small amount of annotation data is acquired here, so that the cost is reduced and the efficiency is improved. The second training sample set may include general sample data or historical sample data used in other fields, and the amount of sample data included in the second training sample set may be very large. The sample data may be a plurality of dialogues or text. Each sample data corresponds to a labeling intent. The annotation intent may include audiovisual requirements, search requirements, and the like.
It is to be appreciated that the first training sample set and the second training sample set may include partially overlapping data, and as such, the first annotation intent may partially overlap the second annotation intent.
In this embodiment, the execution subject may further acquire a trained first intent recognition model and a second intent recognition model to be trained. Here, the first intention recognition model may be a general intention recognition model, or may be an intention recognition model of other application fields. The second intent recognition model may be an intent recognition model applied to a new field. It can be appreciated that the accuracy of the first intent recognition model is low when applied to the new field.
And 203, predicting the first sample data by using the first intention recognition model, and determining each prediction intention and the corresponding confidence.
In this embodiment, the first intention recognition model may be used to predict the sample data, i.e. sample data in the first sample training sample set is input into the first intention recognition model. Confidence corresponding to each prediction intention of the sample data can be obtained. For example, for sample data a, the results of the first intent recognition model output may include: the confidence level of intent 1 is 0.2, the confidence level of intent 2 is 0.4, the confidence level of intent 3 is 0.3, and the confidence level of intent 4 is 0.3. For sample data B, the results of the first intent recognition model output may include: the intent 5 confidence is 0.7. Each predicted intent may be considered herein as an intent tag of the first sample data.
Conventional supervised learning typically assumes that each sample data is associated with a unique one of the tags. However, in many practical tasks, one piece of data typically has multiple tags. For example, in text classification, if documents are classified, the olympics belong to both business and sports; in image annotation, images in Paris scenes are tied to both the tower and the sky. In this embodiment, each intention label corresponding to the first sample data may be obtained by the first intention recognition model.
It is understood that the number of predicted intents obtained is at least one. The first labeling intent may or may not be included in each predicted intent.
In this embodiment, after obtaining the confidence degrees corresponding to the prediction intentions, the first labeling intention, the second sample data and the corresponding second labeling intention may be combined to determine the third training sample set. Specifically, the prediction intent with the highest confidence level may be determined first. And then, taking out second sample data corresponding to a second labeling intention, which is the same as the predicted intention, in the second training sample set. And modifying the second labeling intention of the extracted sample data into the first labeling intention to obtain a third training sample set. For example, the first labeling intent is intent 1, and the confidence corresponding to each prediction intent is: the confidence level of intent 1 is 0.2, the confidence level of intent 2 is 0.4, the confidence level of intent 3 is 0.3, and the confidence level of intent 4 is 0.3. The confidence level of fig. 2 in the predicted intent may be determined to be highest. Sample data corresponding to fig. 2 in the second training sample set may then be retrieved. The labeling intention of the fetched sample data is modified to fig. 1. Thus, new sample data corresponding to fig. 1, i.e., a third training sample set, is obtained. Here, the third training sample set is an extension of the first training sample set.
After the third set of training samples is obtained, it may be used to train a second intent recognition model. It is understood that each sample data in the third training sample set corresponds to at least one intention label. Thus, the result output by the second intent recognition model also includes at least one intent tag.
Referring to fig. 3, a schematic diagram of an application scenario of the method for training a model of the present application is shown. In the application scenario of fig. 3, the server 301 acquires history labeling data applied to the medical field, the building field, and the financial field from the database 302. Meanwhile, the server 301 acquires a small amount of annotation data applied to a new field (legal field) from the terminal device 303. Through the processing of steps 202 to 205, training is performed to obtain an intention recognition model applied to the legal field. The server 301 may return the above-described intent recognition model to the terminal device 303 so that the technician may apply the above-described intent recognition model to the human-machine conversation robot.
The method for training the model provided by the embodiment of the application can fully utilize the historical data used for building the man-machine conversation robot, reduce the cost of the newly built man-machine conversation robot, and promote the intention recognition effect of the newly built man-machine conversation robot under the condition of a small number of labeling samples.
With continued reference to fig. 4, a flow 400 of another embodiment of a method for training a model according to the present application is shown. As shown in fig. 4, the method for training a model of the present embodiment may include the steps of:
In this embodiment, the first training sample set may include a plurality of subsets, and sample data in each subset corresponds to the same labeling intention. Likewise, the second training sample set may also include multiple subsets, with sample data in each subset corresponding to the same labeling intent.
In this embodiment, the execution body may first process the sample data in the second training sample set. The above-described processing may include merging, deleting, modifying, and the like. Specifically, the execution body may consider the number of second sample data corresponding to each second labeling intention when performing the processing. For example, the second labeling intent and the corresponding second sample data whose number is smaller than the preset threshold are deleted.
In some alternative implementations of the present embodiment, the execution body may process the second set of training samples by:
step 4021, selecting a first training sample subset and a second training sample subset from the second training sample set.
In this implementation, the first training sample subset and the second training sample subset may be selected from the second training sample set first. The first training sample subset and the second training sample subset are different in second labeling intention, and the number of second sample data in the first training sample subset is larger than that in the second training sample subset. That is, the first training sample subset and the second training sample subset are sets of second sample data corresponding to different second labeling intents. And, the number of second sample data in the first training sample subset is greater.
In step 4022, a number of common sample data for the first subset of training samples and the second subset of training samples is determined.
After the first subset of training samples and the second subset of training samples are selected, a number of common sample data for the first subset of training samples and the second subset of training samples may be determined. Here, the common sample data means second sample data existing in both the first training sample subset and the second training sample subset. The execution body may count the common sample data to determine the number thereof.
In step 4023, whether the preset condition is satisfied is determined according to the number of the common sample data and the number of the second sample data in the second training sample subset.
Next, the execution body may share the number of sample data, the number of samples in the second training sample subset, and determine whether the preset condition is satisfied. The preset conditions may include, but are not limited to: the number of samples in the second subset of training samples is greater than or equal to a certain number threshold, and the number of common sample data is greater than or equal to a certain number threshold.
In some alternative implementations, it may be determined whether the preset condition is met by the following steps, not shown in fig. 4: determining a ratio of the number of common sample data to the number of second sample data in the second subset of training samples; in response to determining that the number of second sample data in the second subset of training samples is less than the first preset number threshold and the ratio is greater than the first preset ratio threshold, or the ratio is greater than the second preset ratio threshold, it is determined that the preset condition is met.
In this implementation, the execution body may first determine a ratio of the number of common sample data to the number of second sample data in the second training sample subset. The preset condition is considered satisfied if the number of second sample data in the second subset of training samples is greater than a first preset number threshold (e.g., 50) while the ratio is greater than a first preset ratio threshold. Or the ratio is larger than a second preset ratio threshold value, and the preset condition is determined to be met.
In step 4024, in response to the preset condition being met, modifying the second labeling intention corresponding to the second training sample subset to the second labeling intention corresponding to the first training sample subset.
If the preset condition is determined to be met, the execution body can modify the second labeling intention corresponding to the second training sample subset into the second labeling intention corresponding to the first training sample subset. In this way, a reduction and merging of the second sample data in the second training sample set may be achieved.
In some alternative implementations of the present embodiment, the above-described process may further include the following steps, not shown in fig. 4: deleting the training sample subset with the second sample data less than the second preset number threshold.
In this implementation manner, if the number of second sample data included in a certain subset in the second training sample set is too small, the subset is considered to be unavailable, and the second sample data in the subset and the corresponding second labeling intention may be directly deleted.
Through the processing, the sample data corresponding to each labeling intention in the second training sample set can be more, and the method is convenient for subsequent use.
The executing body may train the initial intent recognition model with the processed second training sample set to obtain the first intent recognition model. Specifically, the execution subject may use the second sample data as an input of the initial intention recognition model, and use the corresponding second labeling intention as an output of the initial intention recognition model. Here, the initial intention recognition model may be an initialized intention recognition model, that is, an intention recognition model in which parameters are initialized values, or a pre-trained intention recognition model.
It should be noted that, the first steps 402 to 403 may be performed by the execution body of the method for training a model of the present application, or may be implemented by other electronic devices. If implemented by other electronic devices, the other electronic devices may send the first intent recognition model to the executing body.
In this embodiment, the second intention recognition model may be an intention recognition model having the same structure as the first intention recognition model but different parameters, and the second intention recognition model may be an intention recognition model having a different structure from the first intention recognition model. The domain to which the second intent recognition model is applied may be different from the domain to which the first intent recognition model is applied.
And step 405, predicting the first sample data by using the first intention recognition model, and determining the confidence corresponding to each prediction intention.
The principle of steps 405 to 407 is similar to that of steps 203 to 205 and will not be described here again.
According to the method for training the model, provided by the embodiment of the application, the intention recognition model can be trained by using the history labeling data, and then a new intention recognition model is trained based on the intention recognition model, so that the utilization rate of the history labeling data is improved.
With continued reference to fig. 5, a flow 500 of another embodiment of a method for training a model according to the present application is shown. As shown in fig. 5, the method for training a model of the present embodiment may include the steps of:
In step 503, the first intention recognition model is used to predict the first sample data, and the confidence corresponding to each prediction intention is determined.
In this embodiment, the execution subject may determine, from the second training sample set, a second labeling intention similar to the first labeling intention according to the first labeling intention and the confidence degrees corresponding to the respective prediction intentions. Specifically, the executing body may search for a second predicted intention with a confidence coefficient greater than a preset threshold in the training sample set, and use the searched second labeling intention as a second labeling intention similar to the first labeling intention.
In some alternative implementations of the present embodiment, the executing entity may determine a second labeling intent similar to the first labeling intent by: for each first labeling intention, sorting the prediction intents according to the confidence level of the prediction intents of the first sample data corresponding to the first labeling intention from high to low; and taking the second labeling intention related to the first preset number of predicted intents in the sequence in the second training sample set as a second labeling intention similar to the first labeling intention.
In this implementation, for each first labeling intention, the execution subject may first determine first sample data corresponding to the first labeling intention, and then determine a confidence level of each prediction intention of each first sample data. The confidence levels are ranked in order of high to low for each predicted intent. The pre-set number (N) of predicted intents is then taken. And determining first N second labeling intents related to the prediction intents, and finally, taking the second labeling intents as second labeling intents similar to the first labeling intents. Here, correlation means the same as or similar to the first N prediction intents.
After determining the second labeling intents which are close to each other, the execution body can select part of second sample data from the second sample data corresponding to the second labeling intents. When selecting, the data can be selected randomly, or the data of multiple rounds of conversational samples can be selected preferentially. The number of the second sample data corresponding to each second labeling intention may be the same or different.
In some optional implementations of this embodiment, the execution body may select a portion of the second sample data by: for each determined second labeling intention, selecting second sample data from second sample data corresponding to the second labeling intention according to the ratio of the confidence degrees corresponding to the prediction intention related to the second labeling intention and the sum of the confidence degrees corresponding to the prediction intentions.
In this embodiment, the execution subject may first calculate the sum of the confidence levels of the prediction intentions, denoted as P. The ratio of the confidence level of each predicted intention to the sum is then calculated. Then, the execution body may calculate a sum of the second sample data corresponding to each second labeling intention, denoted as M. For each determined second labeling intent, the execution subject may select sample data according to a ratio corresponding to a predicted intent to which the second labeling intent relates. Specifically, the number of second sample data is selected from the second sample data corresponding to the second labeling intention to be equal to p×m/P. Where p is the confidence level of the predicted intent associated with the second labeling intent.
After the second sample data is selected, the execution body may combine the second sample data with the first training sample set to determine a third training sample set. Specifically, the execution body may use the second labeling intention and the first training sample set corresponding to the selected second sample data as the third training sample set.
In some alternative implementations of the present embodiment, the executing entity may determine the third set of training samples by: modifying the second labeling intent of the selected second sample data to a similar first labeling intent; and determining a third training sample set according to the labeling intention corresponding to the modified second sample data and the first training sample set.
In this implementation, the execution body may modify the second labeling intent of the selected second sample data to a first labeling intent that is similar to the second labeling intent. In this way, the sample data corresponding to the first labeling intention is increased, and the third training sample set is obtained by combining the first sample data corresponding to each first labeling intention in the first training sample set. It will be appreciated that the annotation intent included in the third training sample set is the same as the annotation intent in the first training sample set and thus may be used for training of the second intent recognition model.
The method for training the model provided by the embodiment of the application can utilize the historical labeling data to expand a small amount of labeling data in a new field, and improves the utilization rate of the historical labeling data.
With further reference to fig. 6, as an implementation of the method shown in the foregoing figures, the present application provides an embodiment of an apparatus for training a model, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus is particularly applicable to various electronic devices.
As shown in fig. 6, the training model apparatus 600 of the present embodiment includes: a first acquisition unit 601, a second acquisition unit 602, a confidence determination unit 603, a sample generation unit 604, and a first model training unit 605.
The first obtaining unit 601 is configured to obtain a first training sample set and a second training sample set. The first training sample set comprises first sample data and corresponding first labeling intention, and the second training sample set comprises second sample data and corresponding second labeling intention.
A second acquisition unit 602 configured to acquire the trained first intent recognition model and a second intent recognition model to be trained.
A confidence determining unit 603 configured to predict the first sample data using the first intention recognition model, and determine a confidence corresponding to each prediction intention;
the sample generating unit 604 is configured to determine the third training sample set according to the first labeling intention, the confidence corresponding to each prediction intention, and the second sample data and the corresponding second labeling intention.
The first model training unit 605 is configured to train the second intention recognition model on the basis of the third set of training samples.
In some alternative implementations of the present embodiment, the apparatus 600 may further include: a sample processing unit and a second model training unit.
And the sample processing unit is configured to process the second training sample set according to the second sample data corresponding to each second labeling intention.
And the second model training unit is configured to train the initial intention recognition model by using the processed second training sample set to obtain a first intention recognition model.
In some alternative implementations of the present embodiment, the sample processing unit may further include: the device comprises a subset determining module, a common sample determining module, a judging module and an intention modifying module.
The sub-set determining module is configured to select a first training sample sub-set and a second training sample sub-set from the second training sample set, wherein the first training sample sub-set is different from a second labeling intention corresponding to the second training sample sub-set, and the number of second sample data in the first training sample sub-set is larger than that of second sample data in the second training sample sub-set;
a common sample determination module configured to determine a number of common sample data for the first subset of training samples and the second subset of training samples;
the judging module is configured to determine whether a preset condition is met according to the number of the shared sample data and the number of the second sample data in the second training sample subset;
the intention modifying module is configured to modify the second labeling intention corresponding to the second training sample subset into the second labeling intention corresponding to the first training sample subset in response to the preset condition being met.
In some optional implementations of this embodiment, the determination module is further configured to: determining a ratio of the number of common sample data to the number of second sample data in the second subset of training samples; in response to determining that the number of second sample data in the second subset of training samples is less than the first preset number threshold and the ratio is greater than the first preset ratio threshold, or the ratio is greater than the second preset ratio threshold, it is determined that the preset condition is met.
In some optional implementations of this embodiment, the sample processing unit may further include a deletion module, not shown in fig. 6, configured to delete a subset of training samples for which the number of second sample data is less than the second preset number threshold.
In some optional implementations of the present embodiment, the sample generation unit may further include: the device comprises a similar intention determining module, a sample data selecting module and a sample generating module.
The similarity intention determining module is configured to determine a second labeling intention similar to the first labeling intention from the second training sample set according to the first labeling intention and the confidence corresponding to each prediction intention.
The sample data selecting module is configured to select part of second sample data from the second sample data corresponding to the determined second labeling intention.
The sample generation module is configured to determine a third training sample set according to the selected second sample data and the first training sample set.
In some optional implementations of the present embodiment, the similarity intent determination module is further configured to: for each first labeling intention, sorting the prediction intents according to the confidence level of the prediction intents of the first sample data corresponding to the first labeling intention from high to low; and taking the second labeling intention related to the first preset number of predicted intents in the sequence in the second training sample set as a second labeling intention similar to the first labeling intention.
In some optional implementations of the present embodiment, the sample data selection module is further configured to: for each determined second labeling intention, selecting second sample data from second sample data corresponding to the second labeling intention according to the ratio of the confidence degrees corresponding to the prediction intention related to the second labeling intention and the sum of the confidence degrees corresponding to the prediction intentions.
In some optional implementations of the present embodiment, the sample generation module is further configured to: modifying the second labeling intent of the selected second sample data to a similar first labeling intent; and determining a third training sample set according to the labeling intention corresponding to the modified second sample data and the first training sample set.
It should be understood that the first acquisition unit 601 to the first model training unit 605 described in the apparatus 600 for training a model correspond to the respective steps in the method described with reference to fig. 2, respectively. Thus, the operations and features described above with respect to the method for training a model are equally applicable to the apparatus 600 and the units contained therein, and are not described in detail herein.
According to embodiments of the present application, an electronic device and a readable storage medium are also provided.
As shown in fig. 7, a block diagram of an electronic device performing a method for training a model according to an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the application described and/or claimed herein.
As shown in fig. 7, the electronic device includes: one or more processors 701, memory 702, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 701 is illustrated in fig. 7.
The memory 702 is used as a non-transitory computer readable storage medium, and may be used to store a non-transitory software program, a non-transitory computer executable program, and modules, such as program instructions/modules (e.g., the first acquisition unit 601, the second acquisition unit 602, the confidence determination unit 603, the sample generation unit 604, and the first model training unit 605 shown in fig. 6) corresponding to the method for training a model in the embodiments of the present application. The processor 701 executes various functional applications of the server and data processing by running non-transitory software programs, instructions, and modules stored in the memory 702, i.e., implements the methods for training the model described in the method embodiments above.
The electronic device performing the method for training a model may further comprise: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected by a bus or otherwise, in fig. 7 by way of example.
The input device 703 may receive input numeric or character information and generate key signal inputs related to performing user settings and function controls of the electronic device used to train the model, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, and the like. The output device 704 may include a display apparatus, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASIC (application specific integrated circuit), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
According to the technical scheme of the embodiment of the application, the historical data used by the man-machine conversation robot can be fully utilized, the cost of the newly built man-machine conversation robot is reduced, and the intention recognition effect of the newly built man-machine conversation robot is improved under the condition of a small number of labeling samples.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, provided that the desired results of the technical solutions disclosed in the present application can be achieved, and are not limited herein.
The above embodiments do not limit the scope of the application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application are intended to be included within the scope of the present application.
Claims (20)
1. A method for training a model, comprising:
acquiring a first training sample set and a second training sample set, wherein the first training sample set comprises first sample data and corresponding first labeling intention, and the second training sample set comprises second sample data and corresponding second labeling intention;
acquiring a trained first intention recognition model and a second intention recognition model to be trained;
Predicting the first sample data by using the first intention recognition model, and determining the confidence coefficient corresponding to each prediction intention;
determining a third training sample set according to the first labeling intention, the confidence coefficient corresponding to each prediction intention, the second sample data and the corresponding second labeling intention; the determining the third training sample set according to the first labeling intention, the confidence corresponding to each prediction intention, the second sample data and the corresponding second labeling intention comprises: determining a prediction intention with highest confidence, taking out second sample data corresponding to a second labeling intention, which is the same as the prediction intention with highest confidence, in the second training sample set, and modifying the second labeling intention of the taken second sample data into the first labeling intention to obtain a third training sample set;
training the second intent recognition model according to the third training sample set.
2. The method of claim 1, wherein the method further comprises:
processing the second training sample set according to second sample data corresponding to each second labeling intention;
And training an initial intention recognition model by using the processed second training sample set to obtain the first intention recognition model.
3. The method according to claim 2, wherein said processing the second training sample set according to the second sample data corresponding to each of the second labeling intents includes:
selecting a first training sample subset and a second training sample subset from the second training sample set, wherein the first training sample subset and a second labeling intention corresponding to the second training sample subset are different, and the number of second sample data in the first training sample subset is larger than that of second sample data in the second training sample subset;
determining a number of common sample data for the first subset of training samples and the second subset of training samples;
determining whether a preset condition is met according to the number of the shared sample data and the number of the second sample data in the second training sample subset;
and responding to the satisfaction of the preset condition, and modifying the second labeling intention corresponding to the second training sample subset into the second labeling intention corresponding to the first training sample subset.
4. A method according to claim 3, wherein said determining whether a preset condition is met based on the number of common sample data, the number of samples in the second subset of training samples, comprises:
determining a ratio of the number of common sample data to the number of second sample data in the second subset of training samples;
and determining that the preset condition is met in response to determining that the number of second sample data in the second training sample subset is less than a first preset number threshold and the ratio is greater than a first preset proportion threshold or the ratio is greater than a second preset proportion threshold.
5. The method according to claim 2, wherein said processing the second training sample set according to the second sample data corresponding to each of the second labeling intents includes:
deleting the training sample subset with the second sample data less than the second preset number threshold.
6. The method of claim 1, wherein the determining a third training sample set from the first labeling intent, the confidence level corresponding to each prediction intent, and the second sample data and the corresponding second labeling intent comprises:
Determining a second labeling intention similar to the first labeling intention from the second training sample set according to the first labeling intention and the confidence degrees corresponding to the prediction intentions;
selecting partial second sample data from the second sample data corresponding to the determined second labeling intention;
and determining the third training sample set according to the selected second sample data and the first training sample set.
7. The method of claim 6, wherein the determining a second labeling intent similar to the labeling intent from the second training sample set according to the first labeling intent and the confidence corresponding to each predicted intent comprises:
for each first labeling intention, sorting the prediction intents according to the confidence level of the prediction intents of the first sample data corresponding to the first labeling intention from high to low;
and taking second labeling intents in the second training sample set, which are related to the preset number of predicted intents in the sequence, as second labeling intents similar to the first labeling intents.
8. The method of claim 6, wherein the selecting a portion of the second sample data from the second sample data corresponding to the determined second labeling intent comprises:
For each determined second labeling intention, selecting second sample data from second sample data corresponding to the second labeling intention according to the ratio of the confidence degrees corresponding to the prediction intention related to the second labeling intention and the sum of the confidence degrees corresponding to the prediction intentions.
9. The method of claim 6, wherein the determining the third set of training samples from the selected second set of sample data and the first set of training samples comprises:
modifying the second labeling intent of the selected second sample data to a similar first labeling intent;
and determining the third training sample set according to the labeling intention corresponding to the modified second sample data and the first training sample set.
10. An apparatus for training a model, comprising:
a first acquisition unit configured to acquire a first training sample set and a second training sample set, wherein the first training sample set includes first sample data and a corresponding first labeling intention, and the second training sample set includes second sample data and a corresponding second labeling intention;
a second acquisition unit configured to acquire a trained first intention recognition model and a second intention recognition model to be trained;
A confidence determining unit configured to predict the first sample data using the first intention recognition model, and determine a confidence corresponding to each predicted intention;
the sample generation unit is configured to determine a third training sample set according to the first labeling intention, the confidence corresponding to each prediction intention, the second sample data and the corresponding second labeling intention; the sample generation unit is further configured to: determining a prediction intention with highest confidence, taking out second sample data corresponding to a second labeling intention, which is the same as the prediction intention with highest confidence, in the second training sample set, and modifying the second labeling intention of the taken second sample data into the first labeling intention to obtain a third training sample set;
a first model training unit configured to train the second intent recognition model according to the third set of training samples.
11. The apparatus of claim 10, wherein the apparatus further comprises:
the sample processing unit is configured to process the second training sample set according to second sample data corresponding to each second labeling intention;
And a second model training unit configured to train an initial intention recognition model by using the processed second training sample set to obtain the first intention recognition model.
12. The apparatus of claim 11, wherein the sample processing unit comprises:
the sub-set determining module is configured to select a first training sample sub-set and a second training sample sub-set from the second training sample set, wherein the first training sample sub-set is different from a second labeling intention corresponding to the second training sample sub-set, and the number of second sample data in the first training sample sub-set is larger than that of second sample data in the second training sample sub-set;
a common sample determination module configured to determine a number of common sample data for the first subset of training samples and the second subset of training samples;
the judging module is configured to determine whether a preset condition is met according to the number of the common sample data and the number of the second sample data in the second training sample subset;
and the intention modifying module is configured to respond to the satisfaction of the preset condition and modify the second labeling intention corresponding to the second training sample subset into the second labeling intention corresponding to the first training sample subset.
13. The apparatus of claim 12, wherein the determination module is further configured to:
determining a ratio of the number of common sample data to the number of second sample data in the second subset of training samples;
and determining that the preset condition is met in response to determining that the number of second sample data in the second training sample subset is less than a first preset number threshold and the ratio is greater than a first preset proportion threshold or the ratio is greater than a second preset proportion threshold.
14. The apparatus of claim 12, wherein the sample processing unit comprises:
and a deleting module configured to delete the training sample subset for which the number of second sample data is smaller than the second preset number threshold.
15. The apparatus of claim 10, wherein the sample generation unit comprises:
the similar intention determining module is configured to determine a second labeling intention similar to the first labeling intention from the second training sample set according to the first labeling intention and the confidence corresponding to each prediction intention;
a sample data selecting module configured to select a part of second sample data from the second sample data corresponding to the determined second labeling intention;
A sample generation module configured to determine the third set of training samples from the selected second sample data and the first set of training samples.
16. The apparatus of claim 15, wherein the similar intent determination module is further configured to:
for each first labeling intention, sorting the prediction intents according to the confidence level of the prediction intents of the first sample data corresponding to the first labeling intention from high to low;
and taking second labeling intents in the second training sample set, which are related to the preset number of predicted intents in the sequence, as second labeling intents similar to the first labeling intents.
17. The apparatus of claim 15, wherein the sample data selection module is further configured to:
for each determined second labeling intention, selecting second sample data from second sample data corresponding to the second labeling intention according to the ratio of the confidence degrees corresponding to the prediction intention related to the second labeling intention and the sum of the confidence degrees corresponding to the prediction intentions.
18. The apparatus of claim 15, wherein the sample generation module is further configured to:
Modifying the second labeling intent of the selected second sample data to a similar first labeling intent;
and determining the third training sample set according to the labeling intention corresponding to the modified second sample data and the first training sample set.
19. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein, the liquid crystal display device comprises a liquid crystal display device,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.
20. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010524463.3A CN111680517B (en) | 2020-06-10 | 2020-06-10 | Method, apparatus, device and storage medium for training model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010524463.3A CN111680517B (en) | 2020-06-10 | 2020-06-10 | Method, apparatus, device and storage medium for training model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111680517A CN111680517A (en) | 2020-09-18 |
| CN111680517B true CN111680517B (en) | 2023-05-16 |
Family
ID=72454836
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010524463.3A Active CN111680517B (en) | 2020-06-10 | 2020-06-10 | Method, apparatus, device and storage medium for training model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680517B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112529624B (en) * | 2020-12-15 | 2024-01-09 | 北京百度网讯科技有限公司 | Method, device, equipment and storage medium for generating business prediction model |
| CN112541739B (en) * | 2020-12-15 | 2022-04-15 | 平安科技（深圳）有限公司 | Method, device, equipment and medium for testing question-answer intention classification model |
| CN112669816B (en) * | 2020-12-24 | 2023-06-02 | 北京有竹居网络技术有限公司 | Model training method, voice recognition method, device, medium and equipment |
| CN112560992B (en) * | 2020-12-25 | 2023-09-01 | 北京百度网讯科技有限公司 | Method, device, electronic equipment and storage medium for optimizing picture classification model |
| CN112966108B (en) * | 2021-03-08 | 2024-03-12 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for detecting data and training classification model |
| CN113569546A (en) * | 2021-06-16 | 2021-10-29 | 上海淇玥信息技术有限公司 | Intention labeling method and device and electronic equipment |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920622A (en) * | 2018-06-29 | 2018-11-30 | 北京奇艺世纪科技有限公司 | A kind of training method of intention assessment, training device and identification device |
| CN109063221A (en) * | 2018-11-02 | 2018-12-21 | 北京百度网讯科技有限公司 | Query intention recognition methods and device based on mixed strategy |
| CN109086814A (en) * | 2018-07-23 | 2018-12-25 | 腾讯科技（深圳）有限公司 | A kind of data processing method, device and the network equipment |
| CN109522556A (en) * | 2018-11-16 | 2019-03-26 | 北京九狐时代智能科技有限公司 | A kind of intension recognizing method and device |
| CN110457449A (en) * | 2019-07-05 | 2019-11-15 | 阿里巴巴集团控股有限公司 | Method, apparatus, equipment and the storage medium of on-line training model |
| CN110990576A (en) * | 2019-12-24 | 2020-04-10 | 用友网络科技股份有限公司 | Intention classification method based on active learning, computer device and storage medium |
| CN111143561A (en) * | 2019-12-26 | 2020-05-12 | 北京百度网讯科技有限公司 | Intention recognition model training method and device and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10832003B2 (en) * | 2018-08-26 | 2020-11-10 | CloudMinds Technology, Inc. | Method and system for intent classification |
-
2020
- 2020-06-10 CN CN202010524463.3A patent/CN111680517B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920622A (en) * | 2018-06-29 | 2018-11-30 | 北京奇艺世纪科技有限公司 | A kind of training method of intention assessment, training device and identification device |
| CN109086814A (en) * | 2018-07-23 | 2018-12-25 | 腾讯科技（深圳）有限公司 | A kind of data processing method, device and the network equipment |
| CN109063221A (en) * | 2018-11-02 | 2018-12-21 | 北京百度网讯科技有限公司 | Query intention recognition methods and device based on mixed strategy |
| CN109522556A (en) * | 2018-11-16 | 2019-03-26 | 北京九狐时代智能科技有限公司 | A kind of intension recognizing method and device |
| CN110457449A (en) * | 2019-07-05 | 2019-11-15 | 阿里巴巴集团控股有限公司 | Method, apparatus, equipment and the storage medium of on-line training model |
| CN110990576A (en) * | 2019-12-24 | 2020-04-10 | 用友网络科技股份有限公司 | Intention classification method based on active learning, computer device and storage medium |
| CN111143561A (en) * | 2019-12-26 | 2020-05-12 | 北京百度网讯科技有限公司 | Intention recognition model training method and device and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| Mateusz Budnik等."Rethinking deep active learning:Using unlabeled data at model training".《ResearchGate》.2019,全文. * |
| 付博 ; 陈毅恒 ; 邵艳秋 ; 刘挺 ; ."基于用户自然标注的微博文本的消费意图识别".《中文信息学报》.2017,(04),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111680517A (en) | 2020-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111680517B (en) | Method, apparatus, device and storage medium for training model | |
| CN111428008B (en) | Method, apparatus, device and storage medium for training a model | |
| JP7317879B2 (en) | METHOD AND DEVICE, ELECTRONIC DEVICE, STORAGE MEDIUM AND COMPUTER PROGRAM FOR RECOGNIZING VIDEO | |
| CN111105800B (en) | Voice interaction processing method, device, equipment and medium | |
| CN111667056B (en) | Method and apparatus for searching model structures | |
| CN112102448B (en) | Virtual object image display method, device, electronic equipment and storage medium | |
| CN111443801B (en) | Man-machine interaction method, device, equipment and storage medium | |
| CN111460135B (en) | Method and device for generating text abstract | |
| CN114036398B (en) | Content recommendation and ranking model training method, device, equipment and storage medium | |
| CN111241234B (en) | Text classification method and device | |
| EP3944097A1 (en) | Method and apparatus for information processing in user conversation, electronic device and storage medium | |
| CN111078878A (en) | Text processing method, device and equipment and computer readable storage medium | |
| CN111797216B (en) | Search term rewriting method, apparatus, device and storage medium | |
| CN111782785B (en) | Automatic question and answer method, device, equipment and storage medium | |
| CN111582477B (en) | Training method and device for neural network model | |
| CN111158924B (en) | Content sharing method and device, electronic equipment and readable storage medium | |
| CN112580723B (en) | Multi-model fusion method, device, electronic equipment and storage medium | |
| CN112559715B (en) | Attitude identification method, device, equipment and storage medium | |
| CN112232089B (en) | Pre-training method, device and storage medium of semantic representation model | |
| CN112382292A (en) | Voice-based control method and device | |
| CN111639234B (en) | Method and device for mining core entity attention points | |
| CN111680599B (en) | Face recognition model processing method, device, equipment and storage medium | |
| CN115378890B (en) | Information input method, device, storage medium and computer equipment | |
| CN111651988B (en) | Method, apparatus, device and storage medium for training model | |
| CN113792876B (en) | Backbone network generation method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |