JP7443649B2 - Model update method, device, electronic device and storage medium - Google Patents

Description

The present disclosure relates to the field of artificial intelligence technology, and particularly to model updating methods, apparatuses, electronic devices, and storage media in fields such as deep learning and natural language processing.

Currently, deep learning models obtained through training are being used in a variety of scenarios. For example, a search scenario may use a semantic search model to recall candidate results from a large amount of content and a semantic matching model to sort the recalled candidate results. However, there is room for performance improvements in all of these models.

The present disclosure provides model updating methods, apparatus, electronic devices, and storage media.

For the data to be processed in the evaluation data set that belongs to at least two different categories, the corresponding processing results are obtained using the model to be optimized and the tools in the toolset, and the processing results are The method includes comparing the processing effects of the model to be optimized and the tool, and identifying a category to be optimized and a corresponding advantage tool based on the comparison result, wherein the category to be optimized is a tool. is a category in which the processing effect is better than the model to be optimized, and the advantage tool is a category in which the processing effect is better than the model to be optimized, and the advantage tool is a category in which the processing effect is better than the model to be optimized, and the advantage tool is a category in which the processing effect is better than the model to be optimized. A model updating method is provided that includes generating categorical training data and updating the model to be optimized using the training data.

The effect comparison module includes an effect comparison module, an information identification module, a sample generation module, and a model update module; For the data, the model to be optimized and the tools in the toolset are used to obtain corresponding processing results, and based on the processing results, the superiority or inferiority of the processing effects of the model to be optimized and the tools are compared; The information identification module identifies a category to be optimized and a corresponding advantage tool based on the comparison result, and the category to be optimized is a category in which the processing effect of the tool is superior to the model to be optimized; The advantage tool is a tool whose processing effect is better than the model to be optimized, the sample generation module generates training data for the category to be optimized based on the corresponding advantage tool, and the model update module generates training data for the category to be optimized based on the corresponding advantage tool. , a model updating device is provided that updates the model to be optimized using the training data.

at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores a command executable by the at least one processor, and the command is executed by the at least one processor. An electronic device is provided which causes said at least one processor to perform a method as described above.

A non-transitory computer-readable storage medium is provided having computer commands stored thereon for causing a computer to perform a method as described above.

A computer program product is provided that includes computer programs/commands that, when executed by a processor, implement a method as described above.

It should be understood that what is described in this section does not label important or essential features of the embodiments of the disclosure or limit the scope of the disclosure. Other features of the disclosure will be readily understood from the following specification.

The drawings are for better understanding of the technical solution and are not limited to the present application. In the drawing,
2 is a flowchart of an embodiment of a model update method described in this disclosure. 1 is a schematic diagram of the overall implementation process of the model update method described in this disclosure; FIG. 3 is a schematic diagram of a configuration of an embodiment 300 of a model update apparatus described in the present disclosure. FIG. 1 shows a schematic block diagram of an electronic device 400 capable of implementing embodiments of the present disclosure.

Hereinafter, exemplary embodiments of the present application will be described based on the drawings. Various details of the embodiments of the present application are included to facilitate understanding and are to be considered as illustrative only. Accordingly, those skilled in the art will appreciate that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, in the interest of clarity, descriptions of well-known functions and structures are omitted in the following description.

Additionally, the term "and/or" herein is an association relationship that simply describes related objects, meaning that three relationships may exist. For example, A and/or B can mean three situations: A is present alone, A and B are present simultaneously, and B is present alone. Furthermore, the character "/" in this specification generally means that related objects before and after are in an "or" relationship.

FIG. 1 is a flowchart of an embodiment of the model update method described in this disclosure. As shown in FIG. 1, the following specific implementation methods are included.

In step 101, for data to be processed in the evaluation data set and belonging to at least two different categories, corresponding processing results are obtained using the model to be optimized and the tools in the toolset, respectively; Compare the processing effects of the model and tool to be optimized based on the processing results.

In step 102, based on the comparison results, a category to be optimized, which is a category in which the processing effect of the tool is better than the model to be optimized, and a corresponding tool, which is a category in which the processing effect of the tool is better than the model to be optimized, are determined. Identify the advantage tools that are.

In step 103, training data for the category to be optimized is generated based on the corresponding advantage tool.

In step 104, the generated training data is used to update the model to be optimized.

According to the technical proposal described in the above method embodiment, the problem in the model to be optimized, i.e. the category to be optimized, is identified through the tools in the toolset, and the training data of the category to be optimized accordingly. The model to be optimized can be updated based on the generated training data. That is, we proposed a data-driven directional model optimization method that can directionally solve problems in the model to be optimized and further improve model performance.

The technical solution described in the method embodiment can be applied to the field of artificial intelligence such as deep learning and natural language processing. Specifically, for example, when applied to a search scenario, the model to be optimized is a semantic matching model, the data to be processed is a query input by a user, and the processing result is a query It may be the result of sorting the candidate results recalled by. The different categories refer to different query categories, and may be divided into categories such as questions, keyword searches, web address searches, audio/video playback, etc., as well as games, automobiles, common knowledge, and They may be divided into categories such as medical, and how they are divided into different categories can be determined according to actual needs. For queries in the evaluation data set, for example queries in text format, the corresponding processing results are obtained using the semantic matching model and the tools in the toolset, respectively, and the processing effects of the semantic matching model and tools are evaluated based on the processing results. Furthermore, based on the comparison results, the category to be optimized and the corresponding advantage tool are identified, and training data for the category to be optimized is generated based on the corresponding advantage tool. The training data can be used to update the semantic matching model. After the optimized semantic matching model is obtained as described above, it can be put into practical use. In other words, it is possible to obtain recall candidate results for a query input by the user, sort the recall candidate results using an optimized semantic matching model, and improve the accuracy of the sort results accordingly. I can do it.

In one embodiment of the present disclosure, the toolset includes the following tools: a historical version model of the model to be optimized, a homogeneous model of the model to be optimized, a homogeneous system of the model to be optimized, a word grammar tool, a syntax tool, a knowledge base. Can include one or any combination. The same species indicates that they have the same function.

In practical applications, as many tools as possible can be added to the toolset, based on the idea of ``capturing many advantages broadly''. Accordingly, by including all of the above in the toolset, it is possible to improve the number and type of training data that is subsequently constructed, increasing the effectiveness of model optimization.

Here, assuming that the model to be optimized is a semantic matching model, the history version model of the model to be optimized refers to each version of the semantic matching model obtained in the past. The historical version model has poor overall functionality compared to the model to be optimized, but may perform better in certain or specific categories. For this reason, all historical versions of the model to be optimized may be added to the toolset.

Further, assuming that the model to be optimized is a certain rule-based semantic matching model, the homogeneous model may be another rule-based semantic matching model. Furthermore, it may be a homogeneous system of models to be optimized for various search engines.

Furthermore, if the data to be processed is the query input by the user, ``What is the average housing price in ** (a certain city name)?'', then one of the keywords in the query can be calculated using a word grammar tool, such as ``* *" can be specified. Accordingly, the candidate results for recall are sorted based on the principle that if "**" does not appear in the candidate result, it is likely that the answer is not very relevant and should be ranked lower. can do. Syntax tools, knowledge bases, etc. can also provide similar functionality.

In an embodiment of the present disclosure, the evaluation data set may further include correct processing results corresponding to each piece of data to be processed.

For example, in a search scenario, the data to be processed may be a user-entered query, and the corresponding processing result may be the correct sort order for each candidate result recalled based on the query.

There is no limit to the method of generating the evaluation data set, and there is no limit to the specific number of data to be processed included in the evaluation data set, which may be determined according to actual needs. Furthermore, the data to be processed in the evaluation data set must belong to at least two different categories. The categories of the at least two different categories may also be determined depending on actual needs. For example, if the data to be processed is a query input by a user, one category may be that the query contains rarely seen words.

Further assume that the evaluation data set contains 100 (an enumeration, but in practice it may be much larger) pieces of data to be processed. Each of these data may belong to ten different categories, and each category may contain ten pieces of data to be processed.

It is possible to compare the processing effects of the model to be optimized and the tools in the toolset on the data to be processed in the evaluation data set.

Here, any one tool in the tool set may be a tool that can be operated individually, or a tool that cannot be operated individually. Accordingly, when comparing the processing effects of models and tools to be optimized, different processing methods can be adopted as described below.

1) Tools that can operate individually In an embodiment of the present disclosure, a model to be optimized can be used to obtain a first processing result corresponding to each data to be processed in the evaluation data set. Then, for any one of the tools in the tool set that can operate individually, the second processing result corresponding to each data to be processed in the evaluation data set is obtained using the tool, and the correct processing result, the first processing result, and Based on the second processing results, processing can be performed to determine the superiority or inferiority of the processing effects of the model to be optimized and the tool for different categories.

For example, if the model to be optimized is a semantic matching model, most of the tools in the toolset are independently operable tools, such as historical version models, homogeneous models, word grammar tools, etc. A separately operable tool is one that can be used directly to sort candidate results corresponding to each piece of data to be processed in the evaluation data set. Conversely, if the tool cannot be used directly for a search engine to sort candidate results for each data item in the evaluation data set, then the tool is an inoperable tool.

Accordingly, obtaining a first processing result corresponding to each data to be processed in the evaluation data set using the model to be optimized means This means sorting the candidate results corresponding to the data, and the obtained sorting result is the first processing result. Similarly, obtaining a second processing result corresponding to each data to be processed in the evaluation data set using the tool includes sorting candidate results corresponding to each data to be processed in the evaluation data set using the tool. This means that the obtained sorting result is the second processing result.

Through such processing, it is possible to efficiently and accurately determine the superiority or inferiority of the processing effects of the model to be optimized and the tool by combining the correct processing result and the processing result corresponding to each of the model and tool to be optimized. can.

Specifically, in one embodiment of the present disclosure, for any one category, correct processing results, first processing results, and second processing results corresponding to data to be processed belonging to the category in the evaluation data set, respectively. Based on this, it is possible to judge the processing effectiveness of the model to be optimized and the tool. Accordingly, in response to determining that the processing effect of the tool is superior to the model to be optimized, the category can be set as the category to be optimized and the tool can be set as an advantage tool.

For example, assume that category a contains 10 pieces of data to be processed. For ease of expression, when they are called data to be processed 1 to data to be processed 10, the first processing corresponding to each of data to be processed 1 to data to be processed 10 is determined by comparing with the correct processing result. Among the results, determine the number of correct first processing results, and then determine the second processing results corresponding to each of data to be processed 1 to data to be processed 10 (assumed to be obtained using tool b). Among them, the number of correct second processing results can be determined, and the accuracy rate of each can be obtained and used as an evaluation index of the model to be optimized and tool b. If it is determined that the evaluation index of tool b is better than the evaluation index of the model to be optimized, it can be determined that the processing effect of tool b is superior to that of the model to be optimized. In the opposite case, it can be determined that the processing effect of the model to be optimized is superior to tool b.

What should be explained is that the above evaluation index acquisition method is only an exemplary explanation and is not intended to limit the technical proposal of the present disclosure, and how to specifically set the evaluation index depends on the actual situation. It can be determined as necessary.

Through the above processing, it is possible to judge the processing effects of models and tools to be optimized for different categories, and to identify categories to be optimized and corresponding advantage tools accordingly. can build a good foundation.

2) Tools that cannot be operated individually For any one tool that cannot be operated individually in the tool set, use the tool to obtain a third processing result corresponding to each data to be processed in the evaluation data set, and , for any one of the data to be processed, use the model to be optimized to verify whether the third processing result is correct, and if the verification result is determined to be inaccurate, artificially It is possible to perform a process of obtaining the judgment results obtained by the method, and determining the processing effects of the model to be optimized and the tool on the data to be processed, based on the judgment results obtained by the artificial judgment.

As an example, the model to be optimized is a semantic matching model, and the data to be processed is a query input by a user. For tools such as search engines, it is not possible to directly use such tools to sort candidate results corresponding to each piece of data to be processed in the evaluation data set, so such tools are incapable of operating independently. We have no choice but to work in the opposite direction depending on what happens.

Specifically, for any one query in the evaluation data set, each candidate result of recall which is the search result from the search engine by inputting it into a search engine, and the sorting rank of each candidate result, that is, the third processing result described above. can be obtained. Thereafter, it is possible to verify whether the third processing results for each data to be processed are correct using the model to be optimized. For example, for each third processing result, each candidate result is re-sorted using the model to be optimized, compared whether the re-sorted result matches the third processing result, and if not, the verification result is can be determined to be inaccurate. Accordingly, the parties involved determine which sorting result is more accurate, obtaining an artificial judgment result, and then determining the model to be optimized for each data to be processed based on this artificial judgment result. It is possible to determine the superiority or inferiority of the processing effect of the tool.

In an embodiment of the present disclosure, it is possible to determine the superiority or inferiority of the processing effects of a model to be optimized and a tool for any one data to be processed based on an artificial determination result. , the artificial judgment result determines that the tool is better than the model to be optimized depending on whether the third processing result corresponding to the data to be processed is correct, and in other cases It is determined that the processing effect of the model to be optimized is better than that of the tool, and accordingly, for any one category, the processing effect of the tool is optimized among the data to be processed belonging to that category in the evaluation dataset. In response to determining that the number of data to be processed that is better than the model to be optimized is greater than the number of data to be processed for which the processing effect of the model to be optimized is better than the tool, the corresponding category This may include determining the category to be optimized and the tool as an advantage tool.

Category b contains 10 pieces of data to be processed, and for 5 of these data, the processing effect of the tool (assumed to be tool c) is superior to that of the model to be optimized, and two For the data to be processed, assuming that the processing effect of the model to be optimized is better than the processing effect of the tool, and for the remaining three data to be processed, the processing effect of the model to be optimized and the processing effect of the tool are the same, then the category b can be a category to be optimized, and tool c can be an advantage tool.

As can be seen from the above description, the aspects described in this disclosure are applicable to both individually operable and non-individually operable tools, i.e., have versatility.

The category to be optimized is the category in which the problem exists. The problem can then be solved in a data-driven manner. That is, training data for a category to be optimized can be generated based on the corresponding advantage tool, and the model to be optimized can be updated using the generated training data.

In one embodiment of the present disclosure, for any one category to be optimized, data to be processed belonging to the category to be optimized is obtained from the database, and the number of advantage tools corresponding to the category to be optimized is one. In response, based on the obtained data to be processed, generate training data for the category to be optimized based on the advantage tool, and the number of advantage tools corresponding to the category to be optimized is greater than 1. Depending on the situation, it is possible to perform a process of respectively generating training data for categories to be optimized based on each advantage tool based on the acquired data to be processed.

Assume that there are three tools in the toolset: tool a, tool b, and tool c. In addition, the data to be processed in the evaluation dataset belong to 10 different categories, category 1 to category 10, and the processing effect of tool a is superior in categories 3 to 5 compared to the model to be optimized. Assuming that the processing effect of tool b is better in categories 4 to 6 and the processing effect of tool c is better in categories 5 to 7, categories 3, 4, 5, 6, and 7 are optimal This is a category that should be developed. Regarding category 3, since the number of tool a, which is the corresponding advantage tool, is one, training data for category 3 can be generated based on tool a accordingly. For category 5, the number of corresponding advantage tools including tool a, tool b, and tool c is more than 1, so we generate training data for category 5 based on tool a and based on tool b accordingly. Category 5 training data can be generated, and category 5 training data can be generated based on tool c. Other explanations will be omitted.

There are no restrictions on how the data to be processed of the category to be optimized is acquired from the database; for example, some queries may be randomly extracted from the queries belonging to the category to be optimized in the database. If the category to be optimized is that queries include words that are rarely seen, some queries may be randomly extracted from the queries of the corresponding category, and all of the extracted queries may contain words that are rarely seen. include.

Through the above processing, it is possible to obtain training data generated based on different advantage tools, thereby enriching the content of the training data and improving the optimization effect of the model.

In actual applications, the advantages of each tool may be concentrated in a specific category or several categories, as in category 5 above. Compared to other categories accordingly. The amount of training data corresponding to category 5 increases. In order to avoid the problem that the proportion of the number of corresponding training data varies greatly depending on the category, before updating the optimization model using the generated training data, perform training corresponding to each generated category. By performing processes such as deduplication and sampling on the data, the ratio of training data of different categories in the entire training data set becomes relatively equal, which further improves the optimization effect of the model.

In an embodiment of the present disclosure, when generating training data of a category to be optimized based on the advantage tool for any one advantage tool, the advantage tool is used for each acquired data to be processed. After obtaining the corresponding fourth processing result using the model to be optimized and obtaining the corresponding fifth processing result using the model to be optimized, the fourth processing result and the fifth processing result are selected from among the obtained data to be processed. It is also possible to select data to be processed that satisfies a condition that the processing result does not match, and to generate training data based on the selected data to be processed.

For example, for each acquired query, corresponding candidate results may be acquired. There are no restrictions on how you can obtain it. Thereafter, the candidate results corresponding to each query are sorted using the advantage tool and the model to be optimized, respectively, to obtain a fourth processing result and a fifth processing result corresponding to each query. If the fourth processing result and the fifth processing result corresponding to any one query do not match, that query is treated as the selected data to be processed, and the desired training is performed based on the selected data to be processed. Data can be generated.

Through the above processing, desired data to be processed can be selected accurately and effectively. Furthermore, it has been experimentally proven that better model optimization effects can be obtained by simply generating training data using less data to be processed.

In an embodiment of the present invention, when generating training data based on the selected data to be processed, the selected data to be processed and the fourth processing result for the selected data to be processed and/or Training data can be generated by combining the fifth processing result with the artificial annotation result.

For example, for one selected piece of data to be processed, the corresponding fourth processing result and fifth processing result can be determined manually. When it is determined that the fourth processing result is correct and the fifth processing result is incorrect, training data with a label of 1, that is, a positive sample, is generated using the data to be processed and the fourth processing result. Then, using the data to be processed and the fifth processing result, generate training data whose label is 0, that is, a negative sample, or if both the fourth processing result and the fifth processing result are correct. If it is determined that there is no such difference, one of the processing results can be manually corrected to obtain a positive sample, and a negative sample can be generated based on the uncorrected processing result.

Artificial annotations can improve the accuracy of the generated training data and, in turn, improve the optimization efficiency and effectiveness of the model.

In one embodiment of the present disclosure, in addition to generating training data for the category to be optimized using the advantage tool as described above, training data for the category to be optimized is generated based on the obtained user behavior log of the model to be optimized. It is also possible to generate training data based on artificial evaluation results that are evaluation results, and to update the model to be optimized using the generated training data.

Once the model to be optimized is put into use, it will generally record user behavior logs. On the other hand, the engineers responsible for the effectiveness of the model determine whether the model is effective or not by periodically performing artificial evaluations. For example, if it turns out that for a certain query, the user clicked on the fourth candidate result and spent time reading it, then that candidate result is better than the top three candidate results. Conceivable. Accordingly, the sort order of the candidate results can be modified to generate positive samples, and negative samples can be generated based on the unmodified sort order.

Through the above processing, it is possible to directly discover problems existing in the model to be optimized based on user behavior logs, generate training data accordingly, and further improve the optimization effect of the model.

As mentioned above, FIG. 2 is a schematic diagram of the overall implementation process of the model update method described in this disclosure. As shown in Figure 2, the tools in the toolset are used to discover problems that exist in the model to be optimized, and the tools in the toolset are used to solve problems for the discovered problems, i.e., the corresponding training The data can be generated, the model to be optimized can be updated based on the generated training data, and the process described above can be repeated. The specific implementation can refer to the related explanation above, and will not be described in detail here.

It should be noted that while the method embodiments described above are expressed as a series of combinations of operations for ease of explanation, those skilled in the art will recognize that some steps in accordance with the present application may be performed in other orders. This application is not limited to the order of the operations described, as they may be performed in accordance with or concurrently. Next, those skilled in the art should understand that any embodiments described in the specification constitute preferred embodiments, and that the associated acts and modules are not necessarily required by the present application.

In summary, the method described in the method embodiments of the present disclosure uses various tools to discover problems that exist in the model to be optimized, and further improves the problem by increasing the variety of tools. Coverage can be improved. In addition, all discovered problems are real problems that cannot be solved by the model, and training data can be generated directionally for each problem, improving the optimization effect of the model and improving the performance of the model. .

The above is a description of method embodiments, and the aspects described in this disclosure are further described below through apparatus embodiments.

FIG. 3 is a schematic diagram of a configuration of an embodiment 300 of a model update apparatus described in this disclosure. As shown in FIG. 3, it includes an effect comparison module 301, an information identification module 302, a sample generation module 303, and a model update module 304.

The effect comparison module 301 obtains corresponding processing results for data to be processed in the evaluation data set and belonging to at least two different categories using the model to be optimized and the tools in the toolset, respectively. Then, based on the processing results, the processing effects of the model to be optimized and the tool are compared.

Based on the comparison results, the information identification module 302 identifies a category to be optimized, which is a category in which the processing effect of the tool is better than the model to be optimized, and a corresponding category in which the processing effect is better than the model to be optimized. Identify the tool, Advantage Tool.

The sample generation module 303 generates training data for the category to be optimized based on the corresponding advantage tool.

The model update module 304 updates the model to be optimized using the generated training data.

According to the aspects described in the above embodiments of the apparatus, the tools in the toolset identify the problem that exists in the model to be optimized, i.e. the category to be optimized, and the training data for the category to be optimized accordingly. The model to be optimized can be updated based on the generated training data. In other words, we proposed a data-driven directional model optimization method that can directionally solve problems existing in the model to be optimized, thereby improving the performance of the model.

In one embodiment of the present disclosure, the toolset includes one of a historical version model of the model to be optimized, a homogeneous model of the model to be optimized, a homogeneous system of the model to be optimized, a word grammar tool, a syntax tool, a knowledge base. or any combination thereof. The same species indicates that they have the same function.

In an embodiment of the present disclosure, the evaluation data set may further include correct processing results corresponding to each piece of data to be processed.

For example, in a search scenario, the data to be processed may be a user-entered query, and the corresponding processing result may be the correct sort order for each candidate result recalled based on the query.

It is possible to compare the processing effects of the model to be optimized and the tools in the toolset on the data to be processed in the evaluation data set.

In one embodiment of the present disclosure, the effectiveness comparison module 301 uses the model to be optimized to obtain a first processing result corresponding to each data to be processed in the evaluation data set, and obtains a first processing result corresponding to each data to be processed in the evaluation data set, and For an operable tool, the tool obtains a second processing result corresponding to each data to be processed in the evaluation data set, and assigns different categories based on the correct processing result, the first processing result, and the second processing result. It is possible to perform processing to determine the superiority or inferiority of the processing effects of the model to be optimized and the tool.

For example, if the model to be optimized is a semantic matching model, obtaining the first processing result corresponding to each data to be processed in the evaluation data set using the model to be optimized means that the model to be optimized is is used to sort candidate results corresponding to each data to be processed in the evaluation data set. The obtained sorting result is the first processing result. Similarly, obtaining a second processing result corresponding to each data to be processed in the evaluation data set using the tool includes sorting candidate results corresponding to each data to be processed in the evaluation data set using the tool. It means that. The obtained sorting result is the second processing result.

In an embodiment of the present disclosure, for any one category, the effect comparison module 301 respectively calculates a correct processing result, a first processing result, and a second processing result corresponding to data to be processed belonging to the category in the evaluation data set. Based on the results, it is possible to judge the processing effectiveness of the model to be optimized and the tool. Accordingly, the information identification module 302 determines that the processing effect of the tool is superior to the model to be optimized, and sets this category to be the category to be optimized and the tool to be an advantage tool. can.

In one embodiment of the present disclosure, the effectiveness comparison module 301 performs a third process corresponding to each data to be processed in the evaluation data set using the tool for any one of the tools in the tool set that cannot be individually operated. Obtain the results, use the model to be optimized for any one of the data to be processed, and verify whether or not this third processing result is correct, and it is determined that the verification result is inaccurate. Depending on the situation, it is possible to perform a process of acquiring an artificial determination result and determining the processing effect of the model to be optimized and the tool on the data to be processed based on the artificial determination result.

In one embodiment of the present disclosure, the effectiveness comparison module 301 determines that the artificial determination result is better than the model that the tool optimizes depending on the third processing result being correct; In this case, it is determined that the processing effect of the model to be optimized is superior to that of the tool. Accordingly, the information identification module 302 selects, for any one category, data to be processed for which the processing effect of the tool is superior to that of the model to be optimized, among the data to be processed belonging to that category in the evaluation data set. is determined to be greater than the number of data to be processed for which the processing effect of the model to be optimized is better than that of the tool, the category is set as the category to be optimized, and the tool is set as an advantage tool. do.

The category to be optimized is the category in which the problem exists. The problem can then be solved in a data-driven manner. That is, training data for a category to be optimized can be generated based on the corresponding advantage tool, and the model to be optimized can be updated using the generated training data.

In one embodiment of the present disclosure, for any one category to be optimized, the sample generation module 303 acquires data to be processed belonging to the category to be optimized from the database, and generates an advantage tool corresponding to the category to be optimized. In response to the number being 1, training data for the category to be optimized is generated based on the advantage tool based on the acquired data to be processed, and the number of advantage tools corresponding to the category to be optimized is 1. Depending on the size of the data, training data for categories to be optimized based on each advantage tool can be generated based on the acquired data to be processed.

There are no restrictions on how the data to be processed for the category to be optimized is obtained from the database. For example, some of the queries belonging to the category to be optimized in the database can be randomly extracted.

In one embodiment of the present disclosure, the sample generation module 303 uses an advantage tool for each acquired data to be processed when generating training data for a category to be optimized based on any one advantage tool. After obtaining the corresponding fourth processing result using the model to be optimized, and obtaining the corresponding fifth processing result using the model to be optimized, the fourth processing result and the fourth processing result are selected from among the obtained data to be processed. It is possible to select data to be processed that satisfies conditions that do not match the results of the 5 processing, and to generate training data based on the selected data to be processed.

In one embodiment of the present disclosure, when generating training data based on the selected data to be processed, the sample generation module 303 includes the selected data to be processed and a Training data can be generated by combining the fourth processing result and/or the fifth processing result with the artificial annotation result.

In one embodiment of the present disclosure, in addition to generating training data for the category to be optimized using the advantage tool as described above, the sample generation module 303 also generates a user behavior log of the acquired model to be optimized. Training data can also be generated based on artificial evaluation results that are evaluation results generated based on. Accordingly, model update module 304 may update the model to be optimized using the generated training data.

The specific workflow of the apparatus embodiment shown in FIG. 3 can refer to the related description in the method embodiment above and will not be described in detail here.

In summary, according to aspects described in embodiments of the presently disclosed apparatus, a variety of tools are used to identify problems that exist in a model to be optimized, and a wide variety of tools can be used to solve problems. coverage can be improved. Furthermore, all the problems discovered are real problems that cannot be solved by the model, and training data can be generated directionally for each problem, improving the optimization effect of the model and improving the performance of the model. .

The technical solution described in this application can be applied in the field of artificial intelligence, especially related to fields such as deep learning and natural language processing. Artificial intelligence is the study of computers simulating human thought processes and intelligent acts (e.g. learning, reasoning, thinking, planning, etc.), and includes both hardware-level and software-level techniques. . Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, etc. Artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology and machine learning/deep learning, big data processing technology, knowledge map technology and other several directions.

The data to be processed in the embodiment of the present disclosure is not targeted at a specific user and does not reflect the personal information of a specific user. Further, the entity implementing the method of the present disclosure can obtain the data to be processed through various disclosures and methods compliant with laws and regulations.

In the technical proposal of the present disclosure, the acquisition, storage, application, processing, transmission, provision, distribution, etc. of the personal information of the users involved comply with the provisions of relevant laws and regulations, and do not violate public order and morals.

According to embodiments of the disclosure, the disclosure further provides electronic devices, readable storage media, and computer program products.

FIG. 4 is a schematic block diagram of an electronic device 400 that may be used to implement embodiments of the present disclosure. Electronic devices represent various forms of digital computers, such as laptops, desktop computers, workbenches, servers, blade servers, large format computers, and other suitable computers. Electronic devices can further represent various forms of mobile devices such as PDAs, cell phones, smart phones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functionality depicted herein are merely examples and are not limitations on the implementation of the present disclosure as described and/or required herein.

As shown in FIG. 4, device 400 performs various suitable operations in accordance with a computer program stored in read-only memory (ROM) 402 or loaded into random access memory (RAM) 403 from storage means 408. and calculation means 401 capable of executing processing. The RAM 403 may store various programs and data necessary for the operation of the device 400. Arithmetic means 401, ROM 402 and RAM 403 are connected via bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

Communication between input means 406 such as a keyboard and mouse, output means 407 such as various types of displays and speakers, storage means 408 such as a magnetic disk or optical disk, and communication such as a network card, modem, wireless communication transceiver, etc. Multiple components of device 400, including means 409, are connected to I/O interface 405. The communication means 409 enable the device 400 to exchange information/data with other devices, for example via computer networks of the Internet and/or various telecommunication networks.

The computing means 401 may be various general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing means 401 are a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, digital signals. including, but not limited to, a processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing means 401 performs the various methods and processes described above, such as those described in this disclosure. For example, in some embodiments, the methods described in this disclosure may be implemented as a computer software program physically embodied in a machine-readable medium, such as storage means 408. In some embodiments, part or all of the computer program may be loaded and/or installed on device 400 via ROM 402 and/or communication means 409. When the computer program is loaded into RAM 403 and executed by computing means 401, one or more steps of the method described in this disclosure can be performed. Alternatively, in other embodiments, the computing means 401 may be configured in any other suitable manner (eg, via firmware) to perform the methods described in this disclosure.

Various embodiments of the systems and techniques described herein above include digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), special purpose integrated circuits (ASICs), special purpose standard products (ASSPs), systems on chips. It may be implemented in a system (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs. The one or more computer programs can be executed and/or interpreted on a programmable system that includes at least one programmable processor. The programmable processor is a special purpose or general purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device; Data and instructions can be transferred to the at least one output device.

Program code for implementing the methods of this disclosure may be written using any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing device so that when executed by the processor or controller, the program codes are specified in flowcharts and/or block diagrams. can be configured to perform functions/operations. The program code may run entirely on a machine, partially on a machine, or as a standalone package, partially on a machine and partially on a remote machine. , or all may be executed on a remote machine or server.

In the context of this disclosure, a machine-readable medium is a tangible medium that includes or stores a program for use by or in conjunction with an instruction-execution system, apparatus, or device. I can do it. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More specific examples of machine-readable storage media are electrical connections based on one or more lines, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory. (EPROM or flash memory), fiber optics, portable compact disc read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination thereof.

To provide user interaction, the systems and techniques described herein include a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; It may be implemented on a computer with a keyboard and pointing device (eg, a mouse or trackball) for providing input to the computer by a user. Other types of devices may also be used to provide user interaction. For example, the feedback provided to the user may be any form of sensing feedback (e.g., visual, auditory, or haptic feedback), and any form of input from the user (e.g., acoustic, audio, or (including tactile input).

The systems and techniques described herein may be used in computing systems that include back-end components (e.g., data servers), or that include middleware components (e.g., application servers), or front-end configurations. A computing system (e.g., a client computer having a graphical user interface or web browser) that includes an element through which a user interacts with embodiments of the systems and techniques described herein. may be implemented in a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and internetworks.

A computer system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communications network. The relationship between client and server is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server may be a cloud server, a distributed system server, or a server combined with a blockchain.

It should be appreciated that steps can be reordered, added, or deleted using the various types of flows illustrated above. For example, each step described in this application may be performed in a parallel order or a sequential order, or may be performed in a different order, and the desired results of the technical solutions disclosed in this application may be achieved. There are no limitations here as long as it is achievable.

The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution, or improvement made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims (23)

A model update method, the method comprising:
For the data to be processed in the evaluation data set that belongs to at least two different categories, the corresponding processing results are obtained using the model to be optimized and the tools in the toolset, and the processing results are comparing the processing effects of the model to be optimized and the tool based on;
The method is to identify a category to be optimized and a corresponding advantage tool based on a comparison result, the category to be optimized is a category in which the processing effect of the tool is superior to the model to be optimized, and the advantage tool is to identify a category to be optimized based on a comparison result. The tool is a tool whose processing effect is superior to the model to be optimized;
generating training data for the category to be optimized based on a corresponding advantage tool;
updating the model to be optimized using the training data;
Model update methods, including: The toolset includes one or more of a historical version model of the model to be optimized, a homogeneous model of the model to be optimized, a homogeneous system of the model to be optimized, a word grammar tool, a syntax tool, a knowledge base. including a combination, indicating that the same type has the same function;
The model updating method according to claim 1. The evaluation data set further includes correct processing results corresponding to each data to be processed,
Obtaining corresponding processing results using the model to be optimized and tools in the toolset, and comparing the processing effects of the model to be optimized and the tools based on the processing results,
Using the model to be optimized, obtaining a first processing result corresponding to each data to be processed in the evaluation data set;
For any one of the tools in the tool set that can operate individually, use the tool to obtain a second processing result corresponding to each data to be processed in the evaluation data set, and combine the correct processing result with the first processing result. performing a process of determining the superiority or inferiority of the processing effects of the model to be optimized and the tool for different categories based on the result and the second processing result;
The model updating method according to claim 1, comprising: Determining the superiority or inferiority of the processing effects of the model to be optimized and the tool for the different categories includes determining the correct processing corresponding to the data to be processed belonging to the category in the evaluation data set, respectively, for any one category. determining the superiority or inferiority of a processing effect between the model to be optimized and the tool based on the first processing result and the second processing result;
Identifying the category to be optimized and the corresponding advantage tool based on the comparison result means that the processing effect of the tool is determined to be superior to the model to be optimized for any one category. In accordance with the above, the category is the category to be optimized and the tool is the advantage tool.
The model updating method according to claim 3. Obtaining corresponding processing results using the model to be optimized and tools in the toolset, and comparing the processing effects of the model to be optimized and the tools based on the processing results,
For any one of the tools in the tool set that cannot be operated individually, use the tool to obtain a third processing result corresponding to each data to be processed in the evaluation data set, and Regarding the data, use the model to be optimized to verify whether or not the third processing result is correct, obtain an artificial judgment result in response to the verification result being determined to be inaccurate, and obtain the artificial judgment result using the model to be optimized. performing a process of determining the superiority or inferiority of the processing effects of the model to be optimized and the tool on the data to be processed based on the determination results;
The model updating method according to claim 1, comprising: Determining the superiority or inferiority of the processing effects of the model to be optimized and the tool on the data to be processed based on the artificial determination result is based on the fact that the artificial determination result is correct in accordance with the third processing result. , determining that the processing effect of the tool is better than the model to be optimized, and if not, determining that the processing effect of the model to be optimized is better than the tool,
Identifying the category to be optimized and the corresponding advantage tool based on the comparison result is to determine the processing effect of the tool among the data to be processed belonging to the category in the evaluation data set for any one category. The number of data to be processed for which the model to be optimized is superior to the model to be optimized is determined to be greater than the number of data to be processed for which the processing effect of the model to be optimized is superior to the tool. Accordingly, the category is the category to be optimized and the tool is the advantage tool.
The model updating method according to claim 5. Generating training data for the category to be optimized based on the corresponding advantage tool;
For any one category to be optimized,
Obtaining data to be processed belonging to the category to be optimized from the database,
In response to the number of the advantage tools corresponding to the category to be optimized being 1, training data for the category to be optimized is generated based on the advantage tool based on the obtained data to be processed. ,
In response to the number of the advantage tools corresponding to the category to be optimized being greater than 1, training data of the category to be optimized is determined based on each advantage tool based on the obtained data to be processed. generate each,
Perform each process,
The model updating method according to any one of claims 1 to 6, comprising: Generating training data for the category to be optimized based on the advantage tool comprises:
For each obtained data to be processed, obtain a corresponding fourth processing result using the advantage tool, obtain a corresponding fifth processing result using the model to be optimized,
Selecting data to be processed from among the acquired data to be processed that satisfies a condition that the fourth processing result and the fifth processing result do not match;
generating the training data based on the selected data to be processed;
8. The model updating method according to claim 7. Generating the training data based on the selected data to be processed includes:
generating the training data by combining the selected data to be processed and an artificial annotation result for a fourth processing result and/or a fifth processing result of the selected data to be processed;
9. The model updating method according to claim 8. Generating training data based on the obtained artificial evaluation result, which is an evaluation result generated based on a user behavior log of the model to be optimized, and updating the model to be optimized using the training data.
The model updating method according to any one of claims 1 to 6, further comprising: A model update device comprising an effect comparison module, an information identification module, a sample generation module, and a model update module,
The effect comparison module calculates corresponding processing results for data to be processed in the evaluation data set and belonging to at least two different categories by using the model to be optimized and the tools in the toolset, respectively. and comparing the processing effects of the model to be optimized and the tool based on the processing results,
The information identification module identifies a category to be optimized and a corresponding advantage tool based on the comparison result, and the category to be optimized is a category in which the processing effect of the tool is superior to the model to be optimized; The advantage tool is a tool whose processing effect is superior to the model to be optimized,
The sample generation module generates training data for the category to be optimized based on a corresponding advantage tool;
The model update module updates the model to be optimized using the training data.
Model update device. The toolset includes one or more of a historical version model of the model to be optimized, a homogeneous model of the model to be optimized, a homogeneous system of the model to be optimized, a word grammar tool, a syntax tool, a knowledge base. including a combination, indicating that the same type has the same function;
The model updating device according to claim 11. The evaluation data set further includes correct processing results corresponding to each data to be processed,
The effect comparison module uses the model to be optimized to obtain a first processing result corresponding to each data to be processed in the evaluation data set, and obtains a first processing result corresponding to each data to be processed in the evaluation data set, and calculates a first processing result for any one of the tools in the toolset that can operate individually. , using the tool to obtain a second processing result corresponding to each data to be processed in the evaluation data set, and classifying the data into different categories based on the correct processing result, the first processing result, and the second processing result. performing a process of determining the superiority or inferiority of the processing effects of the model to be optimized and the tool for each;
The model updating device according to claim 11. The effect comparison module, for any one category, based on the correct processing result, the first processing result, and the second processing result that respectively correspond to data to be processed belonging to the category in the evaluation data set, determining the superiority or inferiority of the processing effect between the model to be optimized and the tool;
In response to determining that the processing effect of the tool is superior to the model to be optimized for any one category, the information specifying module sets the category to the category to be optimized, and the tool be the said advantage tool,
The model update device according to claim 13. The effect comparison module uses the tool to obtain a third processing result corresponding to each data to be processed in the evaluation data set for any one of the tools in the tool set that cannot be operated individually, and For each data to be processed, the model to be optimized is used to verify whether or not the third processing result is correct, and if the verification result is determined to be inaccurate, an artificial judgment result is determined. and perform processing to determine the superiority or inferiority of the processing effects of the model to be optimized and the tool on the data to be processed based on the artificial determination results, respectively.
The model updating device according to claim 11. The effect comparison module determines that the processing effect of the tool is superior to the model to be optimized in response to the artificial determination result that the third processing result is correct, and if not, determining that the processing effect of the model to be optimized is superior to that of the tool;
The information identification module identifies, for any one category, among the data to be processed belonging to the category in the evaluation data set, the processing effect of the tool is superior to that of the model to be optimized. in response to determining that the number of data to be optimized is greater than the number of data to be processed for which the processing effect of the model to be optimized is superior to the tool, the category to be optimized is set as the category to be optimized, and the tool is The said advantage tool,
The model updating device according to claim 15. The sample generation module acquires data to be processed belonging to the category to be optimized from a database for any one category to be optimized, and the number of the advantage tools corresponding to the category to be optimized is 1. Optionally, based on the obtained data to be processed, training data for the category to be optimized is generated based on the advantage tool, and the number of the advantage tools corresponding to the category to be optimized is less than 1. Based on the obtained data to be processed, each process generates training data of the category to be optimized based on each advantage tool, depending on the size of the data to be processed.
The model updating device according to any one of claims 11 to 16. Regarding any one of the advantage tools, the sample generation module acquires a corresponding fourth processing result using the advantage tool for each acquired data to be processed, and handles the obtained data using the model to be optimized. and select data to be processed that satisfies the condition that the fourth processing result and the fifth processing result do not match from among the obtained data to be processed, and generating the training data based on data to be processed;
The model update device according to claim 17. The sample generation module generates the training data by combining the selected data to be processed and an artificial annotation result for a fourth processing result and/or a fifth processing result of the selected data to be processed. ,
The model update device according to claim 18. The sample generation module further generates training data based on the obtained artificial evaluation result, which is an evaluation result generated based on a user behavior log of the model to be optimized.
The model updating device according to any one of claims 11 to 16. at least one processor;
a memory communicatively connected to the at least one processor;
A command executable by the at least one processor is stored in the memory, and when the command is executed by the at least one processor, the at least one processor is An electronic device that performs the described model update method. A non-transitory computer-readable storage medium storing computer commands for causing a computer to execute the model updating method according to any one of claims 1 to 6. A computer program that, when executed by a processor, implements the model updating method according to any one of claims 1 to 6.

Images