WO2023071745A1

WO2023071745A1 - Information labeling method, model training method, electronic device and storage medium

Info

Publication number: WO2023071745A1
Application number: PCT/CN2022/124185
Authority: WO
Inventors: 李春霞; 周祥生; 钟斌; 屠要峰; 徐进
Original assignee: 中兴通讯股份有限公司
Priority date: 2021-10-25
Filing date: 2022-10-09
Publication date: 2023-05-04
Also published as: CN114003690A

Abstract

The present application discloses an information labeling method, a model training method, an electronic device and a storage medium. The information labeling method comprises: acquiring an information text to be processed (S100); inputting said information text into an information labeling model to obtain a first entity and a second entity as well as entity relationship information between the first entity and the second entity, wherein the first entity and the second entity are obtained by the information labeling model performing entity distinguishing on said information text, and the entity relationship information is obtained by the information labeling model performing relationship discrimination on the first entity and the second entity (S200); and performing information labeling processing on said information text according to the first entity, the second entity and the entity relationship information to obtain a target information text (S300).

Description

Information labeling method, model training method, electronic device and storage medium

Cross References to Related Applications

This application is based on a Chinese patent application with application number 202111241199.3 and a filing date of October 25, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this application.

technical field

The present application relates to the technical field of information processing, and in particular to an information labeling method, a model training method, electronic equipment, and a storage medium.

Background technique

In recent years, artificial intelligence has gradually become a hot area of concern. Now most companies use artificial intelligence technology to improve customer service levels, improve enterprise efficiency, and reduce operations. If you want artificial intelligence technology to play a real role, you need a lot of data for model training to train high-quality models.

The more important tasks in natural language processing technology include entity recognition and entity relationship mining. In some cases, CRF (Conditional Random Fields, conditional random field) model, RNN (Recurrent Neural Network, cyclic neural network) model or LSTM (Long Short Term Memory, long short-term memory) are generally used to identify named entities and their relationships. to identify. To generate these models, text data needs to be annotated first, followed by model training. Therefore, high-quality labeled data sets are particularly important for model training, but labeling data is a very cumbersome task that requires a lot of manual input, and the models produced by the above-mentioned techniques are not very accurate, and the input and output are low.

Contents of the invention

The following is an overview of the topics described in detail in this article. This summary is not intended to limit the scope of the claims.

Embodiments of the present application provide an information labeling method, a model training method, an electronic device, and a storage medium.

In the first aspect, the embodiment of the present application provides an information labeling method, including: obtaining the information text to be processed; inputting the message text to be processed into the information labeling model to obtain the first entity, the second entity and the first entity Entity relationship information between an entity and the second entity, wherein the first entity and the second entity are obtained by distinguishing the entities of the information text to be processed by the information annotation model, and the entity relationship The information is obtained by discriminating the relationship between the first entity and the second entity by the information labeling model; Perform information annotation processing to obtain target information text.

In the second aspect, the embodiment of the present application also provides a model training method, including: obtaining a training sample, the training sample is a text with label information; inputting the training sample into an information labeling model to obtain the training An information labeling result of a sample, wherein the information labeling result includes a first labeling entity, a second labeling entity, and relationship labeling information between the first labeling entity and the second labeling entity, and the first labeling entity and the second labeled entity are obtained by distinguishing the entities of the training samples by the information labeling model, and the relationship labeling information is obtained by performing entity identification on the first labeled entity and the second labeled entity by the information labeling model It is obtained through relationship discrimination; and the parameters of the information labeling model are updated according to the information labeling result and the label information.

In a third aspect, the embodiment of the present application also provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and operable on the processor, and the processor implements the above when executing the computer program. The information labeling method described in the first aspect, or implement the model training method described in the second aspect above.

In the fourth aspect, the embodiment of the present application also provides a computer-readable storage medium, which stores a processor-executable program, and when the processor-executable program is executed by the processor, it is used to implement the above-mentioned first aspect. The information labeling method described above, or the model training method described in the second aspect above.

In the fifth aspect, the embodiment of the present application further provides a computer program product, including a computer program or a computer instruction, the computer program or the computer instruction is stored in a computer-readable storage medium, and the processor of the computer device reads from the The computer-readable storage medium reads the computer program or the computer instruction, and the processor executes the computer program or the computer instruction, so that the computer device executes the information labeling method as described in the first aspect above, or Realize the model training method as described in the second aspect above.

Additional features and advantages of the application will be set forth in the description which follows, and, in part, will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Description of drawings

FIG. 1 is a flowchart of an information labeling method provided by an embodiment of the present application;

Fig. 2 is the flowchart of the method to step S200 in Fig. 1;

Fig. 3 is the flowchart of the method to step S200 in Fig. 1;

Fig. 4 is the flow chart of the method to step S300 in Fig. 1;

Fig. 5 is the flowchart of the method to step S320 in Fig. 4;

FIG. 6 is a flow chart of an information labeling method provided by another embodiment of the present application;

FIG. 7 is a schematic diagram of a system architecture for executing a model training method provided by another embodiment of the present application;

Fig. 8 is a frame diagram of a pre-training module based on knowledge fusion provided by an embodiment of the present application;

FIG. 9 is a framework diagram of an entity and relationship automatic labeling module provided by an embodiment of the present application;

Fig. 10 is a frame diagram of an annotation result review module provided by an embodiment of the present application;

Fig. 11 is a frame diagram of an audit data management and incremental training module provided by an embodiment of the present application;

Fig. 12 is a flow chart of the audit data management and incremental training module provided by another embodiment of the present application;

Fig. 13 is a flowchart of a model training method provided by an embodiment of the present application;

Fig. 14 is a flowchart of the method for step S700 in Fig. 13;

Fig. 15 is a flowchart of the method for step S700 in Fig. 13;

Fig. 16 is a flowchart of the method for step S800 in Fig. 13;

Fig. 17 is a flowchart of an information labeling method provided by an example of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, not to limit the present application.

It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than in the flowchart. The terms "first", "second" and the like in the specification and claims and the above drawings are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence.

The present application provides an information labeling method, a model training method, electronic equipment and a storage medium. Firstly, the information text to be processed is obtained, and then the text to be processed is input into the information labeling model to obtain the first entity, the second entity and the first entity. The entity relationship information between the entity and the second entity, wherein, the first entity and the second entity are obtained by distinguishing the entities of the information text to be processed by the information annotation model, and the entity relationship information is obtained by the information annotation model for the first entity and the second entity It is obtained by performing relationship discrimination, and then performing information labeling processing on the information text to be processed according to the first entity, the second entity, and the entity relationship information to obtain the target information text. According to the solution of the embodiment of the present application, the first entity, the second entity, and the entity relationship information between the first entity and the second entity are obtained by obtaining the information text to be processed and inputting the information text to be processed into the information labeling model, and then The information text to be processed is processed by information annotation to obtain the target information text, that is to say, the information text to be processed is processed by the information annotation model to obtain the target information text with annotation information. This process does not require manual participation and realizes automatic information processing. Labeling, thus effectively reducing the cost of pure manual labeling.

The embodiments of the present application will be further described below in conjunction with the accompanying drawings.

As shown in FIG. 1 , FIG. 1 is a flowchart of an information labeling method provided by an embodiment of the present application, and the information labeling method can be applied to an information labeling device. The information tagging method may include but not limited to step S100, step S200 and step S300.

Step S100: Obtain the text of the message to be processed.

In this step, the data of the information text to be processed may come from documents in the business field, database data, and the like.

It should be noted that the information text to be processed may have various types of information. For example, the information text to be processed may be paper information, news information information, speech information, etc., which is not specifically limited in this embodiment.

Step S200: Input the text to be processed into the information annotation model to obtain the first entity, the second entity and the entity relationship information between the first entity and the second entity, wherein the first entity and the second entity are treated by the information annotation model The processed information text is obtained by distinguishing entities, and the entity relationship information is obtained by distinguishing the relationship between the first entity and the second entity by the information labeling model.

In this step, the information text to be processed in step S100 is input into the information annotation model, so in the information annotation model, the entity distinction of the information text to be processed can be performed to obtain the first entity and the second entity, and then the first entity and the second entity Entities perform relationship discrimination to obtain entity relationship information, so that subsequent steps can refer to the first entity, the second entity, and the entity relationship information between the first entity and the second entity.

It should be noted that there may be different implementation manners for obtaining the first entity and the second entity through entity distinction of the information text to be processed, which is not specifically limited in this embodiment. For example, entity distinction can be made between place nouns and organization nouns in the information text to be processed. At this time, the first entity obtained is the noun of the place, and the second entity is the noun of the organization organization; Nouns and job nouns are distinguished as entities. At this time, the first entity obtained is a person noun, and the second entity is an organization noun.

It should be noted that, since the first entity and the second entity have many different implementations and the entity relationship information is obtained through the relationship discrimination between the first entity and the second entity, the entity relationship information also has many different implementations corresponding to it. For example, when the first entity is a place noun and the second entity is an organization noun, the obtained entity relationship information is information such as "at", "located in", and "set"; when the first entity is a person noun, the second entity The entity is a noun of an organization, and the obtained entity relationship information is information such as "employ at" and "work at", which is not specifically limited in this embodiment.

Step S300: According to the first entity, the second entity and the entity relationship information, perform information labeling processing on the information text to be processed to obtain the target information text.

In this step, the first entity, the second entity, and the entity relationship information have been obtained according to step S200, so the target information text can be obtained by performing information labeling processing on the information text to be processed according to the first entity, the second entity, and the entity relationship information, so as to realize Automatic labeling of information is achieved, thereby effectively reducing the cost of pure manual labeling.

It should be noted that there may be different implementation manners for the first entity, the second entity, and the entity relationship information to be processed by performing information annotation processing on the to-be-processed information text to obtain the target information text, which is not specifically limited in this embodiment. For example, the information labeling process can be to highlight the first entity, the second entity, and entity relationship information to obtain the target information text; for another example, the information labeling process can be to perform The line is processed to obtain the target information text.

In an embodiment, as shown in FIG. 2, FIG. 2 further illustrates step S200, which may include but not limited to step S210 and step S220.

Step S210: The information labeling model performs word segmentation processing on the information text to be processed to obtain a plurality of first field information.

In this step, word segmentation is performed on the information to be processed to obtain a plurality of first field information, so as to obtain the first entity and the second entity according to the first field information in a subsequent step.

It should be noted that there may be different implementation manners for performing word segmentation processing on the information text to be processed to obtain a plurality of first field information, which is not specifically limited in this embodiment. For example, word segmentation can be performed on the text to be processed according to the set text length, so as to obtain a plurality of first field information, and the set text length can be eight characters or ten characters, etc. The text of the information to be processed is segmented according to the spacing distance, so as to obtain a plurality of first field information.

Step S220: The information labeling model performs entity recognition processing on a plurality of first field information, and identifies the first entity and the second entity in the plurality of field information.

In this step, entity identification processing is performed on the plurality of first field information obtained in step S220, and the first entity and the second entity are identified in the plurality of field information, so that in subsequent steps, according to the first entity and the second entity, the The relationship judgment obtains entity relationship information.

It should be noted that there may be different implementation manners for performing entity identification processing on a plurality of first field information, and identifying the first entity and the second entity in the plurality of field information, which is not specifically limited in this embodiment. For example, entity recognition can be performed on the location nouns and organization nouns in the first field information. At this time, the obtained first entity is the location noun, and the second entity is the organization noun; In this case, the first entity obtained is the person noun and the second entity is the organization noun.

In another embodiment, as shown in FIG. 3 , which further illustrates step S200, which may include but not limited to step S230 and step S240.

Step S230: The information labeling model performs category identification processing on the first entity and the second entity, and obtains first category information corresponding to the first entity and second category information corresponding to the second entity.

In this step, category identification processing is performed on the first entity and the second entity obtained in step S220, and the first category information corresponding to the first entity and the second category information corresponding to the second entity are obtained, so as to facilitate the following based on the first category information Perform relationship identification processing with the second category information to obtain entity relationship information.

It should be noted that since the first entity and the second entity may have many different entity types, the information of the first category obtained from the first entity and the information of the second category obtained from the second entity will also correspond to how many different types, which are not specifically limited in this embodiment. For example, when the first entity is a city noun, the first category information obtained is the city name, and when the second entity is a country noun, the second category information obtained is the country name; for another example, when the first entity is a doctor, For nouns such as teachers, the first type of information obtained is occupational nouns, and when the second entity is nouns such as hospitals and schools, the second type of information obtained is location nouns.

Step S240: The information labeling model performs relationship recognition processing on the first category information and the second category information to obtain entity relationship information.

In this step, the relationship identification processing is performed on the first category information and the second category information obtained in step S230 to obtain entity relationship information, so as to facilitate subsequent labeling operations on the text to be processed.

It should be noted that since the first category information and the second category information have many different information types, the entity relationship information obtained by performing relationship identification processing according to the first category information and the second category information will also correspond to how many different types, which are not specifically limited in this embodiment. For example, when the first type of information is the name of the capital and the second type of information is the name of the country, the obtained entity relationship information is information such as "belongs to" and "included in"; When the second category information is a place noun, the obtained entity relationship information is information such as "work at", "employ at" and so on.

In an embodiment, as shown in FIG. 4 , which further illustrates step S300 , which may include but not limited to step S310 and step S320 .

Step S310: Highlight the first entity and the second entity in the message text to be processed to obtain the first message text.

In this step, the first entity and the second entity in the information text to be processed are highlighted to obtain the first information text, so as to obtain the target information text in subsequent steps.

It should be noted that there may be different implementation manners for highlighting the first entity and the second entity in the information text to be processed, which is not specifically limited in this embodiment. For example, the first entity and the second entity are highlighted to obtain the first information text; another example is the first entity and the second entity are underlined to obtain the first information text.

Step S320: mark the entity relationship information in the first information text to obtain the target information text, wherein the entity relationship information is used to form an association relationship between the first entity and the second entity in the target information text.

In this step, since the first information text is obtained in step S310, the entity relationship information can be marked in the first information text to obtain the target information text, wherein the entity relationship information is used to combine the first entity and The second entity forms an association relationship, thereby realizing automatic labeling of information, thereby effectively reducing the cost of pure manual labeling.

In an embodiment, as shown in FIG. 5 , which further illustrates step S320, which may include but not limited to step S3210 and step S3220.

Step S3210: In the first information text, mark the first type information for the first entity, and mark the second type information for the second entity.

In this step, since the first information text is obtained in step S310, the first type information can be marked on the first entity, and the second type information can be marked on the second entity, so as to obtain the target information text in subsequent steps.

It should be noted that since the first entity and the second entity may have many different entity types, the first category information and the second category information will also correspond to multiple different types, which is not specifically limited in this embodiment , has been specifically described in the above embodiments, and will not be described in detail here.

Step S3220: According to the first category information and the second category information, mark the entity relationship information in the first information text to obtain the target information text.

In this step, since the first category information and the second category information are obtained in step S3210, the entity relationship information can be marked in the first information text to obtain the target information text, and the automatic labeling of information is realized, thereby effectively reducing the The cost of purely manual labeling is eliminated.

As shown in FIG. 6 , FIG. 6 provides a flowchart of an information labeling method according to another embodiment. The information labeling method may include but not limited to step S400 , step S410 and step S420 .

Step S400: Proofread at least one of the first entity, the second entity, and the entity relationship information of the target information text to obtain a proofreading result.

In this step, since the target information text is obtained in step S320, at least one of the first entity, the second entity, and the entity relationship information of the target information text is collated to obtain a proofreading result, so as to verify the proofreading result in subsequent steps judge.

It should be noted that at least one of the first entity, the second entity, and the entity relationship information of the tagged information text may be collated to obtain a collation result, and there may be different implementation modes, which are not specifically limited in this embodiment. For example, manual proofreading can be performed on any one of the first entity, second entity, and entity relationship information to obtain the proofreading result; another example is to perform program proofreading on any one of the first entity, second entity, and entity relationship information, Get the proofreading result.

Step S410: When the collation result shows that at least one of the first entity, the second entity, and the entity relationship information has error information, correct information is obtained according to the error information.

In this step, since the proofreading result was obtained in step S400, when the proofreading result shows that at least one of the first entity, the second entity, and the entity relationship information has error information, correction information is obtained according to the error information, so as to facilitate correcting in subsequent steps The information labeling model is updated.

Step S420: Update the information labeling model according to the correction information.

In this step, since the correction information is obtained in step S410, the information labeling model is updated according to the correction information, which improves the efficiency and quality of labeling, and reduces the cost of purely manual labeling.

It should be noted that the updating process of updating the information labeling model according to the correction information includes iteration of the information labeling model, replacement of the information labeling model, etc., which is not specifically limited in this embodiment.

As shown in FIG. 7 , FIG. 7 is a schematic diagram of a system architecture for executing a model training method provided by an embodiment of the present application. In the example shown in Figure 7, the system architecture includes a general domain training data construction module 100, a pre-training model training module 200 based on knowledge fusion, an entity and relationship automatic labeling review module 300, a labeling result review module 400, review data management and augmentation Quantitative training module 500.

In one embodiment, first construct the training data required for training the pre-training language model based on knowledge fusion through the general domain training data construction module 100, and then send the training data to the pre-training model training module 200 based on knowledge fusion, Among them, the general field training data that needs to be constructed is divided into two categories from the perspective of machine learning, namely unsupervised training data and supervised training data. , news, etc. Supervised data in this scenario specifically refers to data that labels entities and their relationships in text. Entity types usually include person names, place names, institutions, time, etc., and inter-entity relationships include founders, locations, works, etc.

It should be noted that in this embodiment, the ratio between unsupervised data and supervised data is set to 8:2, and it can also be set to other ratios, which is not specifically limited in this embodiment. For example, when the model receives When there is a lot of text information, the ratio between unsupervised data and supervised data can be set to 7:3; for another example, when the model receives less text information, the ratio between unsupervised data and supervised data can be set to It is 9:1.

In one embodiment, after the knowledge fusion-based pre-training model training module 200 receives the training data, it trains a neural network model that is characterized and suitable for entity and relationship extraction tasks according to the training data, wherein the knowledge fusion-based pre-training The model training module 200 introduces the information contained in the knowledge map into the training model, and then sends the trained neural network model to the entity and relationship automatic labeling and review module 300 .

In one embodiment, the rest of the configuration information of the neural network is: the activation function uses the tanh activation function, the model weight and bias values use the random number initialization method, the gradient descent and backpropagation methods are used to solve the model parameters, and the crossover method is used. The entropy loss function, and the use of dropout (drop the out, regularization method) to reduce the impact of overfitting (dropout parameter is 0.8), the number of training of the model is 5*60000 iterations, and the learning rate is 0.00003, among them, dropout regularization The purpose is to reduce overfitting and enhance the generalization ability of the model.

Referring to FIG. 8 , FIG. 8 is a frame diagram of a pre-training module based on knowledge fusion provided by an embodiment of the present application.

In one embodiment, the frame diagram of the pre-training module 200 based on knowledge fusion includes the following subtasks: MLM (Masked Language Mode, mask language model) task, NSP (Next Sentence Prediction, next sentence prediction model) task, entity distinction task, relationship discrimination task, and entity extraction task based on reading comprehension; among them, the MLM task is to perform word segmentation on the sentence, and then select 15% of all words, and 80% of the 15% of the selected words use [MASK] tokens, 10% are represented by original tokens, and 10% are represented by random tokens. After [MASK] is introduced, it is used as model input; the model output is the word representation of the corresponding position of the [MASK] word, and then the loss is calculated by cross entropy, hoping that the model output [MASK] word and the real word just match; the NSP task is the training data The generation method is to randomly extract two consecutive sentences from the parallel corpus, and 50% of them retain the two extracted sentences. The second sentence is randomly extracted from the corpus, and their relationship is NotNext. Among them, the NotNext relationship is a non-inheritance relationship, indicating that there is no relationship between the two sentences before and after, and then compare the model output with the real expected output to perform model The training operation of ; the entity discrimination task is to find the correct tail entity from the current document given the head entity and relationship. For example, a hospital and a doctor have an employment relationship, so the relationship occupation and the head entity hospital are spliced in front of the original document as a reminder. Under this condition, the task of distinguishing the correct tail entity can be transformed into pulling the head entity closer under the framework of contrastive learning The distance between the entity representation of the correct tail entity and the distance between the entity representation of the head entity and other entities (negative samples) in the document; the relationship discrimination task is to distinguish the similarity of the two relationship representations in the semantic space. Randomly sample multiple documents and derive multiple relational representations from each document, where these relations may involve only sentence-level reasoning or complex reasoning across sentences. Then, based on the contrastive learning framework, different relational representations are trained in the relational space according to the distantly supervised labels. As mentioned above, each relational representation consists of two entity representations in the document. Positive samples are relational representations with the same distant supervision label, and negative samples are the opposite; the task of entity extraction based on reading comprehension is given a text sequence X, its length is n, extract each entity in it, where, entity There are corresponding entity types. For example, assuming that the set of all entity labels in a dataset is Y, then for each entity label y in it, such as location LOC, there is a question q(y) about it. Wherein, this question can be a word, also can be a sentence etc. At this point, the model input is X and q(y), and the model is trained and optimized to predict each entity with the label y, so the task can take into account the extraction of ordinary entities and nested entities.

Referring to FIG. 9 , FIG. 9 is a frame diagram of an entity and relationship automatic labeling module provided by an embodiment of the present application.

In one embodiment, the entity and relationship automatic labeling module trains the neural network model to obtain the entity and relationship recognition model, wherein the entity and relationship automatic labeling module also includes REST (Representational State Transfer, representational state transfer) for external submission and interaction service, and synthesize the labeled dataset. First, the dataset to be labeled is input to the entity and relationship automatic labeling module through the externally submitted interactive REST service. At this time, the dataset to be labeled is received as the input of the model, and the model provided Ability to automatically label data, output entities and relationships in the data after model recognition, and write them into a new data set. The original data text and data labels are synthesized in the new data set, and finally the labeling results are sent to the labeling result review module 400 for further processing check.

Referring to FIG. 10 , FIG. 10 is a frame diagram of an annotation result review module provided by an embodiment of the present application.

In one embodiment, the labeling result review module 400 realizes reading the automatically labeled data set, and can perform visual display on the interface of the labeling tool, so as to facilitate the review and proofreading of the automatically marked results on the interface to obtain proofreading data, and then proofread The data is sent to the audit data management and incremental training module 500 .

It should be noted that the review and proofreading operations in the labeling result review module 400 can be performed manually or by a machine, which is not limited in this embodiment.

Referring to Fig. 11 and Fig. 12, Fig. 11 is a frame diagram of an audit data management and incremental training module provided by an embodiment of the present application, and Fig. 12 is a flow chart of the audit data management and incremental training module provided by an embodiment of the present application. In the example of FIG. 12 , the audit data management and incremental training module 500 includes but not limited to step S500 , step S510 , step S520 , step S530 and step S540 .

Step S500: Obtain a dataset to be labeled.

Step S510: Input the dataset to be labeled into the entity and relationship automatic labeling module to perform automatic labeling service to obtain labeling results.

Step S520: Proofreading the labeling results according to the labeling tool to obtain proofreading data.

Step S530: Input the proofreading data into the model for incremental training to obtain a new model.

Step S540: Use the new model to replace the model in the original automatic labeling plug-in.

In one embodiment, the audit data management and incremental training module 500 can manage the proofreaded data on the one hand, and can input the proofreaded data as model incremental training on the other hand, so as to realize continuous iterative optimization of the labeling model and form a label System data and model closed loop. Among them, the audit data management and incremental training module 500 will collect the data that has been proofread during the audit and proofreading, and based on these proofreading data, incrementally train the model, and the new model after training will replace the model in the original automatic labeling plug-in , in order to achieve the effect of continuous optimization of the model. The whole process does not require manual intervention. Model incremental training is triggered by timing or data volume. After the model is generated, the old model can be automatically replaced, and then the new model is used to provide automatic labeling services to the outside world.

In order to clearly illustrate the processing flow of the model training method provided in the embodiment of the present application, an example is used for description below.

As shown in FIG. 13 , FIG. 13 is a flowchart of a model training method provided by an embodiment of the present application, and the information labeling method can be applied to a model training device. The information labeling method may include but not limited to step S600, step S700 and step S800.

Step S600: Obtain training samples, which are texts with label information.

In this step, the text with tag information refers to the text that has been marked manually in advance. The entity type usually includes person name, place name, organization, time, etc., and the relationship between entities includes founder, location, work, etc.

Step S700: Input the training sample into the information labeling model to obtain the information labeling result of the training sample, wherein the information labeling result includes the first labeling entity, the second labeling entity, and the relationship labeling between the first labeling entity and the second labeling entity Information, the first labeled entity and the second labeled entity are obtained by the information labeling model performing entity distinction on the training samples, and the relationship labeling information is obtained by the information labeling model performing relationship discrimination between the first labeled entity and the second labeled entity.

In this step, since the training samples are obtained in step S600, the training samples are input into the information labeling model to obtain the information labeling results of the training samples, wherein the information labeling results include the first labeling entity, the second labeling entity and the first labeling entity The relationship labeling information between the entity and the second labeling entity, the first labeling entity and the second labeling entity are obtained by distinguishing the training samples from the information labeling model, and the relationship labeling information is obtained by the information labeling model on the first labeling entity and the second labeling entity Annotated entities are obtained through relationship discrimination, so that the parameters of the information annotation model can be updated in the subsequent steps according to the information annotation results and label information.

It should be noted that the first labeled entity and the second labeled entity are obtained by distinguishing entities of training samples by an information labeling model, and there may be different implementation manners, which are not specifically limited in this embodiment. For example, entity distinction can be performed on location nouns and organization nouns in the training samples. At this time, the first labeled entity obtained is the location noun, and the second labeled entity is the organization noun; Nouns and job nouns are distinguished as entities. At this time, the first labeled entity is a person noun, and the second labeled entity is an organization noun.

It should be noted that since the first entity and the second labeled entity have many different implementation modes and the relationship labeling information is obtained by distinguishing the relationship between the first entity and the second entity, the relationship labeling information also has many corresponding differences. Embodiments, for example, when the first tagging entity is a place noun, and the second tagging entity is an organization noun, the obtained relationship tagging information is information such as "in", "located in", and "set at"; when the first tagging entity is Person nouns, the second tagged entity is the noun of the organization, and the obtained relationship tagged information is information such as "worked at", "worked at", etc., which is not specifically limited in this embodiment.

Step S800: Update the parameters of the information labeling model according to the information labeling result and label information.

In this step, since the label information is obtained in step S600 and the information labeling result is obtained in step S610, the parameters of the information labeling model are updated according to the information labeling result and the label information.

It should be noted that there may be different implementation manners for updating the parameters of the information labeling model according to the information labeling result and label information, which is not specifically limited in this embodiment. For example, the parameters of the information labeling model can be replaced according to the information labeling results and label information to obtain a new information labeling model; for another example, the parameters of the information labeling model can be added according to the information labeling results and label information to add new labels The dataset is in the information annotation model.

In an embodiment, as shown in FIG. 14 , which further illustrates step S700, which may include but not limited to step S710 and step S720.

Step S710: The information labeling model performs word segmentation processing on the training sample to obtain a plurality of second field information.

In this step, word segmentation processing is performed on the training samples to obtain a plurality of second field information, so that the first labeled entity and the second labeled entity can be obtained according to the second field information in a subsequent step.

It should be noted that the steps in this embodiment have the same technical principle and the same technical effect as the step S210 in the embodiment shown in FIG. 2 above, and the difference between the two embodiments is that the operation objects are different, wherein, The operation object of the above-mentioned embodiment shown in FIG. 2 is the information text to be processed, while the operation object of this embodiment is the training sample with label information. Regarding the technical principles and technical effects of this embodiment, reference may be made to the relevant descriptions in the above embodiment shown in FIG. 2 .

Step S720: The information labeling model performs entity recognition processing on the plurality of second field information, and identifies the first labeled entity and the second labeled entity in the multiple second field information.

In this step, entity recognition processing is performed on the plurality of second field information obtained in step S710, and the first marked entity and the second marked entity are identified in the multiple second field information, so that in subsequent steps, according to the first marked entity Perform relationship judgment with the second labeled entity to obtain information labeling results of the training samples.

It should be noted that the steps in this embodiment have the same technical principle and the same technical effect as the step S220 in the embodiment shown in Figure 2 above, and the difference between the two embodiments is that the operation objects are different, wherein, The operation object of the above-mentioned embodiment shown in FIG. 2 is the information text to be processed, while the operation object of this embodiment is the training sample with label information. Regarding the technical principles and technical effects of this embodiment, reference may be made to the relevant descriptions in the above embodiment shown in FIG. 2 .

In another embodiment, as shown in FIG. 15 , which further illustrates step S700, which may include but not limited to step S730 and step S740.

Step S730: The information labeling model performs category identification processing on the first labeling entity and the second labeling entity to obtain the first labeling category information corresponding to the first labeling entity and the second labeling category information corresponding to the second labeling entity.

In this step, category identification processing is performed on the first labeled entity and the second labeled entity obtained in step S720 to obtain the first labeled category information corresponding to the first labeled entity and the second labeled category information corresponding to the second labeled entity, so as to facilitate Afterwards, a relationship identification process is performed on the first labeling category information and the second labeling category information to obtain relationship labeling information.

It should be noted that the steps in this embodiment have the same technical principle and the same technical effect as the step S230 in the embodiment shown in Figure 3 above, and the difference between the two embodiments is that the operation objects are different, wherein, The operation object of the above-mentioned embodiment shown in FIG. 3 is the information text to be processed, while the operation object of this embodiment is the training sample with label information. Regarding the technical principles and technical effects of this embodiment, reference may be made to the relevant descriptions in the above-mentioned embodiment shown in FIG. 3 .

Step S740: The information labeling model performs relationship identification processing on the first labeling category information and the second labeling category information to obtain relationship labeling information.

In this step, the relationship identification processing is performed on the first labeling category information and the second labeling category information obtained in step S730 to obtain entity relationship information, so as to facilitate subsequent labeling operations on training samples.

It should be noted that the steps in this embodiment have the same technical principle and the same technical effect as the step S240 in the embodiment shown in FIG. 3 above, and the difference between the two embodiments is that the operation objects are different, wherein, The operation object of the above-mentioned embodiment shown in FIG. 3 is the information text to be processed, while the operation object of this embodiment is the training sample with label information. Regarding the technical principles and technical effects of this embodiment, reference may be made to the relevant descriptions in the above-mentioned embodiment shown in FIG. 3 .

In an embodiment, as shown in FIG. 16 , which further illustrates step S800, which may include but not limited to step S810 and step S820.

Step S810: Obtain the training loss value according to the information labeling result and label information.

In this step, since step S700 obtains the information labeling result and step S600 obtains label information, the training loss value is obtained according to the information labeling result and label information, so that the subsequent steps can update the information labeling model according to the training loss value.

It should be noted that the training loss value obtained according to the information labeling result and label information may be implemented in different manners, which is not specifically limited in this embodiment. For example, the information labeling result does not find similar information in the label information of the information labeling model, and the loss value is obtained; another example, the information labeling result does not match the label information of the information labeling model, and labeling errors occur, and the loss value is obtained.

Step S820: Update the parameters of the information labeling model according to the loss value until the loss value satisfies the training stop condition.

In this step, since the loss value is obtained in step S810, the parameters of the information labeling model are updated according to the loss value until the loss value satisfies the training stop condition, thereby improving the efficiency and quality of labeling and reducing the cost of pure manual labeling.

In order to more clearly illustrate the processing flow of the information labeling method provided by the embodiment of the present application, a specific example is used below for description.

As shown in FIG. 17 , FIG. 17 is a processing flowchart of an information labeling method provided by an example. The information labeling method includes the following steps:

Example one:

Step S101: Prepare model training data.

Step S102: Proportionally divide the data.

Step S103: Annotate the entities and relationships in the text, and record the position information and label of each annotated element in the text.

Step S104: Construct the pre-trained language model based on knowledge fusion of this application to predict text entities and their relationships.

Step S105: Use the neural network model to predict and label the text entities and their relationships.

Step S106: Perform REST encapsulation on the model, and add HTTP request and response processing capabilities.

Step S107: Start the model service and provide automatic labeling capability externally.

Step S108: Start automatic labeling on the interface, read the data set in the background, and call the automatic labeling service.

Step S109: After the automatic labeling is completed, the labeling tool filters and loads automatically labeled entities and relationships according to the previously configured entity and relationship categories, and displays them on the interface.

Step S110: Proofread the tagged results, and modify the tagged entities and relationships.

Step S111: When saving after modification, the system records the proofreading data and stores it in the proofreading training data set, which includes the original text and labels.

Step S112: The incremental training trigger periodically judges whether the incremental training trigger condition is met.

Step S113: Take the data set as input, input the above model network structure, and perform incremental training.

Step S114: Update the model online by means of gray scale publishing, and put the new model online for automatic labeling.

In addition, an embodiment of the present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and operable on the processor.

The processor and memory can be connected by a bus or other means.

As a non-transitory computer-readable storage medium, memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may include memory located remotely from the processor, which remote memory may be connected to the processor through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The non-transitory software programs and instructions required to implement the information labeling method or model training method of the above-mentioned embodiments are stored in the memory, and when executed by the processor, the information labeling method in the above-mentioned embodiments is executed, for example, the above-described Method steps S100 to S300 in Fig. 1, method steps S210 and S220 in Fig. 2, method steps S230 and S240 in Fig. 3, method steps S310 and S320 in Fig. 4, method steps S3210 and S3220 in Fig. 5, Method steps S400 to S420 in FIG. 6, or, execute the model training method in the above-mentioned embodiment, for example, execute method steps S600 to S800 in FIG. 13 described above, method steps S710 and S720 in FIG. 14, FIG. 15 Method steps S730 and S740 in , method steps S810 and S820 in FIG. 16 .

The above-described device embodiments or system embodiments are only illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or may be distributed to multiple network units superior. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, an embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by a processor or a controller, for example, by the above-mentioned Executed by a processor in the device embodiment, the above-mentioned processor can execute the information labeling method or the model training method in the above-mentioned embodiment, for example, execute the above-described method steps S100 to S300 in FIG. 1 and the method in FIG. 2 Steps S210 and S220, method steps S230 and S240 in FIG. 3, method steps S310 and S320 in FIG. 4, method steps S3210 and S3220 in FIG. 5, method steps S400 to S420 in FIG. The model training method in the example, for example, executes method steps S600 to S800 in Fig. 13 described above, method steps S710 and S720 in Fig. 14 , method steps S730 and S740 in Fig. 15 , method step S810 in Fig. 16 and S820.

The embodiment of the present application includes: obtaining the information text to be processed, and then inputting the information text to be processed into the information labeling model to obtain the first entity, the second entity, and the entity relationship information between the first entity and the second entity, wherein, the first entity The first entity and the second entity are obtained by distinguishing the entities of the information text to be processed by the information annotation model, and the entity relationship information is obtained by discriminating the relationship between the first entity and the second entity by the information annotation model. The target information text is obtained by performing information annotation processing on the entity relationship information and the information text to be processed. According to the solution of the embodiment of the present application, the first entity, the second entity, and the entity relationship information between the first entity and the second entity are obtained by obtaining the information text to be processed and inputting the information text to be processed into the information labeling model, and then The information text to be processed is processed by information annotation to obtain the target information text, that is to say, the information text to be processed is processed by the information annotation model to obtain the target information text with annotation information. This process does not require manual participation and realizes automatic information processing. Labeling, thus effectively reducing the cost of pure manual labeling.

Those skilled in the art can understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware and an appropriate combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit . Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes both volatile and nonvolatile media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. permanent, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, tape, magnetic disk storage or other magnetic storage devices, or can Any other medium used to store desired information and which can be accessed by a computer. In addition, as is well known to those of ordinary skill in the art, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

The above is a description of several embodiments of the present application, but the present application is not limited to the above-mentioned embodiments. Those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the present application. Any modification or substitution is included within the scope defined by the claims of the present application.

Claims

A method for labeling information, comprising:

Get the message text to be processed;

Input the information text to be processed into the information labeling model to obtain the first entity, the second entity and the entity relationship information between the first entity and the second entity, wherein the first entity and the The second entity is obtained by distinguishing entities of the information text to be processed by the information annotation model, and the entity relationship information is obtained by distinguishing the relationship between the first entity and the second entity by the information annotation model ;

performing information annotation processing on the information text to be processed according to the first entity, the second entity, and the entity relationship information to obtain a target information text.
The information labeling method according to claim 1, wherein the first entity and the second entity are obtained by distinguishing entities of the information text to be processed by the information labeling model, including:

The information labeling model performs word segmentation processing on the information text to be processed to obtain a plurality of first field information;

The information labeling model performs entity recognition processing on the plurality of first field information, and identifies the first entity and the second entity in the plurality of field information.
The information labeling method according to claim 1, wherein the entity relationship information is obtained by discriminating the relationship between the first entity and the second entity by the information labeling model, including:

The information labeling model performs category recognition processing on the first entity and the second entity to obtain first category information corresponding to the first entity and second category information corresponding to the second entity;

The information labeling model performs relationship recognition processing on the first category information and the second category information to obtain the entity relationship information.
The information labeling method according to claim 3, wherein, performing information labeling processing on the information text to be processed according to the first entity, the second entity, and the entity relationship information to obtain the target information text includes :

highlighting the first entity and the second entity on the message text to be processed to obtain a first message text;

Annotating the entity relationship information in the first information text to obtain a target information text, wherein the entity relationship information is used to associate the first entity with the second entity in the target information text relation.
The information labeling method according to claim 4, wherein said labeling said entity relationship information in said first message text to obtain a target message text includes:

In the first information text, the first category information is marked for the first entity, and the second category information is marked for the second entity;

According to the first category information and the second category information, mark the entity relationship information in the first information text to obtain a target information text.
The information labeling method according to claim 1, wherein the information labeling method further comprises:

Proofreading at least one of the first entity, the second entity, and the entity relationship information of the target information text to obtain a proofreading result;

When the proofreading result is that at least one of the first entity, the second entity, and the entity relationship information has error information, correcting information is obtained according to the error information;

The information labeling model is updated according to the correction information.
A model training method, comprising:

Obtain a training sample, the training sample is text with label information;

Input the training samples into the information labeling model to obtain the information labeling results of the training samples, wherein the information labeling results include the first labeling entity, the second labeling entity, and the first labeling entity and the second labeling entity Annotating relationship labeling information between entities, the first labeling entity and the second labeling entity are obtained by the information labeling model performing entity distinction on the training samples, and the relationship labeling information is obtained by the information labeling model obtained by discriminating the relationship between the first marked entity and the second marked entity;

The parameters of the information annotation model are updated according to the information annotation result and the tag information.
The model training method according to claim 7, wherein the first labeled entity and the second labeled entity are obtained by performing entity distinction on the training samples by the information labeling model, comprising:

The information labeling model performs word segmentation processing on the training sample to obtain a plurality of second field information;

The information labeling model performs entity recognition processing on the plurality of second field information, and identifies the first labeling entity and the second labeling entity in the multiple second field information.
The model training method according to claim 7, wherein the relationship labeling information is obtained by the information labeling model performing relationship discrimination between the first labeling entity and the second labeling entity, including:

The information labeling model performs category recognition processing on the first labeling entity and the second labeling entity to obtain the first labeling category information corresponding to the first labeling entity and the second labeling corresponding to the second labeling entity category information;

The information labeling model performs relationship identification processing on the first labeling category information and the second labeling category information to obtain the relationship labeling information.
The model training method according to claim 7, wherein updating the parameters of the information labeling model according to the information labeling result and the label information includes:

Obtaining a training loss value according to the information labeling result and the label information;

Updating the parameters of the information labeling model according to the loss value until the loss value satisfies the training stop condition.
An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein, when the processor executes the computer program, the computer program described in any one of claims 1 to 6 is implemented. The information labeling method described above, or implement the model training method described in any one of claims 7 to 10.
A computer-readable storage medium, in which a processor-executable program is stored, and when the processor-executable program is executed by the processor, it is used to implement the information labeling method according to any one of claims 1 to 6, Or realize the model training method described in any one of claims 7 to 10.
A computer program product comprising a computer program or computer instructions, wherein the computer program or the computer instructions are stored in a computer-readable storage medium, and a processor of a computer device reads the computer-readable storage medium from the computer-readable storage medium A computer program or the computer instruction, the processor executes the computer program or the computer instruction, so that the computer device executes the information labeling method according to any one of claims 1 to 6, or realizes the information labeling method according to any one of claims 1 to 6 The model training method described in any one of 7 to 10.