WO2023226367A1 - 样本标注的校对方法、装置、计算设备集群和存储介质 - Google Patents

样本标注的校对方法、装置、计算设备集群和存储介质 Download PDF

Info

Publication number
WO2023226367A1
WO2023226367A1 PCT/CN2022/137635 CN2022137635W WO2023226367A1 WO 2023226367 A1 WO2023226367 A1 WO 2023226367A1 CN 2022137635 W CN2022137635 W CN 2022137635W WO 2023226367 A1 WO2023226367 A1 WO 2023226367A1
Authority
WO
WIPO (PCT)
Prior art keywords
structure analysis
analysis result
target
sample
target sample
Prior art date
Application number
PCT/CN2022/137635
Other languages
English (en)
French (fr)
Inventor
瞿晓晔
王喆锋
段新宇
李明磊
怀宝兴
Original Assignee
华为云计算技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN202210986086.4A external-priority patent/CN117172250A/zh
Application filed by 华为云计算技术有限公司 filed Critical 华为云计算技术有限公司
Publication of WO2023226367A1 publication Critical patent/WO2023226367A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G06F40/295—Named entity recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present application relates to the field of neural network technology, and in particular to a proofreading method, device, computing device cluster and storage medium for sample annotation.
  • samples with labeled results are usually used to train neural network models.
  • the accuracy of the labeled results of the samples will directly affect the training effect of the neural network model.
  • samples that need to be labeled are manually determined, detailed labeling specifications are formulated, and then the samples are manually labeled according to the labeling specifications using labeling tools in the labeling system.
  • the labeling tool can be a paintbrush, etc.
  • the annotation system records the annotation results of the samples.
  • This application provides a proofreading method, device, computing device cluster and storage medium for sample labeling, which can prompt incorrectly labeled samples and improve the accuracy of sample labeling.
  • the present application provides a proofreading method for sample annotation.
  • the method includes: obtaining a target sample, where the target sample is an annotated sample to be proofread, performing structural analysis on the target sample, and obtaining the third sample of the target sample.
  • a structure analysis result matching the first structure analysis result with the structure analysis results of a plurality of calibrated annotated samples, if the structure analysis results of the plurality of calibrated annotated samples do not match the first structure analysis result
  • a prompt message is output, where the prompt message is used to prompt the user to check the annotation result of the target sample.
  • the corresponding structure analysis results and the structure analysis results of multiple proofread annotated samples can be used to determine whether the annotation result of the sample is wrong, and prompt the user to confirm again, so that The accuracy of the sample annotation results is relatively high, which in turn makes the training effect of the neural network model better.
  • the target sample is a target named entity sample
  • the first structure analysis result is the internal structure of a word
  • the structure analysis is performed on the target sample
  • the first structure analysis result of the target sample is obtained, including: using the word
  • the internal structure analysis model performs word internal structure analysis on the target sample to obtain the word internal structure of the target sample.
  • the structure analysis result is the internal word structure.
  • the internal word structure analysis model can be used to accurately obtain the internal word structure of the target sample.
  • matching the first structure analysis result with the structure analysis results of a plurality of collated annotated samples includes: determining, from the multiple collated annotated samples, a sequence corresponding to the target named entity sample. For one or more named entity samples of the same category, if the first structure analysis result does not exist in the structure analysis results of the one or more named entities, it is determined that the structure analysis results of the multiple proofread annotated samples do not exist in the structure analysis results of the one or more named entity samples. There is a target structure analysis result that matches the first structure analysis result. If the first structure analysis result exists in the structure analysis results of the one or more named entities, then determine the structure analysis of the multiple proofread annotated samples. There is a target structure analysis result matching the first structure analysis result in the result.
  • one or more named entity samples with the same category as the target sample can be determined, and matching is performed in the structure analysis results of the named entities with the same category, so that the matching can be achieved Higher accuracy.
  • the method further includes: generating a confirmation interface for the first structure analysis result, the confirmation interface being used to display the first structure analysis result to the user, receiving the A confirmation instruction input by the user, which is used to modify or confirm the first structure analysis result.
  • the structure analysis result of the target sample is obtained by model analysis, so it may be inaccurate.
  • a confirmation interface is generated for the user to confirm the first structure analysis. Whether the result is accurate or not, the user can confirm or modify the first structure analysis result, which can provide the user with a way to confirm and modify the first structure analysis result.
  • the method further includes: updating the word internal structure analysis model based on the modified first structure analysis result.
  • the first structure analysis result modified by the user is used to update the word internal structure analysis model, which can enable the word internal structure analysis model to subsequently identify the modified first structure analysis result and improve the word internal structure analysis.
  • the generalization ability of the model is used to update the word internal structure analysis model, which can enable the word internal structure analysis model to subsequently identify the modified first structure analysis result and improve the word internal structure analysis.
  • the method further includes: after receiving the confirmation instruction input by the user, adding the confirmed or modified first structure analysis result to the structure analysis result of the proofread annotated sample.
  • the first structure analysis result is confirmed to be correct, it is added to the structure analysis result of the calibrated annotated sample, so that the subsequent first structure analysis result can be consistent with the structure of the calibrated annotated sample.
  • the parsing results match.
  • the method before outputting the prompt message, the method further includes: obtaining a target phrase in the sentence to which the target sample belongs, wherein the target phrase consists of the target sample and words in adjacent positions of the target sample, and the pair The target phrase is subjected to structure analysis, a second structure analysis result of the target phrase is obtained, and it is determined that there is a structure analysis result matching the second structure analysis result among the structure analysis results of the plurality of proofread annotated samples.
  • the prompt message is also used to prompt the correct labeling result corresponding to the target sample. In this way, users can confirm the annotation results more quickly.
  • obtaining the target sample includes: obtaining the target sample labeled by the user; or obtaining the target sample labeled by the pre-labeled model.
  • this application provides a device for proofreading sample annotations.
  • the device includes at least one module, and the at least one module is used to implement the proofreading of sample annotations provided in the above first aspect or any one of the examples of the first aspect. method.
  • the modules in the sample annotation verification device are implemented by software, and the modules in the sample annotation verification device are program modules. In other embodiments, the modules in the sample annotation verification device are implemented by hardware or firmware.
  • the present application provides a computing device cluster.
  • the computing device cluster includes at least one computing device.
  • Each computing device includes a processor and a memory.
  • the processor of the at least one computing device is configured to execute the at least one computing device.
  • the instructions stored in the memory enable the computing device cluster to execute the sample annotation verification method provided by the above-mentioned first aspect or any one of the examples of the first aspect.
  • the present application provides a computer-readable storage medium.
  • the computer-readable storage medium includes computer program instructions.
  • the computing device cluster executes the above first aspect or the third aspect.
  • the proofreading method of sample annotation provided by any of the examples in one aspect.
  • the present application provides a computer program product containing instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to execute the above-mentioned first aspect or any one of the examples of the first aspect. Proofreading method for sample annotation.
  • Figure 1 is a schematic diagram of the system architecture provided by an exemplary embodiment of the present application.
  • Figure 2 is a schematic diagram of the system architecture provided by an exemplary embodiment of the present application.
  • Figure 3 is a schematic flow chart of a proofreading method for sample annotation provided by an exemplary embodiment of the present application
  • Figure 4 is a schematic diagram of an annotation interface provided by an exemplary embodiment of the present application.
  • Figure 5 is a schematic diagram of a named entity library provided by an exemplary embodiment of the present application.
  • Figure 6 is a schematic diagram of the matching process of structure analysis results provided by an exemplary embodiment of the present application.
  • Figure 7 is a schematic flow chart of a proofreading method for sample annotation provided by an exemplary embodiment of the present application.
  • Figure 8 is a schematic diagram of a confirmation interface provided by an exemplary embodiment of the present application.
  • Figure 9 is the structural analysis results of multiple samples provided by an exemplary embodiment of the present application.
  • Figure 10 is a schematic structural diagram of a sample labeling proofreading device provided by an exemplary embodiment of the present application.
  • Figure 11 is a schematic structural diagram of a computing device provided by an exemplary embodiment of the present application.
  • Figure 12 is a schematic structural diagram of a computing device cluster provided by an exemplary embodiment of the present application.
  • Figure 13 is a schematic connection diagram of a computing device provided by an exemplary embodiment of the present application.
  • Named entities refer to person names, organizational names, professional titles and place names, as well as all other entities identified by names. A wider range of named entities also include numbers, dates, addresses, etc.
  • Named entity recognition refers to identifying all named entities in a statement that match a specific type. For example, if the statement is "A organization is organizing activities on the playground today", it needs to be identified in the statement that "A organization” is the name of an organization. .
  • Word internal role refers to the semantic role of each word in the phrase.
  • the internal roles of a word include root, subject-predicate, object (obj), attribute (att), adverbial (adv), verb complement ( complement, cmp), parallel (coordinate, coo), preposition-object (pobj), additional (adjunct, adjct), fragment (frag) and overlap (repet), etc.
  • the internal roles of "A”, “B” and “city” are “att", “frag” and “root” respectively.
  • phrases also have corresponding intra-word roles.
  • the internal role sequence of a word refers to the sequence composed of the roles of the word.
  • the internal role sequence of the word "AB city” is "att frag root”.
  • adjacent roles can be separated by spaces or other characters.
  • the internal structure of the word refers to using the internal role sequence and root words to represent the internal structure of the word.
  • the internal structure of the word "AB city” is the internal role sequence "att frag root” and the root word “city”.
  • sample labeling is crucial to the construction of neural network models.
  • the accuracy of sample labeling results will directly affect the effect of neural network model training.
  • named entity recognition plays an important role in the construction of neural network models for relationship extraction tasks, entity linking, reference ablation, and knowledge graphs. Accurate named entity recognition can make the performance of neural network models for these tasks better.
  • the annotation tools can be brushes, etc.
  • the labeling tool does not have the ability to identify incorrectly labeled samples. In this way, when using incorrectly labeled samples to train a neural network model, it will affect the training effect of the neural network model. For example, the sentence includes the place-name entity "Jiangbian Avenue”, but the user marks "Jiangbian Avenue” as a place-name entity and misses "dao", resulting in poor generalization of the neural network model.
  • the structure analysis structure of the sample can be used to analyze whether the labeling result of the sample is wrong. If the labeling result is wrong, the user is prompted to check the labeling result of the sample, thereby improving the accuracy of sample labeling.
  • system architecture 100 includes terminal device 101 and computing device 102 .
  • the terminal device 101 and the computing device 102 are connected through a wired or wireless network.
  • the terminal device 101 is a device used by the user, such as a desktop computer, a laptop computer, a tablet computer or a mobile phone, etc.
  • the terminal device 101 is used for the user to interact with the computing device 102 .
  • the user can use the terminal device 101 to add annotation results to samples, etc.
  • the computing device 102 may be a server or the like.
  • the computing device 102 is used to determine the structural analysis results of the target sample, and match the structural analysis results of the target sample with the structural analysis results of multiple proofread annotated samples.
  • the target sample is annotated. , and unproofed sample.
  • an embodiment of the present application provides another system architecture 200.
  • the system architecture 200 includes a terminal device 101 and a public cloud 201.
  • the terminal device 101 is connected to the public cloud 201 through a wired or wireless network.
  • the terminal device 101 is a device used by the user, and the terminal device 101 is used for the user to interact with the public cloud 201 .
  • Public cloud 201 is an entity that uses basic resources to provide cloud services to users under the cloud computing model. Public cloud 201 can also be considered a cloud environment.
  • the public cloud 201 includes a cloud data center.
  • the cloud data center includes a large number of basic resources owned by the cloud service provider.
  • the large number of basic resources include computing resources, storage resources and network resources.
  • the computing resources included in the cloud data center can be computing device clusters.
  • the device cluster includes at least one computing device, which may be a server or the like.
  • the user can upload the target sample to the public cloud 201 through the application program interface (application program interface, API) or graphical user interface (graphical user interface, GUI), etc., and the computing device cluster in the public cloud 201 receives the user
  • the uploaded target sample determines the structure analysis result of the target sample, matches the structure analysis result of the target sample with the structure analysis results of multiple proofread annotated samples, and outputs a prompt message to the terminal device 101 based on the matching results.
  • the embodiment of the present application also provides a system architecture.
  • the system architecture includes terminal devices, which are devices used by users.
  • the terminal device executes the proofreading method of sample annotation in the embodiment of the present application.
  • FIG. 3 provides a process flow of a proofreading method for sample annotation.
  • the system architecture 100 shown in FIG. 1 is used as an example to describe the process of a proofreading method for sample annotation.
  • the terminal device is the terminal device 101 mentioned above
  • the computing device is the computing device 102 mentioned above.
  • Step 301 Obtain the target sample.
  • the target sample is an annotated sample to be proofread, that is, an annotated sample that has been annotated but not proofread, and the target sample is any annotated sample to be proofread.
  • the target sample may be a sample that can analyze whether the annotation result of the target sample is accurate based on the structure analysis result.
  • the target sample may be a named entity or word segmentation sample, etc.
  • a start analysis option is displayed in the annotation interface for the target sample.
  • the user can click the start analysis option, and the terminal device sends an analysis request to the computing device.
  • the computing device receives the analysis request and obtains the target sample.
  • the target sample is a target named entity sample.
  • the annotation interface displays the statement "Y University is located in the northwest suburbs of city A", the start analysis option, and the identification of the named entity category.
  • the place name is identified by a solid line frame, and the organization structure is displayed.
  • the first type of dotted box is used to mark the name
  • the second type of dotted box is used to mark the professional title.
  • the user marks "Y University” as the name of the organization in the sentence, and marks "A City” as the place name in the sentence.
  • Figure 4 shows only one example.
  • the logos of named entity categories are distinguished by color. For example, red logos are used for place names, and green logos are used for organization names.
  • the sample becomes the corresponding category. color.
  • the next sentence option and the previous sentence option can also be displayed in the annotation interface. The next sentence option is used to switch to the next sentence for annotation or viewing, and the previous sentence option is used to switch to the previous sentence. Mark or view.
  • a start analysis option is displayed in the sample labeling interface.
  • the user can click the start analysis option, and the terminal device sends an analysis request to the computing device.
  • the computing device receives the analysis request and obtains multiple samples labeled by the user, and the multiple samples include the target sample.
  • the terminal device submits the labeling result of the target sample to the computing device, and the computing device obtains the target sample.
  • a pre-labeling model is trained, and the pre-labeling model is a model that labels samples. After the pre-labeling model completes labeling the target sample, the computing device obtains the labeled target sample.
  • Step 302 Perform structural analysis on the target sample to obtain the first structural analysis result of the target sample.
  • the structural analysis model is stored in the computing device, or the computing device uses samples and the structural analysis results of the samples for training to obtain the structural analysis model, or the computing device obtains the structural analysis model from other devices.
  • the computing device uses the structure analysis model to analyze the structure of the target sample, and obtains the structure analysis result of the target sample, which is called the first structure analysis result.
  • the structure analysis model used to analyze the structure of the target sample is a word internal structure analysis model.
  • the structure analysis model used to analyze the target word segmentation sample is The structure analysis model is a sentence structure analysis model.
  • the target sample is a target named entity sample
  • the first structure analysis result is the internal structure of the word
  • the process of obtaining the first structure analysis result of the target sample is:
  • the word internal structure analysis model is trained, and the word internal structure analysis model is used to analyze the internal structure of words.
  • the target sample is input into the word internal structure analysis model, and the output of the word internal structure analysis model is the word internal structure of the target sample. That is to say, the first structure analysis result is the word internal structure.
  • the target sample is " ⁇ Underground Palace”. After analyzing the internal structure of the word “ ⁇ Underground Palace”, the internal structure of the word is "att att att root”, and the root word is " ⁇ ”. In the embodiment of this application, an “ ⁇ ” represents one character, so “ ⁇ Underground Palace” is four characters.
  • the target sample is "XX District”. After analyzing the internal word structure of "XX District”, the internal structure of the word is “att frag root”, and the root word is "District”.
  • the first structure analysis result is the word segmentation structure of the target word segmentation sample.
  • the process of obtaining the first structure analysis result of the target sample is:
  • a structure parsing model with word segmentation samples is trained.
  • the structure parsing model is used to analyze the structure of the word segmentation samples.
  • the word segmentation sample is input into the structure analysis model, and the output of the structure analysis model is the first structure analysis result of the word segmentation sample.
  • the word segmentation sample is "I/come to/XX/area”
  • the first structural analysis result of "I/come to/XX/area” is "pronoun verb noun”.
  • the computing device may display the first structure analysis result to the user.
  • the user marked the named entity "XX University” as the name of the organization in the sentence "XX University is about to have a holiday”.
  • the schematic diagram of the structure analysis results shown in Figure 5 shows the first name of "XX University”.
  • the structure analysis result, the first structure analysis result is "att att att root”, and the root word is " ⁇ ".
  • Step 303 Match the first structure analysis result with the structure analysis results of multiple proofread annotated samples.
  • the proofread annotated samples are the annotated samples whose annotation results are confirmed to be correct.
  • the computing device stores the structure analysis results of the calibrated annotated samples, or obtains the calibrated structure analysis results of the annotated samples from other devices, or performs structural analysis on the calibrated annotated samples in the computing device to obtain the target structure analysis results.
  • the target structure analysis result is confirmed to be correct by the user, where the user indicates the reviewer of the structure analysis result.
  • the computing device determines whether there is a target structure analysis result matching the first structure analysis result among the structure analysis results of the plurality of calibrated annotated samples. If there is no target structure analysis result matching the first structure analysis result among the structure analysis results of the multiple proofread annotated samples, it is determined that the first structure analysis result matches the structure analysis result, otherwise it is determined that the first structure analysis result does not match. to the structure analysis results.
  • the category of the target sample can be determined.
  • the category of the target sample is called the target category. Select a named entity sample of the target category from the multiple collated named entities, and then match the first structure analysis result with the structure analysis result of the named entity sample of the target category. If the first structure parsing result does not exist in the structure parsing result of the named entity of the target category, then the first structure parsing result does not match the structure parsing result.
  • the first structure parsing result exists in the structure parsing result of the named entity of the target category. , it is determined that the first structure analysis result matches the structure analysis result. In this way, only the structure parsing results of named entities with the same category are matched, not only the number of matches is smaller, but the matching results are more accurate.
  • the structure parsing results of the collated named entities are divided into different named entity libraries according to the categories of the named entities.
  • personal names, organizational names, professional titles and place names correspond to different named entity libraries respectively.
  • Figure 6 shows the named entity library corresponding to place names.
  • the named entity library includes the internal structure of words corresponding to the named entities, as shown in Figure 6 , the internal structure 1 of the word is "att frag root", the root word is "district”, the internal structure 2 of the word is “att root”, the root word is "country”, the internal structure 3 of the word is "att att root”, the root word is "Bay” etc.
  • the target sample is a named entity of a place name.
  • the named entity library corresponding to the place name is used for matching.
  • the target sample is " ⁇ city”
  • the first structure analysis result is “att frag root”
  • the root word is "city”.
  • the structure analysis result "att frag root” does not exist in the named entity database corresponding to the place name
  • the root word is "City" means that the first structure analysis result does not match the structure analysis result in the named entity database corresponding to the place name, see Figure 7.
  • the target sample is " ⁇ city”
  • the first structure analysis result is “att frag root”
  • the root word is "city”.
  • the target structure parsing result is "att frag root” and the root word is "city”.
  • the target sample is " ⁇ Mayor"
  • the labeled category is place name
  • the first structure analysis result is "att att att root”
  • the root word is "long”
  • Consistent the structural analysis result of " ⁇ Underground Palace” underground palace is “att att att root”, the root word is "palace”
  • the structural analysis result of " ⁇ Underground Palace” underground palace is “att att att root”
  • the root word is "palace”
  • there is no structural analysis result with the root word " ⁇ ” indicating that the first structural analysis result corresponds to the place name
  • the structure parsing results in the named entity library do not match.
  • the matching method is consistent with the named entity sample.
  • the word segmentation sample is "I/came to/XX/area”
  • the first structural analysis result of "I/came to/XX/area” is "pronoun verb noun”. If the structure analysis of the proofread annotated sample If "pronoun verb noun" is present in the result, it is determined that the first structure analysis result matches the structure analysis result; otherwise, the structure analysis result is not matched.
  • Step 304 If there is no target structure analysis result matching the first structure analysis result among the structure analysis results of the multiple proofread annotated samples, a prompt message is output, where the prompt message is used to prompt the user to check the target. Sample labeling results.
  • the computing device send a prompt message to the terminal device.
  • the terminal device can display the content of the prompt message.
  • the prompt message is used to prompt the user to check the labeling result of the target sample.
  • the embodiment of this application does not limit the specific content of the prompt message. Users can reconfirm whether the labeling results of the target samples are accurate.
  • a view option is displayed in the interface that displays the prompt message.
  • the user can click the view option to return to the annotation interface.
  • the annotation interface the user can modify the annotation result of the target sample and submit it after the modification is completed.
  • the computing device can execute steps 301 to 303 in Figure 3 again to determine whether the resubmitted annotation result is incorrect.
  • the target sample can also be sent to two other users for review.
  • the other two users are annotators with relatively high annotation levels. If the other two users determine that the original labeling results are correct, the original labeling results will be stored. If the other two users have modified the labeling results and the modified labeling results are the same, the modified labeling results can be stored.
  • the target structure analysis result exists in the structure analysis results of multiple proofread annotation samples, and the target structure analysis result matches the first structure analysis result, indicating that the annotation result of the target sample is accurate, and the annotation result of the target sample is stored.
  • the prompt message is also used to indicate the correct labeling result corresponding to the target sample. In this way, the user can refer to the prompt message when checking the labeling result of the target sample. Correct annotation results indicated in .
  • the processing process refers to steps 305 to 312 included before inputting the prompt message in step 303 and step 304. This processing process is an optional processing process.
  • Step 305 If there is no target structure analysis result matching the first structure analysis result among the structure analysis results of the multiple proofread annotated samples, a confirmation message of the first structure analysis result is output.
  • the computing device sends a confirmation message of the first structure parsing result to the terminal device.
  • the terminal device receives a confirmation message of the first structure analysis result and generates a confirmation interface corresponding to the confirmation message.
  • the computing device sends a confirmation interface to the terminal device.
  • the schematic diagram of the confirmation interface shown in Figure 9. The target sample is " ⁇ city”, the first structure analysis result is "att frag root”, and the root word is "city”.
  • it also displays " Please confirm whether the current structure analysis result is correct. If it is incorrect, please modify it. If it is correct, please confirm" prompt content, confirmation options and modification options.
  • the terminal device will send a confirmation message to the computing device. If the user thinks that the first structure analysis result is incorrect, he can click the modification option to trigger the terminal device to display the modification interface. The user can modify the first structure analysis result and submit it after the modification is completed. For the convenience of description, the modified first structure
  • the analysis results are described as third structure analysis results.
  • the structure analysis model can obtain a variety of structure analysis results when performing structure analysis on the target sample.
  • the first structure analysis result is the structure analysis result with the highest probability.
  • the computing device sends the confirmation message of the first structure analysis result to the terminal device, it may also carry the remaining structure analysis results of the target sample.
  • the remaining structure analysis results may be carried in the confirmation message or sent separately.
  • displaying the confirmation interface of the first structure analysis result multiple structure analysis results may be displayed to provide a reference for the user to confirm the first structure analysis result.
  • the remaining structure analysis results may be displayed in the confirmation interface together with the first structure analysis result, or may be displayed in the modification interface.
  • FIG. 9 is only a schematic diagram of a possible confirmation interface, and any interface that can confirm the first structure analysis result can be applied to the embodiments of the present application.
  • Step 306 Receive the confirmation instruction input by the user.
  • the terminal device receives the confirmation instruction input by the user and sends the confirmation instruction to the computing device. If the confirmation instruction indicates that the confirmation is correct, the computing device can send a prompt message to the terminal device after receiving the confirmation instruction sent by the terminal device. , this prompt message is used to prompt the user to check the labeling results of the target sample.
  • Step 307 Update the word internal structure analysis model based on the first structure analysis result.
  • the first structure analysis result can also be used to update the word internal structure analysis model, so that the accuracy of the structure analysis of the word internal structure analysis model is higher.
  • Step 308 Add the first structure analysis result to the structure analysis result of the calibrated annotated sample.
  • the first structure analysis result can also be added to the structure analysis result of the calibrated annotated sample, so that there will be samples similar to the target sample in the future.
  • the corresponding structure analysis results can be matched.
  • the first structure analysis result when the user confirms that the first structure analysis result is correct, can also be added to the named entity library of the category corresponding to the target sample, so that there will be samples similar to the target sample in the future.
  • the corresponding structure parsing result can be matched in the named entity library.
  • the target sample is " ⁇ Temple”
  • the first structure analysis result is "att frag root”
  • the root word is "Temple”.
  • the named entity database corresponding to the place name there is a sequence of internal roles in the word that is consistent, and the root word is inconsistent.
  • the user confirms that the first structure analysis result is correct, and adds "att frag root”, with the root word "Temple”, to the named entity database corresponding to the place name.
  • Step 309 Receive the third structure analysis result input by the user, and match the third structure analysis result with the structure analysis results of multiple proofread annotated samples.
  • step 305 the user thinks that the first structure analysis result is incorrect and modifies the first structure analysis result, and the obtained modification result is the third structure analysis result.
  • the computing device receives the third structure analysis result and determines whether there is a structure analysis result matching the third structure analysis result among the structure analysis results of the multiple proofread annotated samples. If there is no structure analysis result among the multiple proofread annotation samples, If there is a structure analysis result that matches the third structure analysis result, it is determined that the third structure analysis result does not match the structure analysis result of the proofread annotated sample; otherwise, it is determined that the third structure analysis result is in the proofread annotation sample. The structure analysis result of the sample matches the structure analysis result.
  • the target category of the target sample may be determined first, and the structure analysis result corresponding to the named entity of the target category may be determined. , determine whether the third structure analysis result exists in the structure analysis result corresponding to the named entity of the target category.
  • Step 310 If there is no structural analysis result matching the third structure analysis result among the structural analysis results of the multiple proofread annotated samples, jump to the output prompt message in step 304.
  • the computing device outputs to the terminal device Prompt message, which is used to prompt the user to check the labeling results of the target sample.
  • the first structure analysis result does not match the structure analysis results of the multiple proofread annotation samples. It is because the first structure analysis result is wrong, not because the annotation result of the target sample is wrong, so it can be confirmed that the annotation result of the target sample is correct.
  • Step 311 Update the word internal structure analysis model based on the third structure analysis result.
  • the third structure analysis result when the user inputs the third structure analysis result, can also be used to update the word internal structure analysis model, so that the word internal structure analysis model has a stronger generalization ability.
  • Step 312 Add the third structure analysis result to the structure analysis result of the proofread annotated sample.
  • the third structure analysis result can also be added to the proofreaded annotated sample.
  • the structure analysis results of the annotated samples are included in the structure analysis results, so that when there are subsequent samples similar to the target sample, the corresponding structure analysis results can be matched.
  • the third structure analysis result can also be added to the named entity library of the target category. , so that when a sample similar to the target sample exists later, the corresponding structure analysis result can be matched in the named entity library.
  • step 307 and step 308 have no sequence with step 304
  • step 307 and step 308 have no sequence either.
  • Step 311 and step 312 are not in sequence with step 304, and steps 311 and step 312 are also in no sequence.
  • the target phrase is obtained, where the target phrase consists of the target sample and words in adjacent positions of the target sample.
  • the target phrase is structurally analyzed to obtain the second structure analysis result of the target phrase and determine multiple Among the structure analysis results of the proofread annotated sample, there is a structure analysis result that matches the second structure analysis result.
  • the computing device determines words at adjacent positions of the target sample in the sentence to which the target sample belongs, and the words and the target sample constitute the target phrase.
  • the target sample is "XX”
  • the sentence it belongs to is "I settled in "XX” city”
  • the target phrase is "XX city”.
  • the word in the adjacent position here can be a word after the target sample.
  • the specific number of words after the target sample can be set according to the actual application scenario, which is not limited in the embodiment of this application.
  • the computing device inputs the target phrase into the word internal structure analysis model and obtains the structure analysis result of the target phrase, that is, the second structure analysis result.
  • the computing device determines whether there is a structure analysis result matching the second structure analysis result among the structure analysis results of the plurality of calibrated annotated samples. If it exists, a prompt message can be output, which is used to prompt the user to check the labeling results of the target sample. In this way, when multiple users have inconsistent labeling, incorrectly labeled samples can be mined to improve labeling accuracy.
  • the category of the target phrase can be determined first, and then Among the multiple collated annotated samples, determine one or more named entity samples that are of the same category as the first named entity sample, and determine whether the second structure analysis result exists in the structure analysis results of the one or more named entities, If it exists, a prompt message can be output.
  • the prompt message is also used to indicate the correct labeling result corresponding to the target sample.
  • the user can refer to the correct labeling results indicated in the prompt message when checking the labeling results of the target sample. For example, if the target sample is "XX" and the target phrase is "XX city”, the prompt message can carry "XX city” for prompting.
  • the computing device may also send a confirmation message of the second structural analysis result to the terminal device.
  • the terminal device receives the confirmation message of the second structure analysis result and generates a confirmation interface of the second structure analysis result.
  • the second structure analysis result shown here is the same as the first structure analysis result shown in Figure 9 above, and will not be described again.
  • the computing device will receive a confirmation message input by the user and can send a prompt message to the terminal device, where the prompt message is used to prompt the user to check the annotation results of the target sample. If the user confirms that the second structure analysis result is incorrect, a confirmation message of the first structure analysis result is output (that is, the above step 305 is executed).
  • a confirmation message of the first structural analysis result is output (ie, the above-mentioned step 305 is executed).
  • the computing device can also use the second structure analysis result.
  • the structure analysis results update the word internal structure analysis model, making the word internal structure analysis model more accurate.
  • one sample of the target sample is used as an example.
  • the structure of multiple samples can also be analyzed at the same time to determine whether the multiple samples are labeled incorrectly, so as to dig out the information for the user. Possibly incorrectly labeled samples. For example, if the user labels two samples in one sentence, it can be judged at the same time whether the two samples are labeled incorrectly.
  • system architecture 100 is used as an example.
  • other system architectures are used to implement the proofreading method of sample annotation, the execution process is similar to the previous description and will not be described again here.
  • samples that may be labeled incorrectly can be excavated based on the structural analysis results of the samples, and the user is prompted to confirm again, so that the accuracy of the labeled results of the samples is relatively high, thereby making the neural network more efficient.
  • the training effect of the network model is relatively good.
  • FIG. 10 is a structural diagram of a sample labeling proofreading device provided by an embodiment of the present application.
  • the device can be implemented as part or all of the device through software, hardware, or a combination of both.
  • the device provided by the embodiment of the present application can implement the processes shown in Figure 3 and Figure 7 of the embodiment of the present application.
  • the device includes: an interaction module 1010, a parsing module 1020 and a matching module 1030, wherein:
  • the interactive module 1010 is used to obtain a target sample, where the target sample is an annotated sample to be proofread, and specifically can be used to implement the interactive function of step 301 and execute the implicit steps included in step 301;
  • the analysis module 1020 is used to perform structural analysis on the target sample and obtain the first structure analysis result of the target sample. Specifically, it can be used to implement the analysis function of step 302 and execute the implicit steps included in step 302;
  • the matching module 1030 is used to match the first structure analysis result with the structure analysis results of multiple proofread annotated samples. Specifically, it can be used to implement the matching function of step 303 and execute the implicit steps included in step 303;
  • the interactive module 1010 is also configured to output a prompt message if there is no target structure analysis result matching the first structure analysis result among the structure analysis results of the plurality of proofread annotated samples, wherein:
  • the prompt message is used to prompt the user to check the labeling result of the target sample. Specifically, it can be used to implement the interactive function of step 304 and execute the implicit steps included in step 304.
  • the target sample is a target named entity sample
  • the first structure analysis result is the internal structure of the word
  • the parsing module 1020 is used for:
  • the matching module 1030 is used to:
  • first structure analysis result does not exist in the structure analysis results of the one or more named entities, it is determined that the structure analysis results of the multiple proofread annotated samples do not exist with the first structure analysis result.
  • the result matches the target structure parsing result
  • the first structure analysis result exists in the structure analysis results of the one or more named entities, it is determined that there is a match between the structure analysis results of the multiple proofread annotated samples and the first structure analysis result.
  • the target structure analysis result If the first structure analysis result exists in the structure analysis results of the one or more named entities, it is determined that there is a match between the structure analysis results of the multiple proofread annotated samples and the first structure analysis result. The target structure analysis result.
  • the interaction module 1010 is also used to:
  • the parsing module 1020 is also used to:
  • the word internal structure analysis model is updated.
  • the interaction module 1010 is also used to:
  • the confirmed or modified first structure analysis result is added to the structure analysis result of the proofread annotated sample.
  • the matching module 1030 is also used to:
  • the target phrase consists of the target sample and words in adjacent positions of the target sample
  • the prompt message is also used to prompt the correct labeling result corresponding to the target sample.
  • the interaction module 1010 is used to:
  • the interaction module 1010, the parsing module 1020 and the matching module 1030 can all be implemented by software, or can be implemented by hardware.
  • the following uses the parsing module 1020 as an example to introduce the implementation of the parsing module 1020.
  • the implementation of the interaction module 1010 and the matching module 1030 can refer to the implementation of the parsing module 1020.
  • the parsing module 1020 may include code running on a computing instance.
  • the computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the above computing instance may be one or more.
  • the parsing module 1020 may include code running on multiple hosts/virtual machines/containers. It should be noted that multiple hosts/virtual machines/containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, multiple hosts/virtual machines/containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs. Each AZ includes one data center or multiple AZs. geographically close data centers. Among them, usually a region can include multiple AZs.
  • the multiple hosts/VMs/containers used to run the code can be distributed in the same virtual private cloud (VPC), or across multiple VPCs.
  • VPC virtual private cloud
  • Cross-region communication between two VPCs in the same region and between VPCs in different regions requires a communication gateway in each VPC, and the interconnection between VPCs is realized through the communication gateway. .
  • the parsing module 1020 may include at least one computing device, such as a server.
  • the parsing module 1020 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD).
  • ASIC application-specific integrated circuit
  • PLD programmable logic device
  • the above-mentioned PLD can be a complex programmable logical device (CPLD), a field-programmable gate array (field-programmable gate array, FPGA), a general array logic (generic array logic, GAL), or any combination thereof.
  • CPLD complex programmable logical device
  • FPGA field-programmable gate array
  • GAL general array logic
  • Multiple computing devices included in the parsing module 1020 may be distributed in the same region or in different regions. Multiple computing devices included in the parsing module 1020 may be distributed in the same AZ or in different AZs. Similarly, multiple computing devices included in the parsing module 1020 may be distributed in the same VPC or in multiple VPCs.
  • the plurality of computing devices may be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
  • the interaction module 1010 can be used to perform any step in the proofreading method of sample annotation
  • the parsing module 1020 can be used to perform any step in the proofreading method of sample annotation
  • the matching module 1030 can be used Any step in the calibration method for performing sample annotation.
  • the steps that the interactive module 1010, the parsing module 1020, and the matching module 1030 are responsible for implementing can be specified as needed.
  • the interactive module 1010, the parsing module 1020, and the matching module 1030 respectively implement different steps in the proofreading method of sample annotation to implement the proofreading device of the sample annotation. all functions.
  • the following describes the computing device 102 provided by the embodiment of the present application.
  • computing device 102 includes: bus 1102, processor 1104, memory 1106, and communication interface 1108.
  • the processor 1104, the memory 1106 and the communication interface 1108 communicate through a bus 1102.
  • Computing device 102 may be a server or terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 102.
  • the bus 1102 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc.
  • the bus can be divided into address bus, data bus and control bus. For ease of presentation, only one line is used in Figure 11, but it does not mean that there is only one bus or one type of bus.
  • Bus 1104 may include a path that carries information between various components of computing device 102 (eg, memory 1106, processor 1104, and communications interface 1108).
  • the processor 1104 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP). any one or more of them.
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • Memory 1106 may include volatile memory, such as random access memory (RAM).
  • the processor 1104 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, mechanical hard disk drive (hard disk drive, HDD) or solid state drive (solid state drive). state drive, SSD).
  • ROM read-only memory
  • HDD hard disk drive
  • SSD solid state drive
  • the memory 1106 stores executable program code, and the processor 1104 executes the executable program code to respectively realize the functions of the interaction module 1010, the parsing module 1020 and the matching module 1030 described later, thereby realizing the proofreading method of sample annotation. That is, the memory 1106 stores instructions for executing the calibration method of sample annotation.
  • the communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to implement communication between the computing device 102 and other devices or communication networks.
  • An embodiment of the present application also provides a computing device cluster.
  • the computing device cluster includes at least one computing device.
  • the computing device may be a server, for example, the computing device may be a central server, an edge server, or a local server in a local data center.
  • the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.
  • the computing device cluster includes at least one computing device 102.
  • the memory 1106 of one or more computing devices 102 in the computing device cluster may store the same instructions for performing the calibration method of sample annotation.
  • the memory 1106 of one or more computing devices 102 in the computing device cluster may also store part of the instructions for executing the verification method of sample annotation.
  • a combination of one or more computing devices 102 may collectively execute instructions for performing a calibration method for sample annotation.
  • the memories 1106 in different computing devices 102 in the computing device cluster can store different instructions, which are respectively used to perform part of the functions of the sample annotation proofreading device described below. That is, the instructions stored in the memory 1106 in different computing devices 102 can implement the functions of one or more modules in the interaction module 1010, the parsing module 1020, and the matching module 1030.
  • one or more computing devices in a cluster of computing devices may be connected through a network.
  • the network can be a wide area network or a local area network, etc.
  • Figure 13 shows a possible implementation. As shown in Figure 13, two computing devices (first computing device 102A and second computing device 102B) are connected through a network. Specifically, the connection to the network is made through a communication interface in each computing device.
  • instructions for performing the functions of the parsing module 1020 and the matching module 1030 are stored in the memory 1106 of the first computing device 102A.
  • instructions for executing the functions of the interactive module 1010 are stored in the memory 1106 in the second computing device 102B.
  • connection mode between the computing device clusters shown in Figure 13 can be: Considering that the matching module 1030 in the proofreading method of sample annotation provided by this application needs the output results of the parsing module 1020, it is considered that the parsing module 1020 and the matching module 1030 will be executed.
  • the functions implemented are handed over to the first computing device 102A, and considering that the sample annotation proofreading method provided in this application may interact with the terminal device 101, it is considered that the functions implemented by the interaction module 1010 are handed over to the second computing device 102B. implement.
  • first computing device 102A shown in FIG. 13 can also be performed by multiple computing devices 102 .
  • second computing device 102B can also be performed by multiple computing devices 102 .
  • An embodiment of the present application also provides a computer program product containing instructions.
  • the computer program product may be a software or program product containing instructions capable of running on a computing device or stored in any available medium.
  • the computer program product is run on at least one computing device, the at least one computing device is caused to execute the verification method of sample annotation.
  • An embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium may be any available medium that a computing device can store or a data storage device such as a data center that contains one or more available media.
  • the available media may be magnetic media (for example, floppy disks, hard disks, magnetic tapes), optical media (for example, digital video discs (DVD)), or semiconductor media (for example, solid state drives), etc.
  • the computer-readable storage medium includes instructions that instruct a computing device to perform a sample annotation proofreading method.
  • first and second are used to distinguish identical or similar items with substantially the same functions and functions. It should be understood that there is no logical or logical connection between “first” and “second”. Timing dependencies do not limit the number and execution order. It should also be understood that, although the following description uses the terms “first”, “second”, etc. to describe various elements, these elements should not be limited by the terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, a first structure analysis result may be referred to as a second structure analysis result, and similarly, the second structure analysis result may be referred to as a first structure analysis result. Both the first structure analysis result and the second structure analysis result may be structure analysis results, and in some cases, may be separate and different structure analysis results.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Computation (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Machine Translation (AREA)

Abstract

本申请提供了一种样本标注的校对方法、装置、计算设备集群和存储介质，属于神经网络技术领域。该方法包括：获取目标样本，目标样本为待校对的标注样本，对目标样本进行结构解析，获得目标样本的第一结构解析结果，将第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配，若多个已校对的标注样本的结构解析结果中不存在与第一结构解析结果匹配的目标结构解析结果，则输出提示消息，该提示消息用于提示用户检查目标样本的标注结果。采用本申请的方案，能够为用户提示可能标注错误的样本，提升样本标注的准确率。

Description

样本标注的校对方法、装置、计算设备集群和存储介质
本申请要求于2022年05月23日提交的申请号为202210562530.X，发明名称为“一种构词规则辅助的命名实体标注系统”和2022年08月16日提交的申请号为202210986086.4，发明名称为“样本标注的校对方法、装置、计算设备集群和存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及神经网络技术领域，特别涉及一种样本标注的校对方法、装置、计算设备集群和存储介质。
背景技术
在神经网络技术领域中，通常会使用到有标注结果的样本训练神经网络模型，样本的标注结果准确性，直接会影响神经网络模型训练的效果。
相关技术中，人工确定需要标注的样本，制定详细的标注规范，然后人工按照标注规范，使用标注系统中的标注工具对样本进行标注，标注工具可以是画笔等。标注系统对样本的标注结果进行记录。
由于人工需要标注大量样本，所以有可能会对样本标注错误，但是标注工具并没有确定错误标注的样本的能力。这样，在使用错误标注样本训练神经网络模型时，会影响神经网络模型的训练效果。
发明内容
本申请提供了一种样本标注的校对方法、装置、计算设备集群和存储介质，能够对错误标注的样本进行提示，提升样本标注的准确率。
第一方面，本申请提供了一种样本标注的校对方法，该方法包括：获取目标样本，其中，该目标样本为待校对的标注样本，对该目标样本进行结构解析，获得该目标样本的第一结构解析结果，将该第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配，若该多个已校对的标注样本的结构解析结果中不存在与该第一结构解析结果匹配的目标结构解析结果，则输出提示消息，其中，该提示消息用于提示用户检查该目标样本的标注结果。
本申请所示的方案中，对于待校对的标注样本，能够使用对应的结构解析结果和多个已校对的标注样本的结构解析结果，确定该样本的标注结果是否错误，提示用户再次确认，使得样本的标注结果的准确率比较高，进而使得神经网络模型的训练效果比较好。
在一种示例中，该目标样本为目标命名实体样本，该第一结构解析结果为词内部结构，该对该目标样本进行结构解析，获得该目标样本的第一结构解析结果，包括：使用词内部结构分析模型，对该目标样本进行词内部结构解析，获得该目标样本的词内部结构。
本申请所示的方案中，目标样本为命名实体样本时，结构解析结果为词内部结构，可以使用词内部结构分析模型，准确获得目标样本的词内部结构。
在一种示例中，该将该第一结构解析结果与多个已校对的标注样本的结构解析结果进行 匹配，包括：从该多个已校对的标注样本中，确定与该目标命名实体样本的类别相同的一个或多个命名实体样本，若该第一结构解析结果不存在于该一个或多个命名实体的结构解析结果中，则确定该多个已校对的标注样本的结构解析结果中不存在与该第一结构解析结果匹配的目标结构解析结果，若该第一结构解析结果存在于该一个或多个命名实体的结构解析结果中，则确定该多个已校对的标注样本的结构解析结果中存在与该第一结构解析结果匹配的目标结构解析结果。
本申请所示的方案中，在进行结构解析结果匹配时，可以确定与目标样本的类别相同的一个或多个命名实体样本，在类别相同的命名实体的结构解析结果中进行匹配，能够使得匹配准确率更高。
在一种示例中，在获得该目标样本的第一结构解析结果之后，还包括：生成该第一结构解析结果的确认界面，该确认界面用于向用户显示该第一结构解析结果，接收该用户输入的确认指令，该确认指令用于对该第一结构解析结果进行修改或确认。
本申请所示的方案中，在获得目标样本的第一结构解析结果之后，目标样本的结构解析结果是由模型分析获得的，所以有可能不准确，生成确认界面，供用户确认第一结构解析结果是否准确，用户可以对第一结构解析结果进行确认或者修改，这样能够为用户提供确认和修改第一结构解析结果的方式。
在一种示例中，该方法还包括：基于修改后的第一结构解析结果，更新所述词内部结构分析模型。
本申请所示的方案中，使用用户修改后的第一结构解析结果更新词内部结构分析模型，能够使得词内部结构分析模型后续可以识别出修改后的第一结构解析结果，提升词内部结构分析模型的泛化能力。
在一种示例中，该方法还包括：在接收该用户输入的确认指令之后，将确认后或修改后的该第一结构解析结果添加至该已校对的标注样本的结构解析结果中。
本申请所示的方案中，在第一结构解析结果被确认正确后，被添加至已校对的标注样本的结构解析结果中，能够使得后续第一结构解析结果能够与已校对的标注样本的结构解析结果匹配。
在一种示例中，该输出提示消息之前，还包括：在该目标样本所属的语句中，获取目标短语，其中，该目标短语由该目标样本与该目标样本的相邻位置的词语组成，对该目标短语进行结构解析，获得该目标短语的第二结构解析结果，确定该多个已校对的标注样本的结构解析结果中存在与该第二结构解析结果匹配的结构解析结果。
本申请所示的方案中，在多个已校对的标注样本的结构解析结果中不存在与第一结构解析结果匹配的目标结构解析结果时，能够使用语句中目标样本相邻位置的词语，对目标样本进行扩展，防止由于用户少选择词语导致结构解析结果不匹配。
在一种示例中，该提示消息还用于提示该目标样本对应的正确标注结果。这样，使得用户更快速地确认标注结果。
在一种示例中，该获取目标样本，包括：获取用户标注的目标样本；或者，获取预标注模型标注的目标样本。
本申请所示的方案中，不仅能够对用户标注的样本进行校对，还能够对预标注模型标注的样本进行校对。
第二方面，本申请提供了一种样本标注的校对装置，该装置包括至少一个模块，该至少一个模块用于实现上述第一方面或第一方面中任一种示例所提供的样本标注的校对方法。
在一些实施例中，样本标注的校对装置中的模块通过软件实现，样本标注的校对装置中的模块是程序模块。在另一些实施例中，样本标注的校对装置中的模块通过硬件或固件实现。
第三方面，本申请提供了一种计算设备集群，该计算设备集群包括至少一个计算设备，每个计算设备包括处理器和存储器，该至少一个计算设备的处理器用于执行该至少一个计算设备的存储器中存储的指令，以使得该计算设备集群执行上述第一方面或第一方面中任一种示例所提供的样本标注的校对方法。
第四方面，本申请提供了一种计算机可读存储介质，该计算机可读存储介质包括计算机程序指令，当该计算机程序指令由计算设备集群执行时，该计算设备集群执行上述第一方面或第一方面中任一种示例所提供的样本标注的校对方法。
第五方面，本申请提供了一种包含指令的计算机程序产品，当该指令被计算设备集群运行时，使得所述计算设备集群执行上述第一方面或第一方面中任一种示例所提供的样本标注的校对方法。
附图说明
图1是本申请一个示例性实施例提供的系统架构的示意图；
图2是本申请一个示例性实施例提供的系统架构的示意图；
图3是本申请一个示例性实施例提供的样本标注的校对方法流程示意图；
图4是本申请一个示例性实施例提供的标注界面的示意图；
图5是本申请一个示例性实施例提供的命名实体库的示意图；
图6是本申请一个示例性实施例提供的结构解析结果的匹配过程示意图；
图7是本申请一个示例性实施例提供的样本标注的校对方法流程示意图；
图8是本申请一个示例性实施例提供的确认界面的示意图；
图9是本申请一个示例性实施例提供的多个样本的结构解析结果；
图10是本申请一个示例性实施例提供的样本标注的校对装置的结构示意图；
图11是本申请一个示例性实施例提供的计算设备的结构示意图；
图12是本申请一个示例性实施例提供的计算设备集群的结构示意图；
图13是本申请一个示例性实施例提供的计算设备的连接示意图。
具体实施方式
为使本申请的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
为了便于对本申请实施例的理解，下面首先介绍所涉及到的名词的概念。
1、命名实体，指人名、组织机构名、职称和地名以及其它所有以名称为标识的实体。更广泛的命名实体还包括数字、日期和地址等。
2、命名实体识别，指将语句中所有符合特定类型的命名实体识别出来，例如，语句为“今天操场上A机构在组织活动”，需要在语句中识别出“A机构”是一个组织机构名。
3、词内部角色，指短语中每个字的语义角色。在中文中，字的词内部角色包括词根(root)、 主谓(subject-predicate)、动宾(object，obj)、定中(attribute，att)、状中(adverbial，adv)、动补(complement，cmp)、并列(coordinate，coo)、介宾(preposition-object，pobj)、附加(adjunct，adjct)、碎片(frag)和重叠(repet)等。例如，对于中文中的词语“AB市”中，“A”、“B”和“市”的词内部角色分别为“att”、“frag”和“root”。在其它语种中，短语也有对应的词内部角色。
4、词内部角色序列，指词语的角色组成的序列。例如，词语“AB市”的词内部角色序列为“att frag root”。在词内部角色序列中，相邻的角色之间可以使用空格隔开，也可以使用其它字符隔开。
5、词内部结构，指使用词内部角色序列和root字来表示词语的内部结构。例如，词语“AB市”的词内部结构为内部角色序列“att frag root”以及root字“市”。
下面描述本申请实施例的背景。
在神经网络技术领域中，样本标注对神经网络模型的构建至关重要，样本的标注结果的准确性，直接会影响神经网络模型训练的效果。例如，命名实体识别在关系抽取任务、实体链接、指代消融和知识图谱的神经网络模型构建中发挥了重要作用，命名实体识别准确，可以使得这些任务的神经网络模型的性能更好。
相关技术中，用户(标注人员)使用标注系统中的标注工具对样本进行标注，标注工具可以是画笔等。然而由于用户需要标注大量样本，所以有可能会对样本标注错误。但是标注工具并没有确定错误标注的样本的能力。这样，在使用错误标注样本训练神经网络模型时，会影响神经网络模型的训练效果。例如，语句中包括“江边大道”这个地名实体，但是用户将“江边大”标注为地名实体，漏掉“道”，导致神经网络模型的泛化性较差。再例如，由于标注样本数量大，通常会由多个标注者进行标注，所以有可能会导致对相同命名实体标注不一致。再例如，在标注“A市”地名实体时，有可能会出现一个用户将“A”标注为地名实体，而另一个用户将“A市”标注为地名实体，在使用标注完成的命名实体训练神经网络模型时，使得对神经网络模型训练造成困扰。
本申请实施例中，在样本标注完成后，能够使用样本的结构解析结构分析样本的标注结果是否错误，若标注结果错误，则提示用户检查样本的标注结果，从而能够提升样本标注的准确性。
下面描述本申请实施例的系统架构。
在一种示例中，本申请实施例提供了一种系统架构100。如图1所示，系统架构100包括终端设备101和计算设备102。终端设备101与计算设备102之间通过有线或无线网络连接。其中，终端设备101是用户使用的设备，如台式电脑、笔记本电脑、平板电脑或手机等，终端设备101用于用户与计算设备102进行交互。例如，用户可以使用终端设备101为样本添加标注结果等。计算设备102可以是服务器等，计算设备102用于确定目标样本的结构解析结果，并且将目标样本的结构解析结果与多个已校对的标注样本的结构解析结果进行匹配等，目标样本为标注完成，且未校对的样本。
在另一种示例中，本申请实施例提供了另一种系统架构200。如图2所示，系统架构200包括终端设备101和公有云201。终端设备101与公有云201通过有线或者无线网络连接。终端设备101是用户使用的设备，终端设备101用于用户与公有云201进行交互。公有云201是云计算模式下利用基础资源向用户提供云服务的实体，公有云201也可以认为是一个云环 境。公有云201包括云数据中心，云数据中心包括云服务提供商拥有的大量基础资源，该大量基础资源包括计算资源、存储资源和网络资源，云数据中心包括的计算资源可以是计算设备集群，计算设备集群包括至少一个计算设备，计算设备可以是服务器等。在用户使用云服务时，用户可以通过应用程序接口(application program interface，API)或者图形用户界面(graphical user interface，GUI)上传目标样本至公有云201等，公有云201中的计算设备集群接收用户上传的目标样本，确定目标样本的结构解析结果，并且将目标样本的结构解析结果与多个已校对的标注样本的结构解析结果进行匹配，基于匹配结果向终端设备101输出提示消息。
在再一种示例中，本申请实施例还提供了一种系统架构。该系统架构包括终端设备，终端设备是用户使用的设备。终端设备执行本申请实施例中样本标注的校对方法。
下面描述本申请实施例中样本标注的校对方法流程。
图3提供了样本标注的校对方法流程，在图3中以图1所示的系统架构100为例描述该样本标注的校对方法流程。在图3所示的流程中，终端设备为前文中终端设备101，计算设备为前文中计算设备102。
步骤301，获取目标样本。
其中，该目标样本为待校对的标注样本，即为已标注且未校对的标注样本，目标样本为任一待校对的标注样本。在本申请实施例中，目标样本可以是能够基于结构解析结果分析目标样本的标注结果是否准确的样本，例如，目标样本可以是命名实体或者分词样本等。
在本实施例中，用户使用终端设备对目标样本标注完成后，在对目标样本的标注界面中，显示有开始分析选项，用户可以点击开始分析选项，终端设备向计算设备发送分析请求。计算设备接收该分析请求，获取目标样本。例如，参见图4，目标样本为目标命名实体样本，标注界面中显示有语句“Y大学位于A市西北郊”、开始分析选项以及命名实体类别的标识，地名使用实线框标识，组织机构名使用第一种类型的虚线框标识，职称使用第二种类型的虚线框标识，用户在该语句中标注“Y大学”为组织机构名，并在该语句中标注“A市”为地名。图4中仅一种示例，在另一些实现中，命名实体类别的标识使用颜色进行区分，如地名使用红色标识，组织机构名使用绿色标识，在用户标注样本后，样本变为对应的类别的颜色。另外，在语句中进行标注时，标注界面中还可以显示有下一句选项和上一句选项等，下一句选项用于切换至下一个语句进行标注或查看，上一句选项用于切换至上一个语句进行标注或查看。
或者，用户使用终端设备对多个样本标注完成后，在样本标注的标注界面中，显示有开始分析选项，用户可以点击开始分析选项，终端设备向计算设备发送分析请求。计算设备接收该分析请求，获取用户标注的多个样本，多个样本包括目标样本。
或者，用户使用终端设备对目标样本标注完成后，终端设备将目标样本的标注结果提交至计算设备，计算设备获取到目标样本。
或者，标注样本前，训练有预标注模型，预标注模型为对样本进行标注的模型。预标注模型对目标样本标注完成后，计算设备获取标注完成的目标样本。
以上仅为可能的四种可能实现方式，本申请实施例不对获取目标样本的方式进行限定。
步骤302，对该目标样本进行结构解析，获得该目标样本的第一结构解析结果。
在本实施例中，计算设备中存储有结构解析模型，或者，计算设备使用样本与样本的结 构解析结果进行训练获得结构解析模型，或者，计算设备从其它设备获取结构解析模型。计算设备使用结构解析模型解析目标样本的结构，获得目标样本的结构解析结果，称为是第一结构解析结果。
需要说明的是，在目标样本的类型不相同时，对应的结构解析模型也不相同。例如，在目标样本为目标命名实体样本时，用于对目标样本进行结构解析的结构解析模型为词内部结构分析模型，在目标样本为目标分词样本时，用于对目标分词样本进行结构解析的结构解析模型为句子结构分析模型。
在一种示例中，目标样本为目标命名实体样本，第一结构解析结果为词内部结构，获得目标样本的第一结构解析结果的过程为：
在样本标注前，训练有词内部结构分析模型，该词内部结构分析模型用于分析词内部结构。将目标样本输入词内部结构分析模型中，词内部结构分析模型的输出即为目标样本的词内部结构，也就是说第一结构解析结果为词内部结构。例如，目标样本为“××地宫”，对“××地宫”进行词内部结构解析后，得到的词内部结构为“att att att root”，root字“宫”。在本申请实施例中一个“×”表示一个字，那么“××地宫”为四个字。再例如，目标样本为“××区”，对“××区”进行词内部结构解析后，得到的词内部结构为“att frag root”，root字“区”。
在一种示例中，目标样本为目标分词样本时，第一结构解析结果为目标分词样本的分词结构，获得目标样本的第一结构解析结果的过程为：
在标注样本前，训练有分词样本的结构解析模型，该结构解析模型用于分析分词样本的结构。将分词样本输入该结构解析模型中，该结构解析模型的输出即为分词样本的第一结构解析结果。例如，分词样本为“我/来到/××/区”，“我/来到/××/区”的第一结构解析结果为“代词动词名词”。
在一种示例中，计算设备在确定第一结构解析结果后，可以向用户展示第一结构解析结果。例如，用户在语句中“××大学要放假了”标注了命名实体“××大学”为组织机构名，如图5所示的结构解析结果示意图，示出了“××大学”的第一结构解析结果，第一结构解析结果为“att att att root”，root字为“学”。
步骤303，将第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配。
其中，已校对的标注样本为标注结果被确认为正确的标注样本。计算设备中存储有已校对的标注样本的结构解析结果，或者从其它设备获取已校对的标注样本的结构解析结果，或者，计算设备中对已校对的标注样本进行结构解析，获得目标结构解析结果，目标结构解析结果被用户确认为正确，此处的用户指示结构解析结果的审核者。
在本实施例中，计算设备判断多个已校对的标注样本的结构解析结果中是否存在与第一结构解析结果匹配的目标结构解析结果。若多个已校对的标注样本的结构解析结果中不存在与第一结构解析结果匹配的目标结构解析结果，则确定第一结构解析结果匹配到结构解析结果，否则确定第一结构解析结果未匹配到结构解析结果。
在一种示例中，命名实体的类别有多种，如人名、组织机构名、职称和地名等。为了快速且准确判断第一结构解析结果与已校对的标注命名实体的结构解析结果是否匹配，可以确定目标样本的类别，为了方便描述，将目标样本的类别称为是目标类别。在多个已校对的命名实体中选取目标类别的命名实体样本，然后将第一结构解析结果与目标类别的命名实体样 本的结构解析结果进行匹配。若第一结构解析结果不存在于目标类别的命名实体的结构解析结果中，则第一结构解析结果未匹配到结构解析结果，若第一结构解析结果存在于目标类别的命名实体的结构解析结果中，则确定第一结构解析结果匹配到结构解析结果。这样，仅匹配类别相同的命名实体的结构解析结果，不仅匹配数量比较少，而且匹配结果更准确。
例如，对于已校对的命名实体的结构解析结果，按照命名实体的类别划分为不同的命名实体库。例如，人名、组织机构名、职称和地名分别对应不同的命名实体库，图6示出了地名对应的命名实体库，在命名实体库中包括命名实体对应的词内部结构，如图6所示，词内部结构1为“att frag root”，root字为“区”，词内部结构2为“att root”，root字为“国”，词内部结构3为“att att root”，root字为“湾”等。假设目标样本为地名命名实体，在进行匹配时，使用地名对应的命名实体库进行匹配。
假设目标样本为“××市”，第一结构解析结果为“att frag root”，root字为“市”，地名对应的命名实体库中不存在结构解析结果“att frag root”，root字为“市”，说明第一结构解析结果与地名对应的命名实体库中的结构解析结果不匹配，参见图7。
假设目标样本为“××市”，第一结构解析结果为“att frag root”，root字为“市”，地名对应的命名实体库中存在结构解析结果“att frag root”，root字为“市”，说明地名对应的命名实体库中的结构解析结果中存在与第一结构解析结果匹配的目标结构解析结果，目标结构解析结果为“att frag root”，root字为“市”。
假设目标样本为“××市长”，被标注的类别为地名，第一结构解析结果为“att att att root”，root字为“长”，地名对应的命名实体库中存在词内部角色序列一致(“××地宫”地宫的结构解析结果为“att att att root”，root字“宫”)，但是不存在root字为“长”的结构解析结果，说明第一结构解析结果与地名对应的命名实体库中的结构解析结果不匹配。
在另一种示例中，目标样本为分词样本的情况下，与命名实体样本的匹配方式一致。例如，分词样本为“我/来到/××/区”，“我/来到/××/区”的第一结构解析结果为“代词动词名词”，若已校对的标注样本的结构解析结果中存在“代词动词名词”，则确定第一结构解析结果匹配到结构解析结果，反之则未匹配到结构解析结果。
步骤304，若该多个已校对的标注样本的结构解析结果中不存在与该第一结构解析结果匹配的目标结构解析结果，则输出提示消息，其中，该提示消息用于提示用户检查该目标样本的标注结果。
在本实施例中，在多个已校对的标注样本的结构解析结果中不存在与第一结构解析结果匹配的目标结构解析结果的情况下，说明目标样本的标注结果有可能不准确，计算设备向终端设备发送提示消息。终端设备接收到该提示消息后，可以显示该提示消息的内容，该提示消息用于提示用户检查目标样本的标注结果。本申请实施例不对提示消息的具体内容进行限定。用户可以再次确认目标样本的标注结果是否准确。
在一种示例中，在显示提示消息的界面中显示有查看选项，用户可以点击查看选项，返回标注界面，用户在标注界面中可以对目标样本的标注结果进行修改，在修改完成后进行提交。计算设备接收到用户重新提交的标注结果后，可以再次执行图3中步骤301至步骤303，确定重新提交的标注结果是否错误。
另外，在用户重新提交的标注结果与原来的标注结果相同的情况下，也可以将目标样本发送给另两个用户进行查看，该另两个用户为标注级别比较高的标注者。若该另两个用户确 定原来的标注结果正确，则存储原来的标注结果，若该另两个用户均修改了标注结果，且修改后的标注结果一样，可以存储修改后的标注结果。
另外，多个已校对的标注样本的结构解析结果中存在目标结构解析结果，且目标结构解析结果与第一结构解析结果匹配，说明目标样本的标注结果准确，将目标样本的标注结果进行存储。
在一种示例中，为了使得用户更快速地检查目标样本的标注结果，提示消息中还用于指示目标样本对应的正确标注结果，这样，用户在检查目标样本的标注结果时，可以参考提示消息中指示的正确标注结果。
在一种示例中，由于目标样本的第一结构解析结果是通过结构分析模型获得的，所以为了提升输出提示消息的准确性，可以由用户判断第一结构解析结果是否正确，进而确定是否输出提示消息，如图8所示，处理过程参见步骤303和步骤304中输入提示消息之前包括的步骤305至步骤312，该处理过程为可选的处理过程。
步骤305，若多个已校对的标注样本的结构解析结果中不存在与第一结构解析结果匹配的目标结构解析结果，则输出第一结构解析结果的确认消息。
在本实施例中，计算设备向终端设备发送第一结构解析结果的确认消息。终端设备接收第一结构解析结果的确认消息，生成确认消息对应的确认界面。或者，计算设备向终端设备发送确认界面。例如，参见图9所示的该确认界面的示意图，目标样本为“××市”，第一结构解析结果为“att frag root”，root字为“市”，在图9中，还显示“请确认当前的结构解析结果是否正确，若不正确，请修改，若正确请确认”的提示内容、确认选项和修改选项。用户认为第一结构解析结果正确，可以点击确认选项，终端设备会向计算设备发送确认正确消息。用户认为第一结构解析结果不正确，可以点击修改选项，触发终端设备显示修改界面，用户可以对第一结构解析结果进行修改，在修改完成后进行提交，为了方便描述，修改后的第一结构解析结果描述为第三结构解析结果。
另外，结构解析模型在对目标样本进行结构解析时，可以得到多种结构解析结果，其中，第一结构解析结果是概率最高的结构解析结果。在计算设备向终端设备发送第一结构解析结果的确认消息时，还可以携带目标样本的其余结构解析结果，该其余结构解析结果可以携带在该确认消息中，也可以单独发送。在显示第一结构解析结果的确认界面时，可以将多种结构解析结果显示，为用户确认第一结构解析结果提供参考。此处，其余结构解析结果可以与第一结构解析结果均显示在确认界面中，也可以显示在修改界面中。
需要说明的是，图9仅仅是一种可能的确认界面的示意图，凡是可以对第一结构解析结果进行确认的界面，均可应用于本申请实施例中。
步骤306，接收用户输入的确认指令。
在本实施例中，终端设备接收到用户输入的确认指令，向计算设备发送确认指令，若该确认指令指示确认正确，则计算设备接收终端设备发送的确认指令后，可以向终端设备发送提示消息，该提示消息用于提示用户检查目标样本的标注结果。
步骤307，基于第一结构解析结果，更新词内部结构分析模型。
在本实施例中，在用户确认第一结构解析结果正确的情况下，还可以使用第一结构解析结果更新词内部结构分析模型，使得词内部结构分析模型结构解析的准确率更高。
步骤308，将第一结构解析结果添加至该已校对的标注样本的结构解析结果中。
在本实施例中，在用户确认第一结构解析结果正确的情况下，还可以将第一结构解析结果添加至已校对的标注样本的结构解析结果中，使得后续再存在与目标样本类似的样本时，能匹配到对应的结构解析结果。
在一种示例中，在用户确认第一结构解析结果正确的情况下，还可以将第一结构解析结果添加至目标样本对应的类别的命名实体库中，使得后续再存在与目标样本类似的样本时，可以在该命名实体库中能匹配到对应的结构解析结果。例如，目标样本为“××寺”，第一结构解析结果为“att frag root”，root字为“寺”，在地名对应的命名实体库中存在词内部角色序列一致，且root字不一致的结构解析结果，用户确认第一结构解析结果正确，将“att frag root”，root字为“寺”添加至地名对应的命名实体库中。
步骤309，接收用户输入的第三结构解析结果，将第三结构解析结果与多个已校对的标注样本的结构解析结果进行匹配。
在本实施例中，在步骤305中，用户认为第一结构解析结果不正确，对第一结构解析结果进行修改，得到的修改结果为第三结构解析结果。计算设备接收第三结构解析结果，判断多个已校对的标注样本的结构解析结果中是否存在与第三结构解析结果匹配的结构解析结果，若多个已校对的标注样本的结构解析结果中不存在与第三结构解析结果匹配的结构解析结果，则确定第三结构解析结果在已校对的标注样本的结构解析结果中未匹配到结构解析结果，否则确定第三结构解析结果在已校对的标注样本的结构解析结果中匹配到结构解析结果。
在一种示例中，在将第三结构解析结果与多个已校对的标注样本的结构解析结果进行匹配时，可以先确定目标样本的目标类别，确定该目标类别的命名实体对应的结构解析结果，判断第三结构解析结果是否存在于该目标类别的命名实体对应的结构解析结果中。
步骤310，若多个已校对的标注样本的结构解析结果中不存在与第三结构解析结果匹配的结构解析结果，则跳转至步骤304中的输出提示消息。
在本实施例中，若多个已校对的标注样本的结构解析结果中不存在与第三结构解析结果匹配的结构解析结果，则确定目标样本的标注结果有可能错误，计算设备向终端设备输出提示消息，该提示消息用于提示用户检查目标样本的标注结果。
另外，若多个已校对的标注样本的结构解析结果中存在与第三结构解析结果匹配的结构解析结果，则确认第一结构解析结果与多个已校对的标注样本的结构解析结果不匹配可能是由于第一结构解析结果错误，而不是由于目标样本的标注结果错误，所以可以确认目标样本的标注结果正确。
步骤311，基于第三结构解析结果，更新词内部结构分析模型。
在本实施例中，在用户输入第三结构解析结果的情况下，还可以使用第三结构解析结果更新词内部结构分析模型，使得词内部结构分析模型的泛化能力更强。
步骤312，将第三结构解析结果添加至该已校对的标注样本的结构解析结果中。
在本实施例中，在用户输入第三结构解析结果之后，若第三结构解析结果与多个已校对的标注样本的结构解析结果不匹配，则还可以将第三结构解析结果添加至已校对的标注样本的结构解析结果中，使得后续再存在与目标样本类似的样本时，能匹配到对应的结构解析结果。
可选地，在用户输入第三结构解析结果之后，若第三结构解析结果与目标类别的命名实体的结构解析结果不匹配，则还可以将第三结构解析结果添加至目标类别的命名实体库中， 使得后续再存在与目标样本类似的样本时，可以在该命名实体库中能匹配到对应的结构解析结果。
需要说明的是，在图8所示的流程中，步骤307和步骤308与步骤304没有先后顺序，并且步骤307和步骤308也没有先后顺序。步骤311和步骤312与步骤304没有先后顺序，并且步骤311和步骤312也没有先后顺序。
在一种示例中，当存在多个用户进行命名实体标注时，对于同一个命名实体，有可能出现不同的用户标注不一致的情况，例如，在语句中存在“××市”，有的用户将“××”标注为地名，有的用户将“××市”标注为地名，“××市”中的“××”标注为地名时，结构解析结果不会存在于地名对应的命名实体库中，此种情况下，在步骤304之前可进行如下处理。
在目标样本所属的语句中，获取目标短语，其中，目标短语由目标样本与目标样本的相邻位置的词语组成，对目标短语进行结构解析，获得目标短语的第二结构解析结果，确定多个已校对的标注样本的结构解析结果中存在与第二结构解析结果匹配的结构解析结果。
在本实施例中，计算设备在目标样本所属的语句中，确定目标样本相邻位置的词语，该词语与目标样本组成目标短语。例如，目标样本为“××”，所属的语句为“我在“××”市定居”，目标短语为“××市”。此处相邻位置的词语可以是目标样本之后的一个字，具体取目标样本之后的字的数目可以根据实际的应用场景设置，本申请实施例不进行限定。
计算设备将目标短语输入至词内部结构解析模型中，获得目标短语的结构解析结果，即第二结构解析结果。计算设备判断多个已校对的标注样本的结构解析结果中是否存在与第二结构解析结果匹配的结构解析结果。若存在，则可以输出提示消息，该提示消息用于提示用户检查目标样本的标注结果。这样，在多个用户标注不一致时，可以挖掘出标注错误的样本，提升标注准确性。此处在判断多个已校对的标注样本的结构解析结果中是否存在与第二结构解析结果匹配的结构解析结果时，若目标短语为第一命名实体样本，可以先确定目标短语的类别，在多个已校对的标注样本中，确定与第一命名实体样本的类别相同的一个或多个命名实体样本，判断第二结构解析结果是否存在于该一个或多个命名实体的结构解析结果中，若存在，则可以输出提示消息。
可选地，该提示消息还用于指示目标样本对应的正确标注结果。这样，用户在检查目标样本的标注结果时，可以参考提示消息中指示的正确标注结果。例如，目标样本为“××”，目标短语为“××市”，提示消息中可以携带“××市”进行提示。
可选地，计算设备确定多个已校对的标注样本的结构解析结果中存在与第二结构解析结果匹配的结构解析结果时，计算设备还可以向终端设备发送第二结构解析结果的确认消息。终端设备接收第二结构解析结果的确认消息，生成第二结构解析结果的确认界面。此处显示第二结构解析结果与前文中图9显示第一结构解析结果一样，不再赘述。
若用户确认第二结构解析结果正确，则计算设备会接收到用户输入的确认正确消息，可以向终端设备发送提示消息，该提示消息用于提示用户检查目标样本的标注结果。若用户确认第二结构解析结果不正确，则输出第一结构解析结果的确认消息(即执行上述步骤305)。
另外，若多个已校对的标注样本的结构解析结果中不存在与第二结构解析结果匹配的结构解析结果，则输出第一结构解析结果的确认消息(即执行上述步骤305)。
另外，在用户确认第二结构解析结果正确后，或者计算设备确定多个已校对的标注样本的结构解析结果中存在与第二结构解析结果匹配的结构解析结果后，计算设备还可以使用第 二结构解析结果更新词内部结构分析模型，使得词内部结构分析模型更准确。
需要说明的是，在上述描述中，是以目标样本一个样本为例进行描述，本申请实施例中，也可以同时对多个样本进行结构解析，判断多个样本是否标注错误，为用户挖掘出可能标注错误的样本。例如，用户在一个语句中标注了两个样本，可以同时判断这两个样本是否标注错误。
还需要说明的是，在前文的描述中，是以系统架构100为例进行说明，在以其它系统架构实现样本标注的校对方法时，执行过程与前文中的描述类似，此处不再赘述。
采用本申请所示的方案，对于待校对的标注样本，能够基于样本的结构解析结果，挖掘出可能标注错误的样本，提示用户再次确认，使得样本的标注结果的准确率比较高，进而使得神经网络模型的训练效果比较好。
而且在多个用户进行样本标注时，能够发现标注不一致的情况，可以减少神经网络模型在训练时的混淆，提高神经网络模型的识别能力。
下面描述本申请实施提供的样本标注的校对装置。
图10是本申请实施例提供的样本标注的校对装置的结构图。该装置可以通过软件、硬件或者两者的结合实现成为装置中的部分或者全部。本申请实施例提供的装置可以实现本申请实施例图3和图7所示的流程，该装置包括：交互模块1010、解析模块1020和匹配模块1030，其中：
交互模块1010，用于获取目标样本，其中，所述目标样本为待校对的标注样本，具体可以用于实现步骤301的交互功能以及执行步骤301包含的隐含步骤；
解析模块1020，用于对所述目标样本进行结构解析，获得所述目标样本的第一结构解析结果，具体可以用于实现步骤302的解析功能以及执行步骤302包含的隐含步骤；
匹配模块1030，用于将所述第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配，具体可以用于实现步骤303的匹配功能以及执行步骤303包含的隐含步骤；
所述交互模块1010，还用于若所述多个已校对的标注样本的结构解析结果中不存在与所述第一结构解析结果匹配的目标结构解析结果，则输出提示消息，其中，所述提示消息用于提示用户检查所述目标样本的标注结果，具体可以用于实现步骤304的交互功能以及执行步骤304包含的隐含步骤。
在一种示例中，所述目标样本为目标命名实体样本，所述第一结构解析结果为词内部结构；
所述解析模块1020，用于：
使用词内部结构分析模型，对所述目标样本进行词内部结构解析，获得所述目标样本的词内部结构。
在一种示例中，所述匹配模块1030，用于：
从所述多个已校对的标注样本中，确定与所述目标命名实体样本的类别相同的一个或多个命名实体样本；
若所述第一结构解析结果不存在于所述一个或多个命名实体的结构解析结果中，则确定所述多个已校对的标注样本的结构解析结果中不存在与所述第一结构解析结果匹配的目标结构解析结果；
若所述第一结构解析结果存在于所述一个或多个命名实体的结构解析结果中，则确定所 述多个已校对的标注样本的结构解析结果中存在与所述第一结构解析结果匹配的目标结构解析结果。
在一种示例中，所述交互模块1010，还用于：
在获得所述目标样本的第一结构解析结果之后，生成所述第一结构解析结果的确认界面，所述确认界面用于向用户显示所述第一结构解析结果；
接收所述用户输入的确认指令，所述确认指令用于对所述第一结构解析结果进行修改或确认。
在一种示例中，所述解析模块1020，还用于：
基于所述修改后的第一结构解析结果，更新所述词内部结构分析模型。
在一种示例中，所述交互模块1010，还用于：
在接收所述用户输入的确认指令之后，将确认后或修改后的所述第一结构解析结果添加至所述已校对的标注样本的结构解析结果中。
在一种示例中，所述匹配模块1030，还用于：
在输出提示消息之前，在所述目标样本所属的语句中，获取目标短语，其中，所述目标短语由所述目标样本与所述目标样本的相邻位置的词语组成；
对所述目标短语进行结构解析，获得所述目标短语的第二结构解析结果；
确定所述多个已校对的标注样本的结构解析结果中存在与所述第二结构解析结果匹配的结构解析结果。
在一种示例中，所述提示消息还用于提示所述目标样本对应的正确标注结果。
在一种示例中，所述交互模块1010，用于：
获取用户标注的目标样本；或者，
获取预标注模型标注的目标样本。
其中，交互模块1010、解析模块1020和匹配模块1030均可以通过软件实现，或者可以通过硬件实现。示例性的，接下来以解析模块1020为例，介绍解析模块1020的实现方式。类似的，交互模块1010和匹配模块1030的实现方式可以参考解析模块1020的实现方式。
模块作为软件功能单元的一种举例，解析模块1020可以包括运行在计算实例上的代码。其中，计算实例可以包括物理主机(计算设备)、虚拟机或容器中的至少一种。进一步地，上述计算实例可以是一台或者多台。例如，解析模块1020可以包括运行在多个主机/虚拟机/容器上的代码。需要说明的是，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的区域(region)中，也可以分布在不同的region中。进一步地，用于运行该代码的多个主机/虚拟机/容器可以分布在相同的可用区(availability zone，AZ)中，也可以分布在不同的AZ中，每个AZ包括一个数据中心或多个地理位置相近的数据中心。其中，通常一个region可以包括多个AZ。
同样，用于运行该代码的多个主机/虚拟机/容器可以分布在同一个虚拟私有云(virtual private cloud，VPC)中，也可以分布在多个VPC中。其中，通常一个VPC设置在一个region内，同一region内两个VPC之间，以及不同region的VPC之间跨区通信需在每个VPC内设置通信网关，经通信网关实现VPC之间的互连。
模块作为硬件功能单元的一种举例，解析模块1020可以包括至少一个计算设备，如服务器等。或者，解析模块1020也可以是利用专用集成电路(application-specific integrated circuit， ASIC)实现或可编程逻辑器件(programmable logic device，PLD)实现的设备等。其中，上述PLD可以是复杂程序逻辑器件(complex programmable logical device，CPLD)、现场可编程门阵列(field-programmable gate array，FPGA)和通用阵列逻辑(generic array logic，GAL)或其任意组合实现。
解析模块1020包括的多个计算设备可以分布在相同的region中，也可以分布在不同的region中。解析模块1020包括的多个计算设备可以分布在相同的AZ中，也可以分布在不同的AZ中。同样，解析模块1020包括的多个计算设备可以分布在同一个VPC中，也可以分布在多个VPC中。其中，所述多个计算设备可以是服务器、ASIC、PLD、CPLD、FPGA和GAL等计算设备的任意组合。
需要说明的是，在其他实施例中，交互模块1010可以用于执行样本标注的校对方法中的任意步骤，解析模块1020可以用于执行样本标注的校对方法中的任意步骤，匹配模块1030可以用于执行样本标注的校对方法中的任意步骤。交互模块1010、解析模块1020和匹配模块1030负责实现的步骤可根据需要指定，通过交互模块1010、解析模块1020和匹配模块1030分别实现样本标注的校对方法中不同的步骤来实现样本标注的校对装置的全部功能。
还需要说明的是，本申请实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时也可以有另外的划分方式。
下面描述本申请实施例提供的计算设备102。
本申请实施例还提供了一种计算设备102。如图11所示，计算设备102包括：总线1102、处理器1104、存储器1106和通信接口1108。处理器1104、存储器1106和通信接口1108之间通过总线1102通信。计算设备102可以是服务器或终端设备。应理解，本申请不限定计算设备102中的处理器和存储器的个数。
总线1102可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线和控制总线等。为便于表示，图11中仅用一条线表示，但并不表示仅有一根总线或一种类型的总线。总线1104可包括在计算设备102各个部件(例如，存储器1106、处理器1104和通信接口1108)之间传送信息的通路。
处理器1104可以包括中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
存储器1106可以包括易失性存储器(volatile memory)，例如，随机存取存储器(random access memory，RAM)。处理器1104还可以包括非易失性存储器(non-volatile memory)，例如，只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard disk drive，HDD)或固态硬盘(solid state drive，SSD)。
存储器1106中存储有可执行的程序代码，处理器1104执行该可执行的程序代码以分别实现后文中交互模块1010、解析模块1020和匹配模块1030的功能，从而实现样本标注的校对方法。也即，存储器1106上存有用于执行样本标注的校对方法的指令。
通信接口1108使用例如但不限于网络接口卡和收发器一类的收发模块，来实现计算设备102与其他设备或通信网络之间的通信。
下面描述本申请实施例提供的计算设备集群。
本申请实施例还提供了一种计算设备集群。该计算设备集群包括至少一个计算设备。该计算设备可以是服务器，例如，该计算设备是中心服务器、边缘服务器，或者是本地数据中心中的本地服务器。在一些实施例中，计算设备也可以是台式机、笔记本电脑或者智能手机等终端设备。
如图12所示，该计算设备集群包括至少一个计算设备102。计算设备集群中的一个或多个计算设备102中的存储器1106中可以存有相同的用于执行样本标注的校对方法的指令。
在一些可能的实现方式中，该计算设备集群中的一个或多个计算设备102的存储器1106中也可以分别存有用于执行样本标注的校对方法的部分指令。换言之，一个或多个计算设备102的组合可以共同执行用于执行样本标注的校对方法的指令。
需要说明的是，计算设备集群中的不同的计算设备102中的存储器1106可以存储不同的指令，分别用于执行后文中样本标注的校对装置的部分功能。也即，不同的计算设备102中的存储器1106存储的指令可以实现交互模块1010、解析模块1020和匹配模块1030中的一个或多个模块的功能。
在一些可能的实现方式中，计算设备集群中的一个或多个计算设备可以通过网络连接。其中，该网络可以是广域网或局域网等等。图13示出了一种可能的实现方式。如图13所示，两个计算设备(第一计算设备102A和第二计算设备102B)之间通过网络进行连接。具体地，通过各个计算设备中的通信接口与该网络进行连接。在这一类可能的实现方式中，第一计算设备102A中的存储器1106中存有执行解析模块1020和匹配模块1030的功能的指令。同时，第二计算设备102B中的存储器1106中存有执行交互模块1010的功能的指令。
图13所示的计算设备集群之间的连接方式可以是考虑到本申请提供的样本标注的校对方法中匹配模块1030需要有解析模块1020的输出结果，因此考虑将执行解析模块1020和匹配模块1030实现的功能交由第一计算设备102A执行，并且考虑到本申请提供的样本标注的校对方法有可能与终端设备101进行交互，因此考虑将执行交互模块1010实现的功能交由第二计算设备102B执行。
应理解，图13中示出的第一计算设备102A的功能也可以由多个计算设备102完成。同样，第二计算设备102B的功能也可以由多个计算设备102完成。
本申请实施例还提供了一种包含指令的计算机程序产品。所述计算机程序产品可以是包含指令的，能够运行在计算设备上或被储存在任何可用介质中的软件或程序产品。当所述计算机程序产品在至少一个计算设备上运行时，使得至少一个计算设备执行样本标注的校对方法。
本申请实施例还提供了一种计算机可读存储介质。所述计算机可读存储介质可以是计算设备能够存储的任何可用介质或者是包含一个或多个可用介质的数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，数字多功能光盘(digital video disc，DVD))、或者半导体介质(例如固态硬盘)等。该计算机可读存储介质包括指令，所述指令指示计算设备执样本标注的校对方法。
本领域普通技术人员可以意识到，结合本申请中所公开的实施例中描述的各方法步骤和单元，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各实施例的步骤及组成。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。本领域普通技 术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请中术语“第一”和“第二”等字样用于对作用和功能基本相同的相同项或相似项进行区分，应理解，“第一”和“第二”之间不具有逻辑或时序上的依赖关系，也不对数量和执行顺序进行限定。还应理解，尽管以下描述使用术语“第一”和“第二”等来描述各种元素，但这些元素不应受术语的限制。这些术语只是用于将一元素与另一元素区别分开。例如，在不脱离各种示例的范围的情况下，第一结构解析结果可以被称为第二结构解析结果，并且类似地，第二结构解析结果可以被称为第一结构解析结果。第一结构解析结果和第二结构解析结果都可以是结构解析结果，并且在某些情况下，可以是单独且不同的结构解析结果。
最后应说明的是：以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的保护范围。

Claims (21)

  1. 一种样本标注的校对方法，其特征在于，所述方法包括：
    获取目标样本，其中，所述目标样本为待校对的标注样本；
    对所述目标样本进行结构解析，获得所述目标样本的第一结构解析结果；
    将所述第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配；
    若所述多个已校对的标注样本的结构解析结果中不存在与所述第一结构解析结果匹配的目标结构解析结果，则输出提示消息，其中，所述提示消息用于提示用户检查所述目标样本的标注结果。
  2. 根据权利要求1所述的方法，其特征在于，所述目标样本为目标命名实体样本，所述第一结构解析结果为词内部结构；
    所述对所述目标样本进行结构解析，获得所述目标样本的第一结构解析结果，包括：
    使用词内部结构分析模型，对所述目标样本进行词内部结构解析，获得所述目标样本的词内部结构。
  3. 根据权利要求2所述的方法，其特征在于，所述将所述第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配，包括：
    从所述多个已校对的标注样本中，确定与所述目标命名实体样本的类别相同的一个或多个命名实体样本；
    若所述第一结构解析结果不存在于所述一个或多个命名实体的结构解析结果中，则确定所述多个已校对的标注样本的结构解析结果中不存在与所述第一结构解析结果匹配的目标结构解析结果；
    若所述第一结构解析结果存在于所述一个或多个命名实体的结构解析结果中，则确定所述多个已校对的标注样本的结构解析结果中存在与所述第一结构解析结果匹配的目标结构解析结果。
  4. 根据权利要求2或3所述的方法，其特征在于，在获得所述目标样本的第一结构解析结果之后，还包括：
    生成所述第一结构解析结果的确认界面，所述确认界面用于向用户显示所述第一结构解析结果；
    接收所述用户输入的确认指令，所述确认指令用于对所述第一结构解析结果进行修改或确认。
  5. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    基于所述修改后的第一结构解析结果，更新所述词内部结构分析模型。
  6. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    在接收所述用户输入的确认指令之后，将确认后或修改后的所述第一结构解析结果添加至所述已校对的标注样本的结构解析结果中。
  7. 根据权利要求2或3所述的方法，其特征在于，所述输出提示消息之前，还包括：
    在所述目标样本所属的语句中，获取目标短语，其中，所述目标短语由所述目标样本与所述目标样本的相邻位置的词语组成；
    对所述目标短语进行结构解析，获得所述目标短语的第二结构解析结果；
    确定所述多个已校对的标注样本的结构解析结果中存在与所述第二结构解析结果匹配的结构解析结果。
  8. 根据权利要求1至7任一项所述的方法，其特征在于，所述提示消息还用于提示所述目标样本对应的正确标注结果。
  9. 根据权利要求1至8任一项所述的方法，其特征在于，所述获取目标样本，包括：
    获取用户标注的所述目标样本；或者，
    获取预标注模型标注的所述目标样本。
  10. 一种样本标注的校对装置，其特征在于，所述装置包括：
    交互模块，用于获取目标样本，其中，所述目标样本为待校对的标注样本；
    解析模块，用于对所述目标样本进行结构解析，获得所述目标样本的第一结构解析结果；
    匹配模块，用于将所述第一结构解析结果与多个已校对的标注样本的结构解析结果进行匹配；
    所述交互模块，还用于若所述多个已校对的标注样本的结构解析结果中不存在与所述第一结构解析结果匹配的目标结构解析结果，则输出提示消息，其中，所述提示消息用于提示用户检查所述目标样本的标注结果。
  11. 根据权利要求10所述的装置，其特征在于，所述目标样本为目标命名实体样本，所述第一结构解析结果为词内部结构；
    所述解析模块，用于：
    使用词内部结构分析模型，对所述目标样本进行词内部结构解析，获得所述目标样本的词内部结构。
  12. 根据权利要求11所述的装置，其特征在于，所述匹配模块，用于：
    从所述多个已校对的标注样本中，确定与所述目标命名实体样本的类别相同的一个或多个命名实体样本；
    若所述第一结构解析结果不存在于所述一个或多个命名实体的结构解析结果中，则确定所述多个已校对的标注样本的结构解析结果中不存在与所述第一结构解析结果匹配的目标结构解析结果；
    若所述第一结构解析结果存在于所述一个或多个命名实体的结构解析结果中，则确定所述多个已校对的标注样本的结构解析结果中存在与所述第一结构解析结果匹配的目标结构解析结果。
  13. 根据权利要求11或12所述的装置，其特征在于，所述交互模块，还用于：
    在获得所述目标样本的第一结构解析结果之后，生成所述第一结构解析结果的确认界面，所述确认界面用于向用户显示所述第一结构解析结果；
    接收所述用户输入的确认指令，所述确认指令用于对所述第一结构解析结果进行修改或确认。
  14. 根据权利要求13所述的装置，其特征在于，所述解析模块，还用于：
    基于所述修改后的第一结构解析结果，更新所述词内部结构分析模型。
  15. 根据权利要求13所述的装置，其特征在于，所述交互模块，还用于：
    在接收所述用户输入的确认指令之后，将确认后或修改后的所述第一结构解析结果添加至所述已校对的标注样本的结构解析结果中。
  16. 根据权利要求11或12所述的装置，其特征在于，所述匹配模块，还用于：
    在输出提示消息之前，在所述目标样本所属的语句中，获取目标短语，其中，所述目标短语由所述目标样本与所述目标样本的相邻位置的词语组成；
    对所述目标短语进行结构解析，获得所述目标短语的第二结构解析结果；
    确定所述多个已校对的标注样本的结构解析结果中存在与所述第二结构解析结果匹配的结构解析结果。
  17. 根据权利要求10至16任一项所述的装置，其特征在于，所述提示消息还用于提示所述目标样本对应的正确标注结果。
  18. 根据权利要求10至17任一项所述的装置，其特征在于，所述交互模块，用于：
    获取用户标注的所述目标样本；或者，
    获取预标注模型标注的所述目标样本。
  19. 一种计算设备集群，其特征在于，包括至少一个计算设备，每个计算设备包括处理器和存储器；
    所述至少一个计算设备的处理器用于执行所述至少一个计算设备的存储器中存储的指令，以使得所述计算设备集群执行如权利要求1至9任一项所述的方法。
  20. 一种计算机可读存储介质，其特征在于，包括计算机程序指令，当所述计算机程序指令由计算设备集群执行时，所述计算设备集群执行如权利要求1至9任一项所述的方法。
  21. 一种包含指令的计算机程序产品，其特征在于，当所述指令被计算设备集群运行时，使得所述计算设备集群执行如权利要求的1至9任一项所述的方法。
PCT/CN2022/137635 2022-05-23 2022-12-08 样本标注的校对方法、装置、计算设备集群和存储介质 WO2023226367A1 (zh)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
CN202210562530.X 2022-05-23
CN202210562530 2022-05-23
CN202210986086.4A CN117172250A (zh) 2022-05-23 2022-08-16 样本标注的校对方法、装置、计算设备集群和存储介质
CN202210986086.4 2022-08-16

Publications (1)

Publication Number Publication Date
WO2023226367A1 true WO2023226367A1 (zh) 2023-11-30

Family

ID=88918314

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/137635 WO2023226367A1 (zh) 2022-05-23 2022-12-08 样本标注的校对方法、装置、计算设备集群和存储介质

Country Status (1)

Country Link
WO (1) WO2023226367A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135225A (zh) * 2018-02-09 2019-08-16 北京世纪好未来教育科技有限公司 样本标注方法及计算机存储介质
CN110147551A (zh) * 2019-05-14 2019-08-20 腾讯科技（深圳）有限公司 多类别实体识别模型训练、实体识别方法、服务器及终端
CN110348017A (zh) * 2019-07-15 2019-10-18 苏州大学 一种文本实体检测方法、系统及相关组件
WO2022048210A1 (zh) * 2020-09-03 2022-03-10 平安科技（深圳）有限公司 命名实体识别方法、装置、电子设备及可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110135225A (zh) * 2018-02-09 2019-08-16 北京世纪好未来教育科技有限公司 样本标注方法及计算机存储介质
CN110147551A (zh) * 2019-05-14 2019-08-20 腾讯科技（深圳）有限公司 多类别实体识别模型训练、实体识别方法、服务器及终端
CN110348017A (zh) * 2019-07-15 2019-10-18 苏州大学 一种文本实体检测方法、系统及相关组件
WO2022048210A1 (zh) * 2020-09-03 2022-03-10 平安科技（深圳）有限公司 命名实体识别方法、装置、电子设备及可读存储介质

Similar Documents

Publication Publication Date Title
US10402498B2 (en) Method and system for automatic management of reputation of translators
CN107908635B (zh) 建立文本分类模型以及文本分类的方法、装置
US10235192B2 (en) Self-learning robotic process automation
WO2020119075A1 (zh) 通用文本信息提取方法、装置、计算机设备和存储介质
US8886515B2 (en) Systems and methods for enhancing machine translation post edit review processes
CN108304375A (zh) 一种信息识别方法及其设备、存储介质、终端
CN107908641B (zh) 一种获取图片标注数据的方法和系统
US11163806B2 (en) Obtaining candidates for a relationship type and its label
US11645450B2 (en) Annotation editor with graph
US9626432B2 (en) Defect record classification
US11714636B2 (en) Methods and arrangements to process comments
WO2020259141A1 (zh) 一种文件处理方法、装置及计算机设备
CN108932218A (zh) 一种实例扩展方法、装置、设备和介质
CN109815147A (zh) 测试案例生成方法、装置、服务器和介质
US10049108B2 (en) Identification and translation of idioms
WO2021129074A1 (zh) 用于处理程序代码中的变量的引用的方法和系统
CN110688111A (zh) 业务流程的配置方法、装置、服务器和存储介质
US20180165277A1 (en) Dynamic Translation of Idioms
WO2023226367A1 (zh) 样本标注的校对方法、装置、计算设备集群和存储介质
US11836197B2 (en) Search processing method and apparatus based on clipboard data
US20210271637A1 (en) Creating descriptors for business analytics applications
WO2023173631A1 (zh) 编程方法和装置、设备、存储介质及计算机程序产品
US20230419950A1 (en) Artificial intelligence factsheet generation for speech recognition
US11200215B2 (en) Data quality evaluation
US11182155B2 (en) Defect description generation for a software product

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22943554

Country of ref document: EP

Kind code of ref document: A1