WO2020237872A1 - 语义分析模型准确度的校验方法、装置、存储介质及设备 - Google Patents

语义分析模型准确度的校验方法、装置、存储介质及设备 Download PDF

Info

Publication number
WO2020237872A1
WO2020237872A1 PCT/CN2019/103024 CN2019103024W WO2020237872A1 WO 2020237872 A1 WO2020237872 A1 WO 2020237872A1 CN 2019103024 W CN2019103024 W CN 2019103024W WO 2020237872 A1 WO2020237872 A1 WO 2020237872A1
Authority
WO
WIPO (PCT)
Prior art keywords
keywords
question
data set
question sentence
expected
Prior art date
Application number
PCT/CN2019/103024
Other languages
English (en)
French (fr)
Inventor
王丽杰
Original Assignee
平安科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 平安科技（深圳）有限公司 filed Critical 平安科技（深圳）有限公司
Publication of WO2020237872A1 publication Critical patent/WO2020237872A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation or dialogue systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G06F16/355—Class or cluster creation or modification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/205—Parsing
    • G06F40/216—Parsing using statistical methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/253—Grammatical analysis; Style critique
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis

Definitions

  • This application relates to the technical field of verification. Specifically, this application relates to a method, device, storage medium, and equipment for verifying the accuracy of a semantic analysis model.
  • question and answer sample data are often only hundreds to thousands of magnitudes, and the collection of question data and comparison of results are carried out manually, the coverage of question data is narrow and the verification efficiency is low.
  • This application provides a method and device for verifying the accuracy of a semantic analysis model, a computer-readable non-volatile storage medium, and computer equipment to expand the problem set and solve the problem of low efficiency of semantic analysis model verification.
  • this application provides a method for verifying the accuracy of a semantic analysis model, which includes: obtaining project information, extracting keywords from the project information, and classifying the keywords as test keys according to their semantic attributes Words and expected keywords to obtain a test data set and an expected data set; extract test keywords and expected keywords from the test data set and the expected data set respectively to synthesize question sentences to obtain a question sentence set;
  • the question sentence parses the item information, obtains the reference answer corresponding to each question sentence, and associates the question sentence with the reference answer; input each question sentence in the question sentence set into the semantic analysis model for recognition, and obtain the output identification
  • the answer is to compare the reference answer associated with the question sentence with the recognition answer to obtain the recognition accuracy of the semantic analysis model.
  • an embodiment of the present application also provides a device for verifying the accuracy of a semantic analysis model, including: a keyword extracting module for obtaining item information, extracting keywords from the item information, according to the meaning of the word
  • the attributes divide the keywords into test keywords and expected keywords to obtain a test data set and an expected data set; generating a question sentence module for extracting test keywords and expected keys from the test data set and expected data set respectively Words synthesize question sentences to obtain a set of question sentences; an obtain reference answer module for analyzing the item information according to the question sentences in the question sentence set, obtaining the reference answer corresponding to each question sentence, and combining the question sentence with the reference answer Associate;
  • the verification module is used to input each question sentence in the question sentence set into the semantic analysis model for recognition, obtain the output recognition answer, and compare the reference answer associated with the question sentence with the recognition answer to obtain the semantics Analyze the recognition accuracy of the model.
  • the embodiments of the present application also provide a computer-readable non-volatile storage medium, the computer-readable non-volatile storage medium is used to store computer instructions, when the computer instructions are on the computer At runtime, the computer can execute a method for verifying the accuracy of the semantic analysis model.
  • the method for verifying the accuracy of the semantic analysis model includes the following steps: obtaining item information, extracting keywords from the item information, and according to the meaning of the word
  • the attributes divide the keywords into test keywords and expected keywords to obtain a test data set and an expected data set; extract test keywords and expected keywords from the test data set and expected data set respectively to synthesize question sentences to obtain questions Sentence set; parse the item information according to the question sentences in the question sentence set, obtain the reference answer corresponding to each question sentence, and associate the question sentence with the reference answer; input each question sentence in the question sentence set
  • the semantic analysis model performs recognition, obtains the output recognition answer, and compares the reference answer associated with the question sentence with the recognition answer to obtain the recognition accuracy of the semantic analysis model.
  • the embodiments of the present application also provide a computer device, the computer device includes: one or more processors; a storage device for storing one or more programs, when the one or more The program is executed by the one or more processors, so that the one or more processors implement a method for verifying the accuracy of a semantic analysis model, and the method for verifying the accuracy of a semantic analysis model includes the following steps: obtaining Project information, extract keywords from the project information, divide the keywords into test keywords and expected keywords according to the meaning attributes of the words, to obtain a test data set and an expected data set; respectively from the test data set and the expected data Centrally extract test keywords and expected keywords to synthesize question sentences to obtain a question sentence set; analyze the item information according to the question sentences in the question sentence set, obtain the reference answer corresponding to each question sentence, and compare the question sentence with the reference Answers are associated; each question sentence in the question sentence set is input into the semantic analysis model for recognition, and the output recognition answer is obtained, and the reference answer associated with the question sentence
  • the embodiments of the application provide a method, device, non-volatile storage medium and equipment for verifying the accuracy of a semantic analysis model.
  • a large number of keywords are extracted from project information and a combination of keywords is used to generate a large number of questions.
  • Sentences, these question sentences can cover the entire project information, with a wide coverage and rich quantity, which improves the verification accuracy of the semantic recognition model and also improves the verification efficiency.
  • FIG. 1 is an implementation environment diagram of a method for verifying the accuracy of a semantic analysis model provided by an embodiment of this application;
  • FIG. 2 is a schematic flowchart of a method for verifying the accuracy of a semantic analysis model provided by an embodiment of the application
  • FIG. 3 is a schematic diagram of a process of extracting test keywords and expected keywords to synthesize question sentences from the test data set and the expected data set, respectively, according to an embodiment of the application, to obtain a question sentence set;
  • Figure 4 is a schematic flow diagram of extracting test keywords and expected keywords to synthesize question sentences from the test data set and the expected data set respectively according to an embodiment of the application;
  • FIG. 5 is a schematic flowchart of a method for verifying the accuracy of a semantic analysis model provided by another embodiment of the application.
  • FIG. 6 is a schematic structural diagram of an apparatus for verifying the accuracy of a semantic analysis model provided by an embodiment of this application;
  • FIG. 7 is a schematic structural diagram of a computer device provided by an embodiment of this application.
  • first, second, etc. used in this application can be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from another element.
  • first live video image may be referred to as the second live video image
  • second live video image may be referred to as the first live video image.
  • FIG. 1 is an implementation environment diagram of a method for verifying the accuracy of a semantic analysis model provided in an embodiment.
  • the implementation environment includes a user terminal and a server side.
  • the server obtains project information, extracts keywords from the project information, and divides the extracted keywords into test keywords and expected keywords according to the semantic attributes, and obtains the test
  • the data set and the expected data set are respectively extracted from the test data set and the expected data set to synthesize question sentences with test keywords and expected keywords to obtain a question sentence set; parse the item information according to the question sentences in the question sentence set, Obtain the reference answer corresponding to each question sentence, and associate the question sentence with the reference answer; input each question sentence in the question sentence set into the semantic analysis model for recognition, obtain the output recognition answer, and compare the question sentence
  • the associated reference answer is compared with the recognition answer to obtain the accuracy of the semantic analysis model.
  • the user terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.
  • the server side can be implemented by a computer device with processing functions, but is not limited to this.
  • the server and the user terminal can be connected to the network through Bluetooth, USB (Universal Serial Bus) or other communication connection methods, and this application is not limited here.
  • FIG. 2 is a schematic flowchart of a method for verifying the accuracy of a semantic analysis model provided by an embodiment of the application.
  • the method for verifying the accuracy of a semantic analysis model can be applied to the above-mentioned server side and includes the following steps:
  • Step S210 acquiring item information, extracting keywords from the item information, dividing the keywords into test keywords and expected keywords according to the meaning attributes of the words, to obtain a test data set and an expected data set;
  • Step S220 Extract test keywords and expected keywords from the test data set and expected data set to synthesize question sentences to obtain a question sentence set;
  • Step S230 parse the item information according to the question sentences in the question sentence set, obtain the reference answer corresponding to each question sentence, and associate the question sentence with the reference answer;
  • Step S240 Input each question sentence in the question sentence set into the semantic analysis model for recognition, obtain the output recognition answer, and compare the reference answer associated with the question sentence with the recognition answer to obtain the recognition accuracy of the semantic analysis model .
  • Natural Language Processing is a field in which computer science, artificial intelligence, and linguistics focus on the interaction between computers and human natural language, and studies various theories of effective communication between humans and computers using natural language And method.
  • the semantic analysis model is applied to the field of natural language processing, which can realize the recognition of human natural language by computers.
  • the implementation basis of this application plan is that the project information has been obtained, and subsequent processing is carried out based on the project information.
  • the project information can be a certain scientific research project or R&D project.
  • the project information can contain information in the form of text, charts, etc. Extract keywords from the project information, and classify the extracted keywords according to their semantic attributes. First, divide the keywords into test keywords and expected keywords. Test keywords, such as: A company, B industry, men and women , Price, entry, etc., expect keywords, such as: quantity, proportion, date, etc.
  • Extract test keywords and expected keywords from the test data set and the expected data set respectively to form question sentences such as: the ratio of male to female, and the extraction methods are diverse. They can be selected randomly or according to preset rules, and extracted from project information
  • a large number of keywords are combined to generate multiple question sentences to form a set of question sentences.
  • the project information is parsed according to the extracted question sentences to obtain the reference answer of each question sentence, and the reference answers corresponding to the question sentences are associated, in turn Extract the question sentences in the question sentence set as the input of the semantic analysis model, and obtain the recognition answer output by the model. Compare the reference answer and the recognition answer of the same question. If the recognition answer is consistent with the reference answer, the recognition is correct. If the recognition answer is the reference answer If they are inconsistent, the recognition is wrong, and the recognition accuracy of the semantic analysis model is obtained according to the proportion of correctly recognized problems in all problems.
  • the verification scheme for the accuracy of the semantic analysis model extracts keywords in the project information, combines the extracted keywords to generate a large number of question sentences, and uses a large number of question sentences as test questions of the semantic analysis model. Check the recognition accuracy of the semantic analysis model.
  • This solution extracts a large number of keywords based on project information. The combination of different keywords can generate a large number of question sentences.
  • the coverage of question sentences obtained based on the keywords of the entire project information is wide, and the number of questions in the question sentence set is rich, which is beneficial to improve the model
  • the verification accuracy also improves the verification efficiency.
  • test keywords and expected keywords are extracted from the test data set and the expected data set to synthesize question sentences to obtain the question sentence set, which can be obtained in the following manner.
  • the flowchart is shown in Figure 3 and includes the following sub-steps :
  • S330 Collect the generated question sentences to form a question sentence set.
  • combining the extracted test keywords and expected keywords to generate question sentences can be performed in at least two ways.
  • One is to randomly extract at least one test keyword from the test data set, and randomly select from the expected data. Focus on extracting an expected keyword, randomly combine the extracted test keywords and expected keywords to generate question sentences.
  • test keyword can be extracted from the test data set, which can be two or more test keywords, such as two test keywords: A company, male and female, or three test keywords: A company, For men and women, in the past three months, even if the expected keywords extracted are the same, they still correspond to at least two different question sentences.
  • step S220 the test keywords and the desired keyword synthesis question sentences are extracted from the test data set and the expected data set respectively, which can be obtained in the following manner.
  • the flow diagram is shown in FIG. 4, and includes the following sub-steps:
  • the test data set is divided into multiple data types, which can be divided into: source information, such as file name, company name or industry name, time information, event information, etc., such as the project information has the following information: A company in March 2018 The number of recruits to July is 6, extract the keywords: Company A, March to July 2018, recruits, number of people.
  • the test data set is divided into source information set, time information set, and event information Set, where the source information keyword is: Company A, the time information keyword is: March to July 2018, the event information keyword is: employment, and the expected keyword is: number of people. Extract keywords and expected keywords from multiple test data classification sets in turn to generate question sentences: How many people did company A hire from March to July 2018?
  • test data set can be divided into three test data sets or more, which can include the third test data set, the fourth test data set, etc.
  • test data set is divided into multiple test subsets, the test subsets are randomly combined, and the randomly combined test keywords are randomly combined with the desired keywords.
  • a large number of keywords can be extracted from project-related information.
  • the test data set includes multiple test subsets, and different test subsets are randomly combined. This method can increase the number of test data combinations, test data sets and expectations Random combination of data sets further increases the number of question sentences generated. The number of question sentences generated in this way can easily reach the order of one million, meeting the needs of model training or model verification.
  • the solution provided by the present embodiment is based on the large number of extracted keywords using the random combination of data sets, so that the number of question sentences in the constructed question sentence set has an explosive growth, the efficiency of obtaining question sentences is high, and it is easy to obtain a large number of question sentences.
  • the test samples meet the verification requirements of the recognition accuracy of the semantic analysis model.
  • the method further includes:
  • S221 Invoke grammar rules to perform sentence refinement processing on the question sentence, so that the question sentence conforms to the grammar rule.
  • the question sentence composed of extracting test keywords and expected keywords is: the number of employees of company A from March to July 2018?
  • Question sentences such as: What was the number of employees in Company A from March to July 2018?
  • the question sentence after embellishment is more in line with human grammatical rules, avoiding the ambiguity of the question sentence, and thus making it impossible to use the question sentence for the accuracy of the semantic analysis model.
  • the method further includes: S222, performing semantic analysis on the question sentences in the question sentence set, and removing meaningless question sentences therein.
  • the question statement is generated by randomly combining the test data set and the expected data set, a meaningless combination of question data may appear.
  • the test keywords in the test data set include: Company A, March to July 2018 , Entry, men and women, expected keywords: number, proportion, arbitrary extraction of the question statement composed of the above test data set and expected data set: the proportion from March to July 2018, the question statement becomes meaningless due to the lack of necessary attributives. In order to improve the quality of question sentences, this part of meaningless question sentences is eliminated.
  • the question sentence is first processed to make the question sentence conform to the grammatical rules, and then the question sentence after the refinement is eliminated to avoid eliminating Question sentences that do not conform to the grammatical rules can improve the screening pass rate of question sentences.
  • step S230 the item information is parsed according to the question sentences in the question sentence set, the reference answer corresponding to each question sentence is obtained, and the question sentence is associated with the reference answer.
  • the item information can be expressed in the form of words, charts, etc., Analyze the information represented in the form of text, graphs, etc., extract the keywords and the preset reference answers containing at least one keyword tag, such as: the following text record information in the project information: the number of employees of company A in 2017 is 6 , Extract the keywords: company A, 2017, entry, number of people, the default reference answer is: 6, the keyword tags of the preset reference answer are: company A, 2017, entry, number of people.
  • the keywords in the question sentences are: company A, 2017, employment, number of people
  • compare the keywords of the question sentences with the keyword tags of the preset reference answers if the question is The keyword of the sentence exactly matches the keyword label of the preset reference answer, then the preset reference answer is the reference answer of the question sentence, and the question sentence and the corresponding reference answer are associated and stored for subsequent retrieval of the question sentence The reference answer of, compare it with the recognized answer of the question sentence.
  • the question sentence in this part can also be eliminated. This solution can ensure that the question sentences in the question sentence set are provided with reference answers, which improves the quality of the question sentences and is beneficial to obtain Accurate recognition accuracy of semantic analysis model.
  • the step of obtaining the reference answer corresponding to each question sentence can be performed in the following manner, S231, classifying the question sentence as an unsolved question
  • a unified reference answer is set for the question sentences in the unsolved question sentence set. For example, if the unified reference answer is set in the form of "this question is too important" or "this is a good question", etc., set a unified answer for the question sentences in the set of unsolved question sentences to ensure that each question sentence corresponds to a reference answer. Evaluation results that affect the accuracy of the semantic analysis model.
  • the method further includes: performing a gloss processing on the question sentence, performing semantic analysis on the glossed question sentence, and retaining Question sentences that have practical meaning after embellishment.
  • the question sentences in the question sentence set are all question sentences that conform to the grammatical rules and are meaningful.
  • reference answers are obtained, and a unified reference answer is set for the unsolved question sentences in order to follow the question.
  • the sentence is tested for the accuracy of the semantic analysis model.
  • the question sentence corresponding to the reference answer in the project information can be used as the test data of the model recognition accuracy, and the question sentence without the reference answer in the project information can also be used as the test question of the semantic analysis model.
  • the model's recognition answer to the question sentences in the set of unsolved question sentences means that there is no correct answer, and the recognition result of the semantic analysis model is considered to be correct.
  • this embodiment provides a verification scheme for the accuracy of a semantic analysis model.
  • the flow diagram is shown in Fig. 5.
  • step S221 is performed to call the grammar rule.
  • step S230 Meaningful question sentences to obtain meaningful question sentences that conform to the grammatical rules, and then proceed to step S230 to parse the item information according to the question sentences in the question sentence set, and determine whether there is a reference answer corresponding to the question sentence stored in the item information If the item information is traversed and the reference answer corresponding to the question sentence is not obtained, then step S231 is performed to classify the question sentence into the unsolved question sentence set, and a unified reference answer is set for the question sentences in the unsolved question sentence set.
  • step S231 If the reference answer corresponding to the question sentence is already stored in the item information or a unified reference answer is set after step S231, the question sentence is associated with the reference answer, and then step S240 is performed after the association, and each question sentence in the question sentence set is input semantics Analyze model recognition, obtain the output recognition answer, and compare the reference answer associated with the question sentence with the recognition answer to obtain the recognition accuracy of the semantic analysis model.
  • the question sentences are refined and filtered so that the question sentences are all question sentences that conform to grammatical rules and are meaningful.
  • it is a question sentence that is not stored in the project information Set up a unified reference answer to ensure that each question sentence can be used as a test question for the semantic analysis model, and to improve the quality of the question sentence.
  • the number of question sentences is guaranteed, which is conducive to obtaining an accurate model. Identify the accuracy and obtain the efficiency of model accuracy verification.
  • each question sentence in the question sentence set is sequentially extracted, and the question sentence is input into the semantic analysis model to obtain the output identification answer; and the question sentence is retrieved
  • the associated reference answer is compared with the identified answer.
  • the step of comparing the reference answer associated with the question sentence with the recognition answer in step S240 to obtain the recognition accuracy of the semantic analysis model includes:
  • the reference answer is consistent with the recognition answer, it is marked as correct; if the reference answer is inconsistent with the recognition answer, it is marked as a recognition error;
  • the number of correctly identified question sentences and the number of question sentences in the question sentence set are counted, and the proportion of the number of correctly identified question sentences in the question sentence set is calculated to obtain the recognition accuracy of the semantic analysis model.
  • the solution for obtaining the recognition accuracy of the semantic analysis model is to judge the consistency of the reference answer and the recognition answer, where the consistency judgment is the semantic consistency, such as: the recognition answer is "6 people", if the answer is referred to "Six persons" means that the reference answer is consistent with the recognized answer. If they are consistent, the question sentence is marked as correctly recognized, and the proportion of the correctly recognized question sentence in the question sentence set is counted. The number of question sentences in the set is equal Input the number of question sentences of the semantic recognition model, this solution can obtain the recognition accuracy of the semantic analysis model simply and clearly.
  • the embodiment of the present application also provides a device for verifying the accuracy of a semantic analysis model.
  • the schematic diagram of the structure is shown in FIG. 6, including: a keyword extraction module 610, a question sentence generation module 620, a reference answer obtaining module 630, and verification Module 640 is as follows:
  • the keyword extraction module 610 is used to obtain project information, extract keywords from the project information, and divide the keywords into test keywords and expected keywords according to the meaning attributes of the words to obtain a test data set and an expected data set;
  • a generating question sentence module 620 configured to extract test keywords and expected keywords from the test data set and the expected data set to synthesize question sentences to obtain a question sentence set;
  • the reference answer obtaining module 630 is configured to parse the item information according to the question sentences in the question sentence set, obtain the reference answer corresponding to each question sentence, and associate the question sentence with the reference answer;
  • the verification module 640 is used to input each question sentence in the question sentence set into the semantic analysis model for recognition, obtain the output recognition answer, and compare the reference answer associated with the question sentence with the recognition answer to obtain the semantic analysis model Recognition accuracy.
  • an embodiment of the present application also provides a computer-readable non-volatile storage medium, on which computer instructions are stored, and when the computer instructions are executed by a processor, the accuracy of the semantic analysis model described in any one of the above is achieved. Steps of the verification method.
  • the non-volatile storage medium includes, but is not limited to, any type of disk (including floppy disk, hard disk, optical disk, CD-ROM, and magneto-optical disk), ROM (Read-Only Memory), RAM (Random AcceSS Memory), EPROM (EraSable Programmable Read-Only Memory), EEPROM (Electrically EraSable Programmable Read-Only Memory), flash memory, magnetic card Or light card. That is, a non-volatile storage medium includes any medium that stores or transmits information in a readable form by a device (for example, a computer). It can be a read-only memory, magnetic disk or optical disk, etc.
  • an embodiment of the present application also provides a computer device, and the computer device includes:
  • One or more processors are One or more processors;
  • Storage device for storing one or more programs
  • the one or more processors implement the steps of the method for verifying the accuracy of the semantic analysis model described in any one of the above.
  • Fig. 7 is a block diagram showing a computer device 700 according to an exemplary embodiment.
  • the computer device 700 may be provided as a server.
  • the computer device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732, for storing instructions executable by the processing component 722, such as an application program.
  • the application program stored in the memory 732 may include one or more modules each corresponding to a set of instructions.
  • the processing component 722 is configured to execute instructions to execute the steps of the method for verifying the accuracy of the semantic analysis model described above.
  • the computer device 700 may also include a power supply component 726 configured to perform power management of the computer device 700, a wired or wireless network interface 750 configured to connect the computer device 700 to a network, and an input output (I/O) interface 758 .
  • the computer device 700 can operate based on an operating system stored in the memory 732, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or the like. It should be understood that, although the various steps in the flowchart of the drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless explicitly stated in this article, the execution of these steps is not strictly limited in order, and they can be executed in other orders.
  • steps in the flowchart of the drawings may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the order of execution is also It is not necessarily performed sequentially, but may be performed alternately or alternately with other steps or at least a part of sub-steps or stages of other steps.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Human Computer Interaction (AREA)
  • Machine Translation (AREA)

Abstract

本申请涉及校验技术领域，尤其涉及一种语义分析模型准确度的校验方法、装置、非易失性存储介质及设备。其中，语义分析模型准确度的校验方法，包括：获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获得语义分析模型的识别准确度。本申请提供的方案通过扩充问题集的问题，以解决语义分析模型验证效率低下的问题。

Description

语义分析模型准确度的校验方法、装置、存储介质及设备
本申请要求于2019年5月24日提交中国专利局、申请号为2019104414865，发明名称为“语义分析模型准确度的校验方法、装置、存储介质及设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及校验技术领域，具体而言，本申请涉及一种语义分析模型准确度的校验方法、装置、存储介质及设备。
背景技术
随着人工智能的发展，计算机可以帮助人类做很多工作，计算机能够帮助甚至替代人类“工作”的秘诀在于：模型的建立，训练出的模型能够进行识别、分类等工作，但模型的建立需要大量的训练样本，发明人意识到模型的识别精度及准确度受训练样本数量的直接影响，数据量不足会显著影响模型效果，而且模型建立后，需要用大量的数据对生成的模型进行准确率验证。
尤其是在问答系统研究领域，问答样本数据往往只有数百条到数千条量级，且问题数据的采集及结果的对比均采用手动梳理的方式进行，问题数据的覆盖面窄，验证效率低下。
发明内容
本申请提供了一种语义分析模型准确度的校验方法、装置、计算机可读非易失性存储介质及计算机设备，以扩充问题集，解决语义分析模型验证效率低下的问题。
为解决上述技术问题，本申请提供了一种语义分析模型准确度的校验方法，包括：获取项目信息，从所述项目信息中提取关键词，按照词义属 性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
为解决上述技术问题，本申请实施例还提供了一种语义分析模型准确度的校验装置，包括：提取关键词模块，用于获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；生成问题语句模块，用于分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；获取参考答案模块，用于根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；校验模块，用于将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
为解决上述技术问题，本申请实施例还提供了一种计算机可读非易失性存储介质，所述计算机可读非易失性存储介质用于存储计算机指令，当所述计算机指令在计算机上运行时，使得计算机可以执行一种语义分析模型准确度的校验方法，所述语义分析模型准确度的校验方法包括以下步骤：获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答 案进行对比，获得语义分析模型的识别准确度。
为解决上述技术问题，本申请实施例还提供了一种计算机设备，所述计算机设备包括：一个或多个处理器；存储装置，用于存储一个或多个程序，当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现一种语义分析模型准确度的校验方法，所述语义分析模型准确度的校验方法包括以下步骤：获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
本申请实施例提供了一种语义分析模型准确度的校验方法、装置、非易失性存储介质及设备，通过从项目信息中提取的大量关键词，利用关键词的组合生成大批量的问题语句，这些问题语句能够覆盖整个项目信息，覆盖面广且数量丰富，提高语义识别模型校验准确性的同时也提高了校验效率。
附图说明
图1为本申请一个实施例提供的语义分析模型准确度的校验方法的实施环境图；
图2为本申请一个实施例提供的语义分析模型准确度的校验方法的流程示意图；
图3为本申请一个实施例提供的分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集的流程示意图；
图4为本申请一个实施例提供的分别从所述测试数据集和期望数据 集中抽取测试关键词和期望关键词合成问题语句的流程示意图；
图5为本申请另一个实施例提供的语义分析模型准确度的校验方法的流程示意图；
图6为本申请一种实施例提供的语义分析模型准确度的校验装置的结构示意图；
图7为本申请一种实施例提供的计算机设备的结构示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，仅用于解释本申请，而不能解释为对本申请的限制。
本领域技术人员可以理解，除非特意声明，这里使用的单数形式“一”、“一个”、“所述”和“该”也可包括复数形式。应该进一步理解的是，本申请的说明书中使用的措辞“包括”是指存在所述特征、整数、步骤、操作、元件和/或组件，但是并不排除存在或添加一个或多个其他特征、整数、步骤、操作、元件、组件和/或它们的组。
可以理解，本申请所使用的术语“第一”、“第二”等可在本文中用于描述各种元件，但这些元件不受这些术语限制。这些术语仅用于将第一个元件与另一个元件区分。举例来说，在不脱离本申请的范围的情况下，可以将第一直播视频图像称为第二直播视频图像，且类似地，可将第二直播视频图像称为第一直播视频图像。
图1为一个实施例中提供的语义分析模型准确度的校验方法的实施环境图，在该实施环境中，包括用户终端、服务器端。
本实施例提供的语义分析模型准确度的校验方案，服务器端获取项目信息，从项目信息中提取关键词，按照词义属性将提取出的关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集，分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；根据所述问题语句集中的问题语句解析所述项目信息，获 取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的准确度。
需要说明的是，用户终端可为智能手机、平板电脑、笔记本电脑、台式计算机等，服务器端可以由具有处理功能的计算机设备来实现，但并不局限于此。服务器端与用户终端可以通过蓝牙、USB(Universal Serial Bus，通用串行总线)或者其他通讯连接方式进行网络连接，本申请在此不做限制。
在一个实施例中，图2为本申请实施例提供的语义分析模型准确度的校验方法的流程示意图，该语义分析模型准确度的校验方法可以应用于上述的服务器端，包括如下步骤：
步骤S210，获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；
步骤S220，分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；
步骤S230，根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；
步骤S240，将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
自然语言处理(Natural Language Processing,NLP)，是计算机科学、人工智能、语言学关注计算机和人类自然语言之间的相互作用的领域，研究人与计算机之间用自然语言进行有效通信的各种理论和方法。语义分析模型应用于自然语言处理领域，能够实现计算机识别人类自然语言。
本申请方案的实施基础是已经获得项目信息，基于该项目信息进行后续处理，该项目信息可以是某个科研项目或研发项目，该项目信息中可以 包含文字、图表等形式表示的信息，从这些项目信息中提取关键词，按照关键词的词义属性，将提取出的关键词进行分类，首先将关键词划分为测试关键词和期望关键词，测试关键词，如：A公司、B行业、男女、售价、入职等等，期望关键词，如：数量、比例、日期等。
分别从测试数据集和期望数据集抽取测试关键词和期望关键词，组成问题语句，如：男女比例，抽取的方式多样，可以随机抽取，也可以按照预设规则进行抽取，从项目信息中抽取大量关键词，对关键词进行组合生成多个问题语句，组成问题语句集合，根据提取出来的问题语句解析项目信息，获得各个问题语句的参考答案，并将问题语句对应的参考答案进行关联，依次提取问题语句集中的问题语句作为语义分析模型的输入，获得模型输出的识别答案，对比同一问题的参考答案与识别答案，若识别答案与参考答案一致，则为识别正确，若识别答案与参考答案不一致，则识别错误，根据正确识别的问题占全部问题的比例，获得语义分析模型的识别准确度。
本申请实施例提供的语义分析模型准确度的校验方案，提取项目信息中的关键词，对抽取出的关键词进行组合生成大量问题语句，将大量的问题语句作为语义分析模型的测试问题，进行语义分析模型的识别准确度校验。本方案基于项目信息提取大量关键词，不同关键词的组合能够生成大批量的问题语句，基于整个项目信息的关键词获得的问题语句的覆盖面广，问题语句集的问题数量丰富，有利于提高模型校验准确性的同时也提高了校验效率。
为了更清楚本申请提供的语义分析模型准确度的校验方案及其技术效果，接下来以多个实施例对其具体方案进行详细阐述。
步骤S220的分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集，可以通过如下方式获得，其流程示意图如图3所示，包括如下子步骤：
S310，从测试数据集中抽取至少一个测试关键词，从期望数据集中抽取一个期望关键词；
S320，将抽取出来的所述测试关键词与期望关键词进行组合，生成问 题语句；
S330，将生成的问题语句集合起来形成问题语句集。
其中，将抽取出来的测试关键词与期望关键词进行组合生成问题语句，可以通过至少两种方式进行，其一，随机从所述测试数据集中抽取至少一个测试关键词，随机从所述期望数据集中抽取一个期望关键词，将抽取出来的测试关键词和期望关键词进行随机组合，生成问题语句。
其中，测试数据集中可抽取至少一个测试关键词，可以为两个或两个以上的测试关键词，如抽取两个测试关键词：A公司、男女，或抽取三个测试关键词：A公司、男女、近三个月，即使抽取出的期望关键词为同一个，也至少对应两个不同的问题语句。
其二，通过预先设定抽取规则，如按照词义属性，将测试数据集划分为多个测试子集，依次从一个测试子集中抽取一个关键词，组成问题语句，具体如下：
步骤S220的分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，可以通过如下方式获得，其流程示意图如图4所示，包括如下子步骤：
S410，将测试数据集划分为第一测试数据集及第二测试数据集；
S420，依次抽取第一测试数据集中的第一测试关键词、第二测试数据集中的第二测试关键词及期望关键词生成问题语句。
将测试数据集划分为多个数据种类，可以划分为：来源信息，如文件名、公司名或行业名，时间信息，事件信息等等，如项目信息有如下信息：A公司于2018年3月至7月的入职人数为6个，提取其中的关键词：A公司、2018年3月至7月、入职、人数，按照数据种类将测试数据集划分为来源信息集、时间信息集、事件信息集，其中，来源信息关键词为：A公司，时间信息关键词为：2018年3月至7月，事件信息关键词为：入职，期望关键词为：人数。依次提取多个测试数据分类集中的关键词及期望关键词，生成问题语句：A公司2018年3月至7月入职人数？
值得说明的是，可以将测试数据集划分为三个测试数据集甚至更多，可以包含第三测试数据集、第四测试数据集等，划分的测试数据集越多， 表明测试数据划分的类别越详细，对应的参考答案越精准，解析项目信息获得参考答案的效率越高。
进一步地，将测试数据集划分为多个测试子集，将测试子集进行随机组合，并将随机组合的测试关键词与期望关键词进行随机组合。从项目相关信息中可以提取到大量关键词，测试数据集包括多个测试子集，不同测试子集之间进行随机组合，采用该种方式能够增大测试数据的组合数量，测试数据集与期望数据集之间进行随机组合，进一步增大了生成的问题语句的数量，按照这种方式生成的问题语句数量很容易达到百万数量级，满足模型训练或模型验证的需求。举例说明本方案：如测试子集有3个，每个测试子集中均设有1个测试关键词，其随机组合方式有
Figure PCTCN2019103024-appb-000001
期望数据集中设有3个期望关键词，则问题数据组合有
Figure PCTCN2019103024-appb-000002
按照该种组合方式，有限的关键词就能生成大量问题语句，且该种组合方式及问题语句的生成方式能够根据设定程序自动运行，降低问题生成过程中的人工成本及问题生成难度。
综上，本实施例提供的方案基于提取出的大量关键词采用数据集随机组合的方式使得构建的问题语句集合中问题语句数量呈爆炸式增长，获得问题语句的效率高，且容易获得大量的测试样本，满足语义分析模型的识别准确率的验证需求。
进一步地，步骤S220的分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤之后，还包括：
S221，调用语法规则对所述问题语句进行语句润化处理，以使所述问题语句符合语法规则。
结合上一示例，提取测试关键词和期望关键词组成的问题语句是：A公司2018年3月至7月入职人数？该语句的各关键词之间无连接词，不符合人的语法习惯，因此，需要将该问题语句进行语句润化，调用现有的语法规则，将连接词补入欠缺的位置，润化后的问题语句如：A公司于2018年3月至7月的入职人数是多少？
润化之后的问题语句更加符合人的语法规则，避免问题语句产生歧义，进而导致无法利用该问题语句进行语义分析模型的识别准确性。
进一步地，步骤S220的得到问题语句集的步骤之后，还包括：S222，对所述问题语句集中的问题语句进行语义分析，剔除其中无意义的问题语句。
若问题语句的生成方式是通过对测试数据集与期望数据集的随机组合，可能会出现无意义的问题数据组合，如测试数据集中的测试关键词包括：A公司、2018年3月至7月、入职、男女，期望关键词：人数、比例，任意抽取上述测试数据集与期望数据集组成的问题语句：2018年3月至7月比例，该问题语句由于缺乏必要的定语变得无意义，为了提高问题语句的质量，将该部分无意义的问题语句剔除。
在一种实施例中，在步骤S220的得到问题语句集的步骤之后，首先对问题语句进行润化处理，使问题语句符合语法规则，再对润化后的问题语句进行剔除处理，避免剔除由于不符合语法规则的问题语句，提高问题语句的筛选通过率。
步骤S230的根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联，项目信息可以采用文字、图表等形式表示，解析文字、图表等形式表征的信息，提取其中的关键词及包含至少一个关键词标签的预设参考答案，如：项目信息中有如下文字记录的信息：甲公司2017年的入职人数是6个，提取其中的关键词：甲公司、2017年、入职、人数，预设参考答案为：6个，该预设参考答案的关键词标签为：甲公司、2017年、入职、人数。根据提取出来的问题语句解析项目信息，如：问题语句中的关键词为：甲公司、2017年、入职、人数，根据问题语句的关键词与预设参考答案的关键词标签进行对比，若问题语句的关键词与预设参考答案的关键词标签完全匹配，则该预设参考答案为该问题语句的参考答案，并将问题语句与对应的参考答案进行关联存储，以便后续调取该问题语句的参考答案，将其与问题语句的识别答案进行对比。
若解析项目信息并未获得问题语句对应的参考答案，也可以剔除该部分的问题语句，该种方案能够保证问题语句集中的问题语句都设有参考答案，提高了问题语句的质量，有利于获得准确的语义分析模型的识别准确 性。
一种实施例中，若遍历项目信息并未获得问题语句的参考答案，所述获取各个问题语句对应的参考答案的步骤，可以通过如下方式进行，S231，将该问题语句划归到未解问题语句集合中，为所述未解问题语句集中的问题语句设置统一参考答案。如设置的统一参考答案形如“这个问题超纲了”或“这是个好问题”等，为未解问题语句集中的问题语句设置统一答案，能够保证每个问题语句均对应有参考答案，避免影响语义分析模型准确率的评估结果。
优选地，步骤S230的为所述未解问题语句集中的问题语句设置统一参考答案的步骤之前，还包括：对所述问题语句进行润化处理，对润化后的问题语句进行语义分析，保留润化处理后具有实际意义的问题语句。
经过该步骤之后，问题语句集中的问题语句均为符合语法规则且有意义的问题语句，对该部分问题语句进行参考答案的获取，对其中未解的问题语句设置统一参考答案，以便根据该问题语句进行语义分析模型准确度的测试。
值得说明的是，项目信息中对应有参考答案的问题语句可以作为模型识别准确性的测试数据，项目信息中未有对应参考答案的问题语句，也可以作为语义分析模型的测试问题，若语义分析模型对未解问题语句集中的问题语句的识别答案是表征无正确答案的意思，则认为该语义分析模型的识别结果是正确的。
基于上述润化处理及过滤处理，本实施例提供了一种语义分析模型准确度的校验方案，其流程示意图如图5所示，步骤S220得到问题语句集之后，进行步骤S221，调用语法规则对所述问题语句进行语句润化处理，以使所述问题语句符合语法规则，然后，再对润化后的问题语句进行S222，对所述问题语句集中的问题语句进行语义分析，剔除其中无意义的问题语句，获得符合语法规则且有意义的问题语句，接下来进行步骤S230的根据所述问题语句集中的问题语句解析所述项目信息，判断项目信息中是否存储有问题语句对应的参考答案，若遍历项目信息并未获得问题语句对应的参考答案，则进行步骤S231，将该问题语句划归到未解问题语句集合 中，为所述未解问题语句集中的问题语句设置统一参考答案。若项目信息中已存储有问题语句对应的参考答案或经过步骤S231设置有统一参考答案，将问题语句与参考答案进行关联，关联之后进行步骤S240，将所述问题语句集中的各个问题语句输入语义分析模型件识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
本实施例提供的方案中，问题语句经过润化和过滤，使得问题语句均为符合语法规则且有意义的问题语句，为各问题语句设置参考答案时，为项目信息中并未存储的问题语句设置统一参考答案，以保证每个问题语句均能作为语义分析模型的测试问题，提高问题语句的质量，与剔除该部分问题语句相比，保证问题语句的数量规模，进而有利于获得准确的模型识别准确度，以及获得模型准确度校验的效率。
一种实施例中，将问题语句与参考答案进行关联之后，依次提取问题语句集中的各个问题语句，将所述问题语句输入语义分析模型中，获取输出的识别答案；调取与所述问题语句相关联的参考答案，对比所述参考答案与识别答案。步骤S240的将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度的步骤，包括：
若所述参考答案与识别答案一致，则标记为识别正确；若所述参考答案与识别答案不一致，则标记为识别错误；
统计识别正确的问题语句数量与问题语句集中问题语句的数量，计算所述识别正确的问题语句在问题语句集中的数量占比，获得语义分析模型的识别准确度。
本申请实施例提供的获得语义分析模型的识别准确度的方案，通过对参考答案与识别答案进行一致性判断，其中一致性判断为语义一致，如：识别答案为“6人”，若参考答案为“六人”，则表示参考答案与识别答案一致，若一致，则标记该问题语句为识别正确，统计识别正确的问题语句在问题语句集中的占比，该处问题语句集中的数量是均输入语义识别模型的问题语句的数量，本方案能够简单明了地获取语义分析模型的识别准确度。
以上为本申请提供的语义分析模型准确度的校验方法实施例，针对于该方法，下面阐述与其对应的语义分析模型准确度的校验装置的实施例。
本申请实施例还提供了一种语义分析模型准确度的校验装置，其结构示意图如图6所示，包括：提取关键词模块610、生成问题语句模块620、获取参考答案模块630、校验模块640，具体如下：
提取关键词模块610，用于获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；
生成问题语句模块620，用于分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；
获取参考答案模块630，用于根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；
校验模块640，用于将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
关于上述实施例中的语义分析模型准确度的校验装置，其中各个模块执行操作的具体方式已经在有关该方法的实施例中进行了详细描述，此处将不做详细阐述说明。
进一步地，本申请实施例还提供一种计算机可读非易失性存储介质，其上存储有计算机指令，该计算机指令被处理器执行时实现上述任意一项所述的语义分析模型准确度的校验方法的步骤。其中，所述非易失性存储介质包括但不限于任何类型的盘(包括软盘、硬盘、光盘、CD-ROM、和磁光盘)、ROM(Read-Only Memory，只读存储器)、RAM(Random AcceSS Memory，随即存储器)、EPROM(EraSable Programmable Read-Only Memory，可擦写可编程只读存储器)、EEPROM(Electrically EraSable Programmable Read-Only Memory，电可擦可编程只读存储器)、闪存、磁性卡片或光线卡片。也就是，非易失性存储介质包括由设备(例如，计算机)以能够读的形式存储或传输信息的任何介质。可以是只读存储器，磁 盘或光盘等。
更进一步地，本申请实施例还提供一种计算机设备，所述计算机设备包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序，
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现上述任意一项所述的语义分析模型准确度的校验方法的步骤。
图7是根据一示例性实施例示出的一种用于计算机设备700的框图。例如，计算机设备700可以被提供为一服务器。参照图7，计算机设备700包括处理组件722，其进一步包括一个或多个处理器，以及由存储器732所代表的存储器资源，用于存储可由处理组件722的执行的指令，例如应用程序。存储器732中存储的应用程序可以包括一个或一个以上的每一个对应于一组指令的模块。此外，处理组件722被配置为执行指令，以执行上述语义分析模型准确度的校验方法的步骤。
计算机设备700还可以包括一个电源组件726被配置为执行计算机设备700的电源管理，一个有线或无线网络接口750被配置为将计算机设备700连接到网络，和一个输入输出(I/O)接口758。计算机设备700可以操作基于存储在存储器732的操作系统，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM或类似。应该理解的是，虽然附图的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，其可以以其他的顺序执行。而且，附图的流程图中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，其执行顺序也不必然是依次进行，而是可以与其他步骤或者其他步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
应该理解的是，在本申请各实施例中的各功能单元可集成在一个处理模块中，也可以各个单元单独物理存在，也可以两个或两个以上单元集成 于一个模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。
以上所述仅是本申请的部分实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本申请原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本申请的保护范围。

Claims (20)

  1. 一种语义分析模型准确度的校验方法，包括：
    获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；
    分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；
    根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；
    将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
  2. 根据权利要求1所述的语义分析模型准确度的校验方法，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤，包括：
    将测试数据集划分为第一测试数据集及第二测试数据集；
    依次抽取第一测试数据集中的第一测试关键词、第二测试数据集中的第二测试关键词及期望关键词生成问题语句。
  3. 根据权利要求1所述的语义分析模型准确度的校验方法，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤之后，还包括：
    调用语法规则对所述问题语句进行语句润化处理，以使所述问题语句符合语法规则。
  4. 根据权利要求1或3所述的语义分析模型准确度的校验方法，所述得到问题语句集的步骤之后，还包括：
    对所述问题语句集中的问题语句进行语义分析，剔除其中无意义的问题语句。
  5. 根据权利要求1所述的语义分析模型准确度的校验方法，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问 题语句，得到问题语句集的步骤，包括：
    从测试数据集中抽取至少一个测试关键词，从期望数据集中抽取一个期望关键词；
    将抽取出来的所述测试关键词与期望关键词进行组合，生成问题语句；
    将生成的问题语句集合起来形成问题语句集。
  6. 根据权利要求1所述的语义分析模型准确度的校验方法，若遍历项目信息并未获得问题语句的参考答案，所述获取各个问题语句对应的参考答案的步骤，包括：
    将问题语句划归为未解问题语句集合中，为所述未解问题语句集中的问题语句设置统一参考答案。
  7. 根据权利要求1所述的语义分析模型准确度的校验方法，所述将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度的步骤，包括：
    若所述参考答案与识别答案一致，则标记为识别正确；
    分别统计识别正确的问题语句数量与问题语句集中问题语句的数量，计算识别正确的问题语句在问题语句集中的数量占比，获得语义分析模型的识别准确度。
  8. 一种语义分析模型准确度的校验装置，包括：
    提取关键词模块，用于获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；
    生成问题语句模块，用于分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；
    获取参考答案模块，用于根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；
    校验模块，用于将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
  9. 一种计算机可读非易失性存储介质，所述计算机可读非易失性存储介质用于存储计算机指令，当其在计算机上运行时，使得计算机可以执行一种语义分析模型准确度的校验方法，所述语义分析模型准确度的校验方法包括以下步骤：
    获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；
    分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；
    根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题语句对应的参考答案，并将所述问题语句与参考答案进行关联；
    将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
  10. 根据权利要求9所述的非易失性存储介质，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤，包括：
    将测试数据集划分为第一测试数据集及第二测试数据集；
    依次抽取第一测试数据集中的第一测试关键词、第二测试数据集中的第二测试关键词及期望关键词生成问题语句。
  11. 根据权利要求9所述的非易失性存储介质，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤之后，还包括：
    调用语法规则对所述问题语句进行语句润化处理，以使所述问题语句符合语法规则。
  12. 根据权利要求9或11所述的非易失性存储介质，所述得到问题语句集的步骤之后，还包括：
    对所述问题语句集中的问题语句进行语义分析，剔除其中无意义的问题语句。
  13. 根据权利要求9所述的非易失性存储介质，所述分别从所述测试 数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集的步骤，包括：
    从测试数据集中抽取至少一个测试关键词，从期望数据集中抽取一个期望关键词；
    将抽取出来的所述测试关键词与期望关键词进行组合，生成问题语句；
    将生成的问题语句集合起来形成问题语句集。
  14. 根据权利要求9所述的非易失性存储介质，若遍历项目信息并未获得问题语句的参考答案，所述获取各个问题语句对应的参考答案的步骤，包括：
    将问题语句划归为未解问题语句集合中，为所述未解问题语句集中的问题语句设置统一参考答案。
  15. 根据权利要求9所述的非易失性存储介质，所述将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度的步骤，包括：
    若所述参考答案与识别答案一致，则标记为识别正确；
    分别统计识别正确的问题语句数量与问题语句集中问题语句的数量，计算识别正确的问题语句在问题语句集中的数量占比，获得语义分析模型的识别准确度。
  16. 一种计算机设备，所述计算机设备包括：
    一个或多个处理器；
    存储装置，用于存储一个或多个程序，
    当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现一种语义分析模型准确度的校验方法，所述一种语义分析模型准确度的校验方法包括以下步骤：
    获取项目信息，从所述项目信息中提取关键词，按照词义属性将所述关键词划分为测试关键词和期望关键词，得到测试数据集和期望数据集；
    分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集；
    根据所述问题语句集中的问题语句解析所述项目信息，获取各个问题 语句对应的参考答案，并将所述问题语句与参考答案进行关联；
    将所述问题语句集中的各个问题语句输入语义分析模型进行识别，获取输出的识别答案，将所述问题语句所关联的参考答案与识别答案进行对比，获得语义分析模型的识别准确度。
  17. 根据权利要求16所述的计算机设备，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤，包括：
    将测试数据集划分为第一测试数据集及第二测试数据集；
    依次抽取第一测试数据集中的第一测试关键词、第二测试数据集中的第二测试关键词及期望关键词生成问题语句。
  18. 根据权利要求16所述的计算机设备，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句的步骤之后，还包括：
    调用语法规则对所述问题语句进行语句润化处理，以使所述问题语句符合语法规则。
  19. 根据权利要求16或18所述的计算机设备，所述得到问题语句集的步骤之后，还包括：
    对所述问题语句集中的问题语句进行语义分析，剔除其中无意义的问题语句。
  20. 根据权利要求16所述的计算机设备，所述分别从所述测试数据集和期望数据集中抽取测试关键词和期望关键词合成问题语句，得到问题语句集的步骤，包括：
    从测试数据集中抽取至少一个测试关键词，从期望数据集中抽取一个期望关键词；
    将抽取出来的所述测试关键词与期望关键词进行组合，生成问题语句；
    将生成的问题语句集合起来形成问题语句集。
PCT/CN2019/103024 2019-05-24 2019-08-28 语义分析模型准确度的校验方法、装置、存储介质及设备 WO2020237872A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910441486.5A CN110298030B (zh) 2019-05-24 2019-05-24 语义分析模型准确度的校验方法、装置、存储介质及设备
CN201910441486.5 2019-05-24

Publications (1)

Publication Number Publication Date
WO2020237872A1 true WO2020237872A1 (zh) 2020-12-03

Family

ID=68027215

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/103024 WO2020237872A1 (zh) 2019-05-24 2019-08-28 语义分析模型准确度的校验方法、装置、存储介质及设备

Country Status (2)

Country Link
CN (1) CN110298030B (zh)
WO (1) WO2020237872A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112989789A (zh) * 2021-03-15 2021-06-18 京东数科海益信息科技有限公司 文本审核模型的测试方法、装置、计算机设备及存储介质
CN114239575A (zh) * 2021-12-20 2022-03-25 北京聆心智能科技有限公司 语句分析模型的构建方法、语句分析方法、装置、介质和计算设备
CN115293749A (zh) * 2022-10-08 2022-11-04 深圳市地铁集团有限公司 一种基于bim导出工程量清单的方法、系统及设备
CN116503889A (zh) * 2023-01-18 2023-07-28 苏州工业园区航星信息技术服务有限公司 档案及电子文件的处理方法、装置、设备及存储介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111652595A (zh) * 2020-06-15 2020-09-11 南京倍时佳信息科技有限公司 一种企业管理咨询用的培训系统
CN112989002B (zh) * 2021-03-31 2022-08-23 中国工商银行股份有限公司 一种基于知识图谱的问答处理方法、装置及设备
CN114240101A (zh) * 2021-12-02 2022-03-25 支付宝(杭州)信息技术有限公司 一种风险识别模型的验证方法、装置以及设备

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170140289A1 (en) * 2015-11-17 2017-05-18 International Business Machines Corporation Automatically Assessing Question Answering System Performance Across Possible Confidence Values
US20170140290A1 (en) * 2015-11-18 2017-05-18 International Business Machines Corporation Automated Similarity Comparison of Model Answers Versus Question Answering System Output
CN108960574A (zh) * 2018-06-07 2018-12-07 百度在线网络技术（北京）有限公司 问答的质量确定方法、装置、服务器和存储介质

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101678787B1 (ko) * 2015-07-15 2016-12-06 포항공과대학교 산학협력단 자동질의응답 방법 및 그 장치
RU2640297C2 (ru) * 2016-05-17 2017-12-27 Общество с ограниченной ответственностью "Аби Продакшн" Определение степеней уверенности, связанных со значениями атрибутов информационных объектов
US10394950B2 (en) * 2016-08-22 2019-08-27 International Business Machines Corporation Generation of a grammatically diverse test set for deep question answering systems
RU2636098C1 (ru) * 2016-10-26 2017-11-20 Общество с ограниченной ответственностью "Аби Продакшн" Использование глубинного семантического анализа текстов на естественном языке для создания обучающих выборок в методах машинного обучения
US10275515B2 (en) * 2017-02-21 2019-04-30 International Business Machines Corporation Question-answer pair generation
CN109582761A (zh) * 2018-09-21 2019-04-05 浙江师范大学 一种基于网络平台的词语相似度的中文智能问答系统方法
CN109766423A (zh) * 2018-12-29 2019-05-17 上海智臻智能网络科技股份有限公司 基于神经网络的问答方法及装置、存储介质、终端

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170140289A1 (en) * 2015-11-17 2017-05-18 International Business Machines Corporation Automatically Assessing Question Answering System Performance Across Possible Confidence Values
US20170140290A1 (en) * 2015-11-18 2017-05-18 International Business Machines Corporation Automated Similarity Comparison of Model Answers Versus Question Answering System Output
CN108960574A (zh) * 2018-06-07 2018-12-07 百度在线网络技术（北京）有限公司 问答的质量确定方法、装置、服务器和存储介质

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112989789A (zh) * 2021-03-15 2021-06-18 京东数科海益信息科技有限公司 文本审核模型的测试方法、装置、计算机设备及存储介质
CN114239575A (zh) * 2021-12-20 2022-03-25 北京聆心智能科技有限公司 语句分析模型的构建方法、语句分析方法、装置、介质和计算设备
CN115293749A (zh) * 2022-10-08 2022-11-04 深圳市地铁集团有限公司 一种基于bim导出工程量清单的方法、系统及设备
CN115293749B (zh) * 2022-10-08 2022-12-27 深圳市地铁集团有限公司 一种基于bim导出工程量清单的方法、系统及设备
CN116503889A (zh) * 2023-01-18 2023-07-28 苏州工业园区航星信息技术服务有限公司 档案及电子文件的处理方法、装置、设备及存储介质
CN116503889B (zh) * 2023-01-18 2024-01-19 苏州工业园区航星信息技术服务有限公司 档案及电子文件的处理方法、装置、设备及存储介质

Also Published As

Publication number Publication date
CN110298030B (zh) 2022-06-17
CN110298030A (zh) 2019-10-01

Similar Documents

Publication Publication Date Title
WO2020237872A1 (zh) 语义分析模型准确度的校验方法、装置、存储介质及设备
CN110209764B (zh) 语料标注集的生成方法及装置、电子设备、存储介质
US10169706B2 (en) Corpus quality analysis
US10769552B2 (en) Justifying passage machine learning for question and answer systems
CN106503055B (zh) 一种从结构化文本到图像描述的生成方法
US10565533B2 (en) Systems and methods for similarity and context measures for trademark and service mark analysis and repository searches
US9286290B2 (en) Producing insight information from tables using natural language processing
US9621601B2 (en) User collaboration for answer generation in question and answer system
Chen et al. BigGorilla: An open-source ecosystem for data preparation and integration.
US9146987B2 (en) Clustering based question set generation for training and testing of a question and answer system
US9230009B2 (en) Routing of questions to appropriately trained question and answer system pipelines using clustering
US20160171095A1 (en) Identifying and Displaying Relationships Between Candidate Answers
WO2020057021A1 (zh) 数据表处理方法、装置、计算机设备和存储介质
CN104484336B (zh) 一种中文评论分析方法及其系统
WO2019232893A1 (zh) 文本的情感分析方法、装置、计算机设备和存储介质
CN108549723B (zh) 一种文本概念分类方法、装置及服务器
CN107436916B (zh) 智能提示答案的方法及装置
US20150269142A1 (en) System and method for automatically generating a dataset for a system that recognizes questions posed in natural language and answers with predefined answers
CN110287314B (zh) 基于无监督聚类的长文本可信度评估方法及系统
US9558462B2 (en) Identifying and amalgamating conditional actions in business processes
US10248648B1 (en) Determining whether a comment represented as natural language text is prescriptive
KR102280490B1 (ko) 상담 의도 분류용 인공지능 모델을 위한 훈련 데이터를 자동으로 생성하는 훈련 데이터 구축 방법
KR102126911B1 (ko) KeyplayerRank를 이용한 소셜 미디어상의 주제별 키플레이어 탐지 방법
US11507593B2 (en) System and method for generating queryeable structured document from an unstructured document using machine learning
CN110941638A (zh) 应用分类规则库构建方法、应用分类方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19931421

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19931421

Country of ref document: EP

Kind code of ref document: A1