CN111931488A

CN111931488A - Method, device, electronic equipment and medium for verifying accuracy of judgment result

Info

Publication number: CN111931488A
Application number: CN202011016753.3A
Authority: CN
Inventors: 邢智慧; 陈俊; 陆超; 黄海峰
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2020-09-24
Filing date: 2020-09-24
Publication date: 2020-11-13
Anticipated expiration: 2040-09-24
Also published as: US20210209482A1; CN111931488B

Abstract

The embodiment of the application discloses a method, a device, electronic equipment and a computer readable storage medium for verifying the accuracy of a judgment result, relates to the field of artificial intelligence such as knowledge graph and natural language processing, and can be used for assisting diagnosis decision. One embodiment of the method comprises: acquiring user description information and a judgment result corresponding to the user description information; extracting at least one description element from the user description information; respectively determining the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value, wherein the verification model is obtained by training based on the word frequency and inverse text frequency index relationship between the description elements and the judgment results; and determining the description elements with the actual matching values exceeding the preset matching values as first elements, and determining the accuracy of the judgment result based on the number of the first elements. By applying the embodiment, the accuracy of the judgment result can be verified, so that whether the result is credible or not can be definitely judged based on the accuracy verification result.

Description

Method, device, electronic equipment and medium for verifying accuracy of judgment result

Technical Field

The present application relates to the field of data processing technologies, and in particular, to the field of artificial intelligence, such as knowledge graph and natural language processing, and more particularly, to a method and an apparatus for verifying accuracy of a determination result, an electronic device, and a computer-readable storage medium.

Background

With the continuous improvement of electronic informatization degree and the continuous development of big data technology, many schemes for preliminarily sorting and analyzing user description information and obtaining preliminary conclusions, such as whether the user description information belongs to a certain type of data, whether the user description information belongs to valid data, whether the user description information is matched with certain preset information, and the like, are created, so that the processing efficiency is improved through the information obtained through automatic analysis.

Disclosure of Invention

The embodiment of the application provides a method and a device for verifying the accuracy of a judgment result, electronic equipment and a computer-readable storage medium.

In a first aspect, an embodiment of the present application provides a method for verifying accuracy of a determination result, including: acquiring user description information and a judgment result corresponding to the user description information; extracting at least one description element from the user description information; respectively determining the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value; the verification model is obtained by training based on the word frequency between the description elements and the judgment result and the inverse text frequency index TF-IDF relation; and determining the description elements with the actual matching values exceeding the preset matching values as first elements, and determining the accuracy of the judgment result based on the number of the first elements.

In a second aspect, an embodiment of the present application provides an apparatus for verifying accuracy of a determination result, including: a user description information and determination result acquisition unit configured to acquire user description information and a determination result corresponding to the user description information; a description element extraction unit configured to extract at least one description element from the user description information; the matching degree verification unit is configured to respectively determine the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value; the verification model is obtained by training based on the word frequency between the description elements and the judgment result and the inverse text frequency index TF-IDF relation; and a first element and judgment accuracy determination unit configured to determine, as the first element, a description element in which the actual matching value exceeds the preset matching value, and determine the accuracy of the judgment result based on the number of the first elements.

In a third aspect, an embodiment of the present application provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to implement the method for verifying the accuracy of the determination result as described in any implementation manner of the first aspect.

In a fourth aspect, the present application provides a non-transitory computer-readable storage medium storing computer instructions for enabling a computer to implement the method for verifying the accuracy of a determination result as described in any implementation manner of the first aspect when executed.

According to the method, the device, the electronic equipment and the computer-readable storage medium for verifying the accuracy of the judgment result, firstly, user description information and the judgment result corresponding to the user description information are obtained; then, extracting at least one description element from the user description information; secondly, respectively determining the matching degree between each description element and the judgment result by utilizing a preset verification model obtained by training based on the word frequency between the description element and the judgment result and the inverse text frequency index TF-IDF relation to obtain each actual matching value; and finally, determining the description elements with the actual matching values exceeding the preset matching values as first elements, and determining the accuracy of the judgment result based on the number of the first elements.

In order to determine whether the judgment result is accurate in the user description information, the method utilizes a pre-trained verification model of TF-IDF relation between the characterization description elements and the judgment result to verify whether the actual description elements extracted from the user description information are matched with the actual judgment result, so that the matching degree between each description element and the judgment result is obtained, and finally the accuracy of the judgment result is reflected by the number of the matched description elements, so that the reliability of the judgment result is improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

Other features, objects and advantages of the present application will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, made with reference to the accompanying drawings in which:

FIG. 1 is an exemplary system architecture to which the present application may be applied;

fig. 2 is a flowchart of a method for verifying the accuracy of a determination result according to an embodiment of the present disclosure;

fig. 3 is a flowchart of another method for verifying the accuracy of the determination result according to the embodiment of the present application;

fig. 4 is a schematic flowchart of a method for verifying accuracy of a judgment result of whether a malicious webpage belongs to in a malicious webpage identification scenario according to an embodiment of the present application;

fig. 5 is a schematic flowchart of a method for verifying accuracy of a diagnosis result of a suspected disease in a disease diagnosis scenario according to an embodiment of the present application;

FIG. 6 is a schematic flow chart of an embodiment of the present application for obtaining a diagnosis reason corresponding to FIG. 5;

fig. 7 is a schematic flowchart of an authentication process based on the TF-IDF relationship, which corresponds to fig. 5 and is provided in an embodiment of the present application;

fig. 8 is a block diagram illustrating an apparatus for verifying the accuracy of a determination result according to an embodiment of the present disclosure;

fig. 9 is a schematic structural diagram of an electronic device suitable for executing the method for verifying the accuracy of the determination result according to an embodiment of the present disclosure.

Detailed Description

The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.

It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.

Fig. 1 illustrates an exemplary system architecture 100 to which embodiments of the present methods, apparatuses, electronic devices and computer-readable storage media for verifying the accuracy of determination results may be applied.

As shown in fig. 1, the system architecture 100 may include

terminal devices

101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, and 103 and the server 105 may be installed with various applications for communicating information therebetween, such as an information transceiving application, a description information analysis application, a determination result analysis application, and the like.

The

terminal apparatuses

101, 102, 103 and the server 105 may be hardware or software. When the

terminal devices

101, 102, 103 are hardware, they may be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like; when the

terminal devices

101, 102, and 103 are software, they may be installed in the electronic devices listed above, and they may be implemented as multiple software or software modules, or may be implemented as a single software or software module, and are not limited in this respect. When the server 105 is hardware, it may be implemented as a distributed server cluster composed of multiple servers, or may be implemented as a single server; when the server is software, the server may be implemented as a plurality of software or software modules, or may be implemented as a single software or software module, which is not limited herein.

The server 105 may provide various services through various built-in applications, taking as an example a determination result analysis application that may provide a service that verifies whether a determination result corresponding to the user description information is accurate, and when running the determination result analysis application, the server 105 may achieve the following effects: first, user description information input by a user through the

terminal apparatus

101, 102, 103 used by the user and a judgment result given by the

terminal apparatus

101, 102, 103 to the input user description information are received through the network 104; then, extracting at least one description element from the user description information; secondly, respectively determining the matching degree between each description element and the judgment result by utilizing a preset verification model obtained by training based on the word frequency between the description element and the judgment result and the inverse text frequency index TF-IDF relation to obtain each actual matching value; and finally, determining the description elements with the actual matching values exceeding the preset matching values as first elements, and determining the accuracy of the judgment result based on the number of the first elements. Further, the server 105 may also output a parameter representing the accuracy of the determination result according to a preset path. The preset path includes, but is not limited to, a path to return to the

terminal apparatus

101, 102, 103, and a storage path of another data server.

It should be noted that the user description information and the determination result may be acquired from the

terminal apparatuses

101, 102, and 103 in real time through the network 104, or may be stored locally in the server 105 in advance in various ways. Thus, when the server 105 detects that such data is already stored locally (e.g., pending accuracy computing tasks remaining before starting processing), it may choose to retrieve such data directly from locally, in which case the exemplary system architecture 100 may also not include the

terminal devices

101, 102, 103 and the network 104.

Since the calculation of accuracy involves a large amount of operations, and requires a large amount of computing resources and a strong computing power, the method for verifying the accuracy of the determination result provided in the following embodiments of the present application is generally executed by the server 105 having a strong computing power and a large amount of computing resources, and accordingly, the apparatus for verifying the accuracy of the determination result is generally disposed in the server 105. However, it should be noted that when the

terminal devices

101, 102, and 103 also have computing capabilities and computing resources that meet the requirements, the

terminal devices

101, 102, and 103 may also complete the above-mentioned operations that are originally delivered to the server 105 through the determination result analysis application installed thereon, and then output the same result as the server 105. Especially, when there are a plurality of terminal devices having different computation capabilities at the same time, but the determination result analysis application determines that the terminal device has a strong computation capability and a large amount of computing resources are left, the terminal device may execute the above computation, so as to appropriately reduce the computation pressure of the server 105, and accordingly, the device for verifying the accuracy of the determination result may be disposed in the

terminal devices

101, 102, and 103. In such a case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Referring to fig. 2, fig. 2 is a flowchart of a method for verifying accuracy of a determination result according to an embodiment of the present application, wherein the process 200 includes the following steps:

step 201: acquiring user description information and a judgment result corresponding to the user description information;

this step is intended to acquire user description information (sent by the

terminal apparatuses

101, 102, 103 shown in fig. 1 through the network 104, for example) and a determination result corresponding to the user description information by an execution subject (server 105 shown in fig. 1, for example) of the method for verifying the accuracy of the determination result.

The user description information refers to information obtained by describing a certain aspect of the user or content known by the user, such as a written conference log, written webpage content, article content, description of recent physical health conditions of the user, and the like; the judgment result corresponding to the user description information refers to an analysis result output by some functional modules (e.g., SDK provided by a third party, that is, third party application toolkit) capable of realizing a judgment function according to content in the user description information, for example, when the user description information is a conference log written by the user, the functional module may obtain a judgment result whether the conference log is written according to a preset format, for example, when the user description information is web page article content written by the user, the functional module may obtain a judgment result whether the web page article content includes a preset prohibited sensitive word, for example, when recent user description information is medical history information describing a health state of the user, the functional module may obtain a suspected disease that the user may have a certain disease, and the like.

Specifically, how the functional module gives the judgment result for the content of the user description information is not the key point to be researched in the application, and only the situation that the user description information and the judgment result corresponding to the user description information can be obtained is explained in the step, that is, how to verify the accuracy of the judgment result given by the functional module through subsequent steps is performed for the situation, so that a user or other personnel can more clearly know how the judgment result is obtained, the judgment result is used for supporting the recognition degree of the user on the judgment result, meanwhile, the inaccurate judgment result can be screened, and the accuracy of the judgment result obtained by the functional module is guided to be improved.

Step 202: extracting at least one description element from the user description information;

on the basis of step 201, this step is intended to extract at least one description element from the user description information by the execution body described above. Further, at least one of the descriptive elements may also be represented as a set of descriptive elements to better facilitate the use of the subsequent steps as a whole.

The operation of extracting the description elements from the user description information is feature extraction or key information extraction, and the key description elements can be extracted from the user description information containing the non-description elements and the description elements in the modes of word segmentation, keyword extraction, synonym processing and the like, so that the interference caused by the participation of the non-description elements in the subsequent steps is avoided.

In step 201, it has been clarified that the user description information can be represented as different types of information according to different actual situations, and for different types of user description information, the extraction of the description element should be performed not only in combination with the type thereof, but also in combination with the function module to obtain the entry point of the determination result, so as to extract the description element that is more matched with the determination result obtained from the function module. Specifically, when the user description information is a conference log and the determination result is a result indicating whether the format is a preset format, the extracted description elements should be focused on some elements representing the writing format of the conference log, such as character strings of a first segment/a last segment, whether a second segment is blank, and the like; when the user description information is the content of a webpage article and whether the judgment result contains sensitive words or not, the extracted description elements are focused on related keywords related to the sensitive content; when the user description information is an electronic medical record and the judgment result is that the user description information is related to a suspected disease, the extracted description elements should focus on medical entities, disease elements and the like related to disease symptoms.

Step 203: respectively determining the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value;

the verification model is obtained by training based on TF-IDF (Term Frequency-Inverse text Frequency index) relation between the description elements and the judgment result.

TF-IDF is a statistical method to evaluate the importance of a word to one of a set of documents or a corpus. The importance of a word increases in proportion to the number of times it appears in a document, but at the same time decreases in inverse proportion to the frequency with which it appears in the corpus. Thus, the more a word appears in an article, the less it appears in all documents, and the more representative the article. Based on this characteristic, the application actually helps the machine to understand the association degree between some words and the user description of the words from the semantic level by means of the TF-IDF statistical method as much as possible.

It should be noted that, the reason why the matching degree between the description element and the judgment result is determined by the verification model trained based on the TF-IDF relationship is because the judgment result is usually obtained by looking at whether the description element hits the support element constituting the judgment result in the manner of supporting the judgment result, and the hit manner is performed by the "hard" matching manner, that is, whether there are identical characters, and descriptions of identical contents by different users are often different, so that the problem that all descriptions cannot be hit easily occurs, and the reason for supporting the judgment result is insufficient and the reliability is not high. Therefore, in order to obtain a comprehensive and correct support basis as much as possible, other methods are needed to hit the description elements as much as possible, the TF-IDF relationship is used in the step to solve the problem from a semantic level, and a new support element can be found, so that the number of hit elements can be increased and the reliability can be improved by increasing the total number of support elements (which can be obtained through knowledge related to the judgment result recorded by the knowledge map).

On the basis of step 202, this step is intended to determine the matching degree between each description element and the judgment result by the execution subject using the verification model for representing the association relationship between the description element and the judgment result at the semantic level, and obtain the actual matching value of the quantifiable description.

Step 204: and determining the description elements with the actual matching values exceeding the preset matching values as first elements, and determining the accuracy of the judgment result based on the number of the first elements.

On the basis of step 203, this step is intended to determine, by the execution subject described above, a description element whose actual matching value exceeds a preset matching value as a first element, and determine the accuracy of the determination result based on the number of the first elements. That is, the greater the number of the first elements, the greater the evidence that can be used to support the determination result, and thus the more accurate the determination result can be verified. Specifically, the parameter for representing the accuracy of the determination result may be only the number of the first elements, may also be a ratio between the number of the first elements and the total number of all the supporting elements, or another operation manner that the accuracy is higher as the number of the first elements increases, and is not limited specifically here.

Further, the determined accuracy may be expressed in various manners, for example, the accuracy may be simply divided into three levels, i.e., low, medium, and high, and may also be converted into specific numerical values under percentage, and specifically, which manner may be further used in combination with which degree the subsequent accuracy is in and also performs targeted processing, for example, the subsequent execution measures are only divided into two types, which correspond to what kind of operations should be executed accurately and inaccurately, and then the accuracy determined in this step may be divided into two levels, for example, the division exceeding 80 times in percentage is accurate, and the remaining division is inaccurate.

In order to determine whether the judgment result is accurate or not for the user description information, the method for verifying the judgment result accuracy provided by the application verifies whether the actual description element extracted from the user description information is matched with the actual judgment result or not by using a verification model of a TF-IDF relation between a pre-trained characterization description element and the judgment result, so that the matching degree between each description element and the judgment result is obtained, and finally the accuracy of the judgment result is reflected by the number of the matched description elements, so that the reliability of the judgment result is improved.

Referring to fig. 3, fig. 3 is a flowchart of another method for verifying the accuracy of the determination result according to the embodiment of the present application, wherein the process 300 includes the following steps:

step 301: acquiring user description information and a judgment result corresponding to the user description information;

step 302: extracting at least one description element from the user description information;

step 303: respectively determining the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value;

step 304: determining the description element with the actual matching value exceeding the preset matching value as a first element;

the above steps 301-304 are the same as the previous parts of the steps 201-203 and 204 shown in fig. 2, and the contents of the same parts refer to the corresponding parts of the previous embodiment, which are not described herein again.

Step 305: determining a plurality of knowledge elements related to the judgment result by using a preset knowledge graph;

step 306: determining a description element consistent with any knowledge element as a second element;

in a manner different from the manner of determining the first element from the semantic level based on the TF-IDF relationship provided in steps 304-304, step 305 determines a plurality of knowledge elements related to the determination result through a preset knowledge graph, and then determines the description element consistent with any knowledge element as the second element through step 306, that is, the determination of the hit element is also performed in combination with the "hard" knowledge recorded by the knowledge graph.

In some other embodiments of the present embodiment, regarding how to determine the second element, the following implementation is also provided:

acquiring the confidence coefficient of each knowledge element;

respectively calculating to obtain the similarity between each description element and each knowledge element;

and determining the knowledge elements and the description elements with consistency according to the similarity and the confidence, and determining the description elements with consistency with any knowledge elements as second elements.

The determination of the knowledge elements and description elements with consistency from the similarity and confidence may be expressed as a formula: score (sim, spo _ item) confi (spo _ item), where sim represents similarity calculation, finding represents the extracted description element, spo _ item represents the relationship pair between the element and the determination result, and confi represents the confidence.

The similarity measures the similarity between the description elements and the knowledge elements, is used for replacing a conventional hard matching mode, and is calculated by adopting a mode of combining a model with rule judgment: and (3) rule judgment: upper and lower relation, parent-child character string; unsupervised learning: and pre-training a large amount of related historical data in an off-line manner to obtain pre-training vectors, and calculating the similarity based on the word vectors and the word vectors. Similarity model: training data: mining the superior-inferior relation of elements in the knowledge graph big picture, wherein the elements with the superior-inferior relation have higher similarity and are used as basic training data, and more similar pairs are mined through model iteration; model structure: the model structure can fully extract similar and same words and characters between elements, reflect the similarity between the elements, increase mask and random disturbance, and increase the robustness and the migration capability of the model.

The confidence degree of the knowledge elements represents the credibility of each knowledge element, so the parameter exists because the data in the knowledge graph is possibly wrong, namely the accuracy problem exists in the knowledge graph mining elements, so the confidence degrees of different knowledge elements can be given by evaluating the sources of the knowledge elements, such as information issued by professional books, textbooks and authoritative certification authorities, the confidence degree of the extracted knowledge elements is marked as 1, and the confidence degrees can be properly adjusted by taking the extracted confidence degrees as the standard, and the accuracy of consistency determination is improved by combining the two aspects.

Step 307: and determining the accuracy of the judgment result according to the number of the first elements and the second elements.

Unlike the embodiment shown in the flowchart 200, the embodiment adds the solution provided by steps 305 to 306 for determining the second element from another angle through the knowledge map, so that the accuracy of the determination result is adaptively adjusted according to the number of the first element and the second element when determining the accuracy.

Specifically, considering that there may be duplicate items in the first element and the second element, the first element and the second element may be subjected to deduplication processing to obtain an element set, and then the ratio of the number of elements in the element set to the total number of knowledge elements is used as the accuracy of the determination result, so as to obtain an accuracy conclusion as accurate as possible by using the characteristic that the same element only has one item in the set.

Unlike the previous embodiment shown in the flowchart 200, the present embodiment provides another scheme for attempting to hit elements related to the determination result and capable of supporting that the determination result is accurate based on the knowledge graph through steps 305 to 306, and combines with the scheme for verifying the matching degree between the description elements and the determination result from the semantic level through training based on the TF-IDF relationship, which is provided through steps 303 to 304, so that the accuracy determined according to the number of the first elements and the second elements at the same time is more accurate.

In addition to the embodiment of the flow 300 shown in fig. 3, in response to the existence of the first element different from any knowledge element, the verification model trained by the TF-IDF relationship also finds a new knowledge element that is not mined during the construction of the knowledge graph, so that the first element different from any knowledge element can be added to the knowledge graph as a supplementary element to improve the comprehensiveness of the knowledge elements related to the determination result recorded in the knowledge graph.

Furthermore, because the first element obtained by the verification model and the second element obtained by the knowledge graph are combined at the same time, the calculated quantity in the two modes possibly has different influences on the finally determined accuracy, and therefore the accuracy can be improved by combining a weighting idea.

One implementation, including but not limited to, may be:

acquiring a first quantity weight and a second quantity weight which are respectively set for a first element and a second element in advance;

calculating the number of the first elements, the first number weight, the number of the second elements and the second number weight according to a weighting calculation method to obtain the number of the weighted elements;

and determining the accuracy of the judgment result according to the weighted element number.

For further understanding, the present application also incorporates a specific application scenario: the user description information is specifically a web page article written by the user, and the determination result is a result of whether the content of the web page article is prohibited from being displayed because there are more sensitive words, and a specific implementation scheme that can finally verify whether the determination result is accurate is provided, please refer to a flow 400 shown in fig. 4:

step 401: acquiring input webpage article content and a judgment result of forbidden display;

step 402: extracting at least one suspected sensitive word from the webpage article content;

specifically, the suspected sensitive word extracted should be a feature that characterizes whether the word belongs to a sensitive word, such as whether the word includes a direct sensitive word like "bend", "bay", "expose", "free" or a non-direct sensitive word slightly modified from the direct sensitive word, such as "expose", "free 0".

Step 403: respectively determining the matching degree between each suspected sensitive word and the judgment result by using a preset verification model to obtain each actual matching value;

assuming that 60 suspected sensitive words are obtained by extraction, actual matching values of the suspected sensitive words are obtained by calculation respectively, wherein 48 actual matching values exceed 85 points (full points 100 points), 6 actual matching values exceed 75 points and do not exceed 85 points, and 6 actual matching values do not exceed 75 points.

Step 404: determining the suspected sensitive words with the actual matching values exceeding the preset matching values as first sensitive words;

assuming that the preset match value is 85 points, there are 48 first sensitive words.

Step 405: determining a plurality of actual sensitive words related to the webpage sensitive words by using a preset knowledge graph;

assume that there are 100 actual sensitive words recorded in the knowledge-graph.

Step 406: determining a suspected sensitive word consistent with any actual sensitive word as a second sensitive word;

assume that a total of 45 second sensitive words are determined by this step.

Step 407: highlighting the first sensitive words and the second sensitive words in the actual sensitive words, and generating the reason for forbidding the display of the webpage article content according to the elements including non-highlighted and highlighted.

Combining the 48 first sensitive words determined in the step 404 and the 45 second sensitive words determined in the step 406, the second sensitive words are found to be 43 in number by de-duplication, so that there are 50 sensitive words in total. Therefore, 100 actual sensitive words are used as candidate sensitive words, and are highlighted together with the above-mentioned 50 sensitive words, so as to present which sensitive words specifically exist in the web page article content through highlighting, and present the web page article content to the writer and the background manager as a specific reason (more than 30 sensitive words) that the web page article content is prohibited from being displayed.

To further enhance understanding, the present embodiment also incorporates another application scenario different from the process 400: a specific implementation manner is provided for the user description information being an electronic medical record of the user and the determination result being a suspected disease, please refer to a flow 500 shown in fig. 5:

step 501: acquiring an electronic medical record of a patient and a suspected disease corresponding to the content of the electronic medical record;

assume that the electronic medical record content is: cough with white sputum with headache for two days. The suspected diseases are: acute upper respiratory infection.

The intersection of the AB sets is cough and white sputum (cough is hard match, white sputum and expectoration are correlation obtained by the similarity model above), and according to the developed tfidf relation and disease element model, we judge that headache and upper respiratory tract infection are also related, so we highlight cough, white sputum and headache, and highlight show that the rest elements in the knowledge graph are fever, sneeze, rhinorrhea, aversion to cold and the like. This distinguishes between highlighted and non-highlighted, both of which are related to the disease, only those that appear in the medical record will be highlighted, and those that do not appear will not be highlighted

Step 502: extracting at least one disease element from the electronic medical record;

the extracted disease factors are cough, expectoration and headache.

Step 503: determining a plurality of knowledge elements under suspected diseases by using a preset medical knowledge map;

the elements of the knowledge map that provide relevance for upper respiratory tract infection are: cough, expectoration, fever, sneezing, runny nose, aversion to cold, and the like.

Step 504: determining a disease element consistent with an arbitrary knowledge element as a first element;

based on the "hard" matching between the disease elements described in the knowledge map and the actually obtained disease elements, "cough" and "expectoration" can be determined as the first elements.

Step 505: respectively determining the matching degree between each disease element and the suspected disease by using a preset verification model to obtain each actual matching value;

the steps are respectively obtained through the provided verification model: the actual match value for "cough" was 97 points, the actual match value for "expectoration" was 93 points, and the actual match value for "headache" was 86 points.

Step 506: determining the disease elements with the actual matching values exceeding the preset matching values as second elements;

in the case where the preset matching value is 85 points, "cough", "expectoration", and "headache" all belong to the second elements.

Step 507: highlighting the first element and the second element of the knowledge elements, and generating a diagnosis reason for the suspected disease based on the elements including the non-highlighted and highlighted elements.

Therefore, the disease element "headache" is also supplemented to the disease element structure of the disease of the upper respiratory tract infection by the first element and the second element, three items "cough", "expectoration" and "headache" among all the disease elements of the disease are highlighted, and the three highlighted disease elements are used to generate a diagnosis reason for judging that the patient has acute upper respiratory tract disturbance.

The above process can also be seen in the schematic flow chart of fig. 6 for obtaining the diagnosis reason, wherein the attention weight refers to the degree of association between each disease element and the suspected disease, and a higher weight represents a higher support strength between the disease element and the determination of the suspected disease; the score of the highlight element is the score calculated by combining the attention weight and the two parts, and the highlight element is highlighted when a certain score is exceeded; and (5) relation pairs of diseases, disease elements and types are mined by the knowledge graph. The process shown in fig. 6 can be briefly described as follows: and regarding each disease element and each suspected disease extracted from the electronic medical record as a relationship pair between the diseases and the disease elements, recalling a reasonable relationship pair and highlighting the relationship pair, and displaying the diagnosis reason according to a highlighted result and a non-highlighted result. The recall part also comprises recall based on the knowledge graph and recall based on semantic association of disease-disease elements, and the combination of the two schemes gives consideration to knowledge evidence and natural language understanding of medical record contents, thereby improving the accuracy.

The above-mentioned verification model trained based on the TF-IDF relationship between disease-disease elements can be embodied as a flow diagram shown in fig. 7, that is, the TF-IDF relationship between disease and disease elements is mined off-line according to a disease condition map, a disease element matching model (i.e., the above-mentioned verification model) is trained, a TF-IDF relationship pair between disease and disease elements, a relationship pair of disease, disease elements and types mined from a knowledge map are used as an initial training set, and a text type neural network and cosine matching calculation mechanism can be specifically selected by combining the models. The process shown in FIG. 7 can be briefly described as follows: and when the matching score of the disease and disease elements is higher than a set threshold value after being calculated through TF-IDF and a model, adding a highlight set, when the matching score is lower than the set threshold value, introducing a weight generated in an AI diagnosis process for additional weighting, wherein the higher weight represents that the element has higher importance when the suspected disease is diagnosed, and giving a final score of the element after the comprehensive evaluation of the two strategies is combined to judge whether the highlight set is added.

With further reference to fig. 8, as an implementation of the method shown in the above figures, the present application provides an embodiment of an apparatus for verifying the accuracy of the determination result, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be applied to various electronic devices.

As shown in fig. 8, the apparatus 600 for verifying the accuracy of the determination result of the present embodiment may include: a user description information and judgment result acquisition unit 601, a description element extraction unit 602, a matching degree verification unit 603, and a first element and judgment accuracy determination unit 604. A user description information and determination result obtaining unit 601 configured to obtain user description information and a determination result corresponding to the user description information; a description element extraction unit 602 configured to extract at least one description element from the user description information; a matching degree verification unit 603 configured to determine a matching degree between each description element and the determination result by using a preset verification model, to obtain actual matching values; the verification model is obtained by training based on the word frequency between the description elements and the judgment result and the inverse text frequency index TF-IDF relation; the first element and judgment accuracy determining unit 604 is configured to determine, as the first element, a description element in which the actual matching value exceeds the preset matching value, and determine the accuracy of the judgment result based on the number of the first elements.

In the present embodiment, in the apparatus 600 for verifying the accuracy of the determination result: the detailed processing and the technical effects brought by the user description information and determination result obtaining unit 601, the description element extracting unit 602, the matching degree verifying unit 603, and the first element and determination accuracy determining unit 604 can refer to the related descriptions of step 201 and step 204 in the corresponding embodiment of fig. 2, and are not repeated herein.

In some optional implementations of this embodiment, the apparatus 600 for verifying the accuracy of the determination result may further include:

a knowledge element determination unit configured to determine a plurality of knowledge elements related to the determination result using a preset knowledge map;

a second element determination unit configured to determine a description element that coincides with an arbitrary knowledge element as a second element; and the first element and judgment accuracy determination unit includes:

and a two-factor discrimination result accuracy determining subunit configured to determine the accuracy of the discrimination result according to the number of the first factors and the second factors.

In some optional implementations of the present embodiment, the second element determining unit may be further configured to:

acquiring the confidence coefficient of each knowledge element;

In some optional implementations of the present embodiment, the dual element discrimination result accuracy determination subunit may be further configured to:

carrying out duplicate removal processing on the first element and the second element to obtain an element set;

and taking the ratio of the number of the elements in the element set to the total number of the knowledge elements as the accuracy of the judgment result.

a knowledge element supplementing unit configured to add a first element different from any knowledge element as a supplementing element to the knowledge map in response to the presence of the first element different from any knowledge element.

The present embodiment exists as an embodiment of an apparatus corresponding to the foregoing method embodiment, and in order to determine whether the determination result is accurate in determining the user description information, the apparatus for verifying the accuracy of the determination result provided in the present application verifies whether the actual description element extracted from the user description information matches the actual determination result by using a verification model of a TF-IDF relationship between a pre-trained characterization description element and the determination result, so as to obtain a matching degree between each description element and the determination result, and finally reflects the accuracy of the determination result by the number of the matched description elements, thereby improving the reliability of the determination result.

According to an embodiment of the present application, an electronic device and a computer-readable storage medium are also provided.

Fig. 9 shows a block diagram of an electronic device suitable for implementing the method for verifying the accuracy of the determination result according to the embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.

As shown in fig. 9, the electronic apparatus includes: one or more processors 701, a memory 702, and interfaces for connecting the various components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions for execution within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to the interface). In other embodiments, multiple processors and/or multiple buses may be used, along with multiple memories and multiple memories, as desired. Also, multiple electronic devices may be connected, with each device providing portions of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). In fig. 9, one processor 701 is taken as an example.

The memory 702 is a non-transitory computer readable storage medium as provided herein. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the method for verifying the accuracy of the determination result provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to perform the method for verifying the accuracy of a determination result provided by the present application.

The memory 702, which is a non-transitory computer-readable storage medium, may be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/modules corresponding to the method for verifying the accuracy of the determination result in the embodiment of the present application (for example, the user description information and determination result acquisition unit 601, the description element extraction unit 602, the matching degree verification unit 603, and the first element and determination accuracy determination unit 604 shown in fig. 6). The processor 701 executes various functional applications of the server and data processing by running the non-transitory software programs, instructions, and modules stored in the memory 702, that is, implements the method for verifying the accuracy of the determination result in the above-described method embodiment.

The memory 702 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area can store various types of data and the like created by the electronic equipment in the process of executing the method for verifying the accuracy of the judgment result. Further, the memory 702 may include high speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory 702 may optionally include memory located remotely from the processor 701, which may be connected via a network to an electronic device adapted to perform a method for verifying the accuracy of the determination. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The electronic device adapted to perform the method for verifying the accuracy of the determination result may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected by a bus or other means, and fig. 9 illustrates an example of a connection by a bus.

The input device 703 may receive input numeric or character information and generate key signal inputs related to user settings and function control of an electronic apparatus suitable for performing a method for verifying the accuracy of the determination result, such as an input device of a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointing stick, one or more mouse buttons, a track ball, a joystick, or the like. The output devices 704 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibrating motors), among others. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device can be a touch screen.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented using high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

In order to determine whether the judgment result is accurate in the user description information, in this embodiment, a pre-trained verification model of the TF-IDF relationship between the characterization description elements and the judgment result is used to verify whether the actual description elements extracted from the user description information are matched with the actual judgment result, so as to obtain the matching degree between each description element and the judgment result, and finally, the accuracy of the judgment result is reflected by the number of the matched description elements, so as to improve the reliability of the judgment result.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.

The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for verifying the accuracy of a determination, comprising:

acquiring user description information and a judgment result corresponding to the user description information;

extracting at least one description element from the user description information;

respectively determining the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value; the verification model is obtained by training based on the word frequency between the description elements and the judgment result and the inverse text frequency index TF-IDF relation;

and determining the description elements with the actual matching values exceeding the preset matching values as first elements, and determining the accuracy of the judgment result based on the number of the first elements.

2. The method of claim 1, further comprising:

determining a plurality of knowledge elements related to the judgment result by using a preset knowledge map;

determining a description element consistent with any of the knowledge elements as a second element; and

the determining the accuracy of the judgment result based on the number of the first elements comprises:

and determining the accuracy of the judgment result according to the number of the first elements and the second elements.

3. The method of claim 2, wherein said determining a description element consistent with any of said knowledge elements as a second element comprises:

obtaining the confidence of each knowledge element;

respectively calculating the similarity between each description element and each knowledge element;

and determining knowledge elements and description elements with consistency according to the similarity and the confidence degrees, and determining the description elements with consistency with any knowledge elements as the second elements.

4. The method of claim 2, wherein said determining the accuracy of the determination based on the number of the first and second elements comprises:

5. The method of claim 2, further comprising:

in response to the presence of a first element that is different from any of the knowledge elements, adding a first element that is different from any of the knowledge elements as a supplemental element to the knowledge graph.

6. The method according to any one of claims 2 to 5, wherein the determining the accuracy of the determination result according to the number of the first and second elements comprises:

acquiring a first quantity weight and a second quantity weight which are respectively set for the first element and the second element in advance;

calculating the number and the first number weight of the first elements and the number and the second number weight of the second elements according to a weighting calculation method to obtain the number of weighted elements;

7. An apparatus for verifying accuracy of a determination, comprising:

a user description information and determination result acquisition unit configured to acquire user description information and a determination result corresponding to the user description information;

a description element extraction unit configured to extract at least one description element from the user description information;

the matching degree verification unit is configured to respectively determine the matching degree between each description element and the judgment result by using a preset verification model to obtain each actual matching value; the verification model is obtained by training based on the word frequency between the description elements and the judgment result and the inverse text frequency index TF-IDF relation;

a first element and judgment accuracy determination unit configured to determine, as a first element, a description element in which the actual matching value exceeds a preset matching value, and determine accuracy of the judgment result based on the number of the first elements.

8. The apparatus of claim 7, further comprising:

a second element determination unit configured to determine a description element that coincides with any of the knowledge elements as a second element; and

the first element and judgment accuracy determination unit includes:

a two-factor discrimination result accuracy determining subunit configured to determine the accuracy of the determination result according to the number of the first factor and the second factor.

9. The apparatus of claim 8, wherein the second element determination unit is further configured to:

obtaining the confidence of each knowledge element;

10. The apparatus of claim 8, wherein the two-factor discrimination result accuracy determination subunit is further configured to:

11. The apparatus of claim 8, further comprising:

a knowledge element supplementing unit configured to add a first element different from any of the knowledge elements as a supplementing element to the knowledge graph in response to the presence of the first element different from any of the knowledge elements.

12. The apparatus according to any one of claims 8 to 11, wherein the two-factor discrimination result accuracy determination subunit is configured to:

13. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method for verifying the accuracy of a determination of any one of claims 1-6.

14. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method for verifying the accuracy of determination results recited in any one of claims 1-6.