CN114706886A

CN114706886A - Evaluation method and device, computer equipment and storage medium

Info

Publication number: CN114706886A
Application number: CN202210284680.9A
Authority: CN
Inventors: 黄庭峰; 肖敏; 彭晶
Original assignee: Ping An Life Insurance Company of China Ltd
Current assignee: Ping An Life Insurance Company of China Ltd
Priority date: 2022-03-22
Filing date: 2022-03-22
Publication date: 2022-07-05

Abstract

The embodiment relates to the technical field of image recognition, in particular to an evaluation method and device, computer equipment and a storage medium. Reading original marking information of a marking data file in a test environment; creating a test file according to the labeled data file; the test file comprises test marking information, and the test marking information is obtained by removing field data values from original marking information; obtaining a test result data value according to the read address in the test file; adding the test result data value to the test file to obtain a test result file; merging the marked data file and the test result file to obtain a target file; in a computing environment, comparing the obtained test result data value with the field data value according to the position information to obtain a comparison result; and calculating according to the comparison result to obtain the recall rate and the accuracy rate. The technical method of the embodiment of the application can improve the evaluation efficiency of the recognition test algorithm.

Description

Evaluation method and device, computer equipment and storage medium

Technical Field

The invention relates to the technical field of image recognition, in particular to an evaluation method and device, computer equipment and a storage medium.

Background

At present, content in an image is identified by methods such as OCR, NLP, etc., and a data value obtained by identification is compared with an annotated data value, so as to obtain a comparison result of a corresponding image, and then, the performance of an image recognition algorithm is evaluated according to the comparison result.

In the method, the evaluation method of the performance of the image recognition algorithm comprises a recognition test process and a comparison process. Therefore, when the labeled data value is adjusted or modified, the whole evaluation method needs to be repeatedly executed from the beginning of the identification test process, thereby affecting the evaluation efficiency of the performance of the image identification algorithm.

Disclosure of Invention

The embodiment of the application mainly aims to provide an evaluation method and device, computer equipment and a storage medium, which can improve the evaluation efficiency of a recognition test algorithm.

In order to achieve the above object, a first aspect of the embodiments of the present application provides an evaluation method, including:

reading original marking information of a marking data file in a test environment; the method comprises the steps that the label data file is labeled with original label information of at least two target insurance policies, the original label information comprises a reading address, a classification field and at least two label fields of each target insurance policy, and each label field comprises position information and a field data value;

creating a test file according to the labeled data file; the test file comprises test marking information, and the test marking information is obtained by removing the field data value from the original marking information;

obtaining a test result data value according to the read address in the test file; the test result data value is obtained by carrying out policy identification test on the test file, and the test result data value is used for representing the identification result of the target policy;

adding the test result data value to the test file to obtain a test result file;

merging the marked data file and the test result file to obtain a target file;

in a computing environment, acquiring the test result data value and the field data value in the target file, and comparing the acquired test result data value with the field data value according to the position information to obtain a comparison result;

calculating according to the comparison result to obtain a recall rate and an accuracy rate; wherein the recall rate and the accuracy rate are used to evaluate the policy identification test.

In some embodiments, the classification fields include primary fields and secondary fields, each of the primary fields corresponding to at least one of the secondary fields, each of the secondary fields corresponding to one of the field data values; the first-level field carries a reading identifier;

the obtaining of the test result data value according to the read address in the test file includes:

reading the reading identification of the primary field in the test file;

and performing policy identification test on the test file according to the reading identification and the reading address to obtain the test result data value corresponding to the primary field.

In some embodiments, the secondary field carries an alignment identifier;

comparing the obtained test result data value with the field data value according to the position information to obtain a comparison result, wherein the comparison result comprises:

reading the comparison identification of the secondary field in the target file, and reading the reading identification of the primary field in the target file;

comparing the test result data value with the field data value according to the comparison identifier, the reading identifier and the position information to obtain a comparison result; the comparison result comprises any one of accurate identification, wrong identification, missed identification and multiple identification.

In some embodiments, the read identifier comprises a multi-attribute identifier, and the comparison identifier comprises a similarity comparison identifier; taking a plurality of field data values corresponding to the same primary field as a group of label groups, and taking a plurality of test result data values corresponding to the same primary field as a group of test groups;

the comparing the test result data value with the field data value according to the comparison identifier, the reading identifier and the position information to obtain the comparison result includes:

if the reading identifier is the multi-attribute identifier and the comparison identifier is the similarity comparison identifier, comparing a plurality of data values to be tested with corresponding field data values according to the position information to obtain a plurality of comparison similarity values; a plurality of test result data values corresponding to the same type of the secondary fields are used as the data values to be tested;

comparing the comparison similarity values to obtain a maximum similarity value;

if the maximum similarity value is larger than a preset first threshold value, the identification is accurate according to the comparison result; taking the field data value corresponding to the maximum similarity value as a first target marking data value, and taking the data value to be tested corresponding to the maximum similarity value as a first target test data value;

acquiring a first sub-annotation group corresponding to the first target annotation data value, and acquiring a second target annotation data value according to the first sub-annotation group and the first target annotation data value; the second target labeling data value and the first target labeling data value correspond to the same first sub-labeling group;

acquiring a first sub-test group corresponding to the first target test data value, and acquiring a second target test data value according to the first sub-test group and the first target test data value; wherein the second target test data value and the first target test data value correspond to the same first sub-test group;

and comparing the second target test data value with the second target marking data value according to the position information to obtain the comparison result.

In some embodiments, the comparing the test result data value with the field data value according to the comparison identifier, the reading identifier and the location information to obtain the comparison result further includes:

if the maximum similarity value is larger than a preset second threshold value, the comparison result is the identification error; the second threshold is smaller than the first threshold, the field data value corresponding to the maximum similarity value is used as a third target marking data value, and the data value to be tested corresponding to the maximum similarity value is used as a third target test data value;

acquiring a second sub-annotation group corresponding to the third target annotation data value, and acquiring a fourth target annotation data value according to the second sub-annotation group and the third target annotation data value; the fourth target annotation data value and the third target annotation data value correspond to the same second sub-annotation group;

acquiring a second sub-test group corresponding to the third target test data value, and acquiring a fourth target test data value according to the second sub-test group and the third target test data value; wherein the fourth target test data value and the third target test data value correspond to the same second sub-test group;

and comparing the fourth target test data value with the fourth target labeling data value according to the position information to obtain the comparison result.

In some embodiments, the comparing the test result data value with the field data value according to the comparison identifier, the read identifier, and the location information to obtain the comparison result further includes:

if the maximum similarity value is smaller than the second threshold value, the comparison result is the identification error; taking the field data value corresponding to the maximum similarity value as a fifth target marking data value, and taking the data value to be tested corresponding to the maximum similarity value as a fifth target test data value;

acquiring a third sub-label group corresponding to the fifth target test data value, and acquiring a label group to be tested according to the third sub-label group; the to-be-detected labeling group and the third sub-labeling group correspond to the same class of the primary field;

acquiring a third sub-test group corresponding to the fifth target test data value, and acquiring a sixth target test data value according to the third sub-test group and the fifth target test data value; wherein the sixth target test data value and the fifth target test data value correspond to the same third sub-test group;

traversing and comparing the sixth target test data value with the field data value in the to-be-tested labeling group according to the position information to obtain a maximum traversal similarity value;

and obtaining the comparison result according to the maximum traversal similarity value.

In some embodiments, the calculating a recall ratio and an accuracy ratio according to the comparison result includes:

acquiring the comparison result as the accurate identification number with accurate identification, the comparison result as the identification error number with error identification, the comparison result as the missing identification number with missing identification, and the identification result as the multi-identification number with multi-identification;

calculating to obtain the recall rate according to the accurate identification number, the error identification number and the missed identification number;

and calculating the accuracy rate according to the identification accurate quantity, the identification error quantity and the multi-identification quantity.

In order to achieve the above object, a second aspect of the embodiments of the present application provides an evaluation apparatus, including:

the test execution module is used for executing in a test environment;

reading original marking information of a marking data file; the method comprises the steps that the label data file is labeled with original label information of at least two target insurance policies, the original label information comprises a reading address, a classification field and at least two label fields of each target insurance policy, and each label field comprises position information and a field data value;

merging the marked data file and the test result file to obtain a target file;

a computing module for execution in a computing environment;

acquiring the test result data value and the field data value in the target file, and comparing the acquired test result data value with the field data value according to the position information to obtain a comparison result;

calculating according to the comparison result to obtain a recall rate and an accuracy rate; and evaluating the recognition result of the target insurance policy according to the recall rate and the accuracy rate.

To achieve the above object, a third aspect of the embodiments of the present disclosure provides a computer device, including a memory and a processor, where the memory stores therein a computer program, and the processor is configured to execute, when executed by the processor:

the method of evaluating according to the first aspect.

To achieve the above object, a fourth aspect of the embodiments of the present application further provides a storage medium, where a computer readable storage medium stores a computer program, and when the computer program is executed by a computer, the computer is configured to perform:

the method of evaluating according to the first aspect.

According to the evaluation method and device, the computer equipment and the storage medium, the policy identification test is carried out on the target policy in the test environment, and the test result data value corresponding to each target policy is obtained. And comparing the test result data value with the field data value in the computing environment to obtain the recall rate and the accuracy rate for evaluating the policy identification test. Therefore, the evaluation method provided by the embodiment of the application respectively sets the identification test flow and the comparison flow in two independent environments, so that when the field data value is adjusted, the comparison flow set in the computing environment is only required to be executed again, thereby avoiding the operation of repeatedly executing the identification test flow and the comparison flow in the related technology, and further improving the evaluation efficiency of the policy identification test.

Drawings

FIG. 1 is a schematic flow chart of an evaluation method according to an embodiment of the present application;

FIG. 2 is a schematic diagram of an embodiment of the present application for marking a data file;

FIG. 3 is a flowchart illustrating a specific method of step S130 according to the embodiment of the present application;

FIG. 4 is a diagram illustrating a correspondence between a primary field and a secondary field in an embodiment of the present application;

FIG. 5 is a flowchart illustrating a specific method of step S160 according to the embodiment of the present invention;

FIG. 6A is another schematic diagram of a markup data file according to an embodiment of the present application;

FIG. 6B is a diagram of a test result file according to an embodiment of the present application;

FIG. 7 is a flowchart illustrating a specific method of step S162 according to the present embodiment;

FIG. 8 is another flowchart illustrating a specific method of step S162 according to an embodiment of the present application;

FIG. 9 is another flowchart illustrating a specific method of step S162 according to an embodiment of the present application;

FIG. 10 is another flowchart illustrating a specific method of step S162 according to an embodiment of the present application;

FIG. 11 is a schematic view of an evaluation device according to an embodiment of the present application;

fig. 12 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of and not restrictive on the broad application.

It should be noted that although functional blocks are partitioned in a schematic diagram of an apparatus and a logical order is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the partitioning of blocks in the apparatus or the order in the flowchart. The terms first, second and the like in the description and in the claims, and the drawings described above, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the application.

First, several terms referred to in the present application are resolved:

optical Character Recognition (OCR): OCR refers to a process in which an electronic device examines characters printed on paper, determines their shapes by detecting dark and light patterns, and then translates the shapes into corresponding characters using a character recognition method. Correspondingly, OCR character recognition refers to a process of directly converting the character content on an image or a photo into an editable text by using an OCR technology.

Natural Language Processing (NLP): NLP uses computer to process, understand and use human language (such as chinese, english, etc.), and belongs to a branch of artificial intelligence, which is a cross discipline between computer science and linguistics, also commonly called computational linguistics. Natural language processing includes parsing, semantic analysis, discourse understanding, and the like. Natural language processing is commonly used in the technical fields of machine translation, character recognition of handwriting and print, speech recognition and text-to-speech conversion, information retrieval, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, and relates to data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, linguistic research related to language calculation, and the like, which are related to language processing.

Artificial Intelligence (AI): is a new technical science for researching and developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produces a new intelligent machine that can react in a manner similar to human intelligence, and research in this field includes robotics, language recognition, image recognition, natural language processing, and expert systems, among others. The artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results.

Recall, pointer to sample, probability that the positive class in the sample is predicted to be correct. The prediction result includes two possibilities, one is to predict the original positive class as a positive class (TP), and the other is to predict the original positive class as a negative class (FN). Therefore, the recall ratio R can be calculated by the following formula (1):

the accuracy rate, the probability that the original sample in which the prediction result is the positive type, for the prediction result of the pointer. The prediction result includes two possibilities, one is to predict the original positive class as a positive class (TP), and the other is to predict the original negative class as a positive class (FP). Therefore, the accuracy P can be calculated by the following formula (2):

taking the policy as an example, in the related art, OCR character recognition and NLP field extraction are performed on an image of the electronic policy or an image generated after the paper policy is shot, so as to obtain a test result data value corresponding to the policy content, thereby realizing statistical analysis of the policy data. The evaluation of the performance of the policy identification test algorithm is realized through two indexes, namely recall rate and accuracy rate.

The evaluation method and apparatus, the computer device, and the storage medium provided in the embodiments of the present application are specifically described in the following embodiments, and first, the evaluation method in the embodiments of the present application is described.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

The embodiment of the application provides an evaluation method, which relates to the technical field of artificial intelligence, in particular to the technical field of image recognition. The evaluation method provided by the embodiment of the application can be applied to a terminal, a server side and software running in the terminal or the server side. In some embodiments, the terminal may be a smartphone, tablet, laptop, desktop computer, smart watch, or the like; the server can be an independent server, and can also be a cloud server providing basic cloud computing services such as cloud service, a cloud database, cloud computing, a cloud function, cloud storage, Network service, cloud communication, middleware service, domain name service, security service, Content Delivery Network (CDN), big data and artificial intelligence platform and the like; the software may be an application or the like implementing the evaluation method, but is not limited to the above form.

The application is operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

Embodiments of the present application are further described below with reference to the accompanying drawings.

Referring to fig. 1, in a first aspect, the present application provides an evaluation method, which includes, but is not limited to, steps S110 to S170.

S110, reading original marking information of a marking data file in a test environment; the system comprises a labeling data file, a data processing unit and a data processing unit, wherein the labeling data file is labeled with original labeling information of at least two target insurance policies, the original labeling information comprises a reading address, a classification field and at least two labeling fields of each target insurance policy, and each labeling field comprises position information and a field data value;

specifically, the test environment is used to characterize a software environment, a hardware environment, etc. that performs the recognition test procedure on the target policy. And constructing an annotation data file according to at least two target insurance policies and the original annotation information corresponding to the target insurance policies. The target policy represents a test object to be identified, and the original labeling information includes a reading address of the target policy, field data values obtained through manual labeling and the like, and classification fields for classifying and identifying the field data values.

It will be appreciated that the annotation data file comprises a text format, a tabular format, and the like. In the embodiments of the present application and the following embodiments, the data file is labeled in an excel table format as an example for specific description. Referring to fig. 2, the read address of the target policy is set in the first column of the annotation data file, the classification field is set in the first row of the annotation data file, and the field data value (e.g., "xxx" in fig. 2) is set in the same row as the corresponding target policy. Therefore, the position information expressed in (row, column) form can be obtained by the target policy and the classification field corresponding to the field data value.

It is understood that when the annotation data file is in an excel table format, the read address can be a hyperlink corresponding to the target policy or a storage address of the target policy. Secondly, the target policy can be directly set in the annotation data file in an image format. The test object to be identified may further include a bill, a certificate, an invoice, a contract, and the like, and the embodiment of the present application is not particularly limited.

S120, creating a test file according to the marked data file; the test file comprises test marking information, and the test marking information is obtained by removing field data values from original marking information;

s130, obtaining a test result data value according to the reading address in the test file; the test result data value is obtained by carrying out policy identification test on the test file, and the test result data value is used for representing the identification result of the target policy;

s140, adding the data value of the test result to the test file to obtain a test result file;

s150, merging the marked data file and the test result file to obtain a target file;

specifically, a new worksheet is created by copying according to the read marked data file, and field data values in the new worksheet are deleted to form a test file. And calling the policy identification test interface, and testing the target policy in the test file according to the read address. And processing and analyzing the image by means of OCR character recognition, NLP field extraction and the like to obtain a test result data value for representing each target policy recognition result. And analyzing and classifying the test result data value, namely acquiring the policy content represented by the test result data value, thereby determining the classification field corresponding to the test result data value. And adding the test result data value to the corresponding target policy place line according to the classification field. And traversing and testing all target policy in the test file according to the method to obtain a test result file containing the identification result (namely the test result data value) of the target policy. And creating a target file containing the test result file and the annotation data file, wherein the target file comprises two worksheets.

It will be appreciated that process data values may also be added to the test results file in order to facilitate problem location. Wherein, the process data value comprises all test result data values, pixel position information and the like corresponding to the target policy data. For example, the process data value corresponding to each target policy is added to a cell in the last column of the test result file.

S160, in a computing environment, obtaining a test result data value and a field data value in a target file, and comparing the obtained test result data value with the field data value according to the position information to obtain a comparison result;

s170, calculating according to the comparison result to obtain a recall rate and an accuracy rate; wherein the recall rate and the accuracy rate are used for evaluating the policy identification test.

In particular, the computing environment is used to characterize a software environment, a hardware environment, etc., that performs the comparison process on the target policy. And reading the target file created according to the method, and acquiring the test result data value in the target file and the field data value with the same position information as the test result data value. And according to the attribute of the classification field corresponding to the test result data value, carrying out complete comparison or similarity comparison on the test result data value and the field data value to obtain a comparison result. Wherein, the complete comparison indicates that the test result data value and the field data value need to be completely matched; the similarity comparison indicates that the test result data value is allowed to be different from the field data value. And traversing and comparing all the test result data values in the target file according to the comparison method to obtain a plurality of comparison results. And respectively calculating according to the comparison results to obtain a recall rate and an accuracy rate, wherein the recall rate and the accuracy rate are used for evaluating an identification test algorithm executed in the identification test process.

It is understood that the comparison result may be displayed in the form of numerical value, font color, background shading, graphic identification, etc. Taking the background color as an example, in a computing environment, initializing cells corresponding to all test result data values in the test result file to fill a white background, and filling cells corresponding to the test result data values in the test result file with a preset color according to a comparison result. And counting the number of each preset color, and calculating according to the number to obtain the recall rate and the accuracy rate.

It will be appreciated that a new worksheet may also be created in the target file to form a test report file, i.e., the test report file includes three worksheets. And the new worksheet is used for recording the corresponding quantity of each preset color obtained through statistics and recording the recall rate and the accuracy rate obtained through calculation.

According to the evaluation method provided by the embodiment of the application, the policy identification test is carried out on the target policy in the test environment, so that the test result data value corresponding to each target policy is obtained. And comparing the test result data value with the field data value in the computing environment to obtain the recall rate and the accuracy rate for evaluating the policy identification test. Therefore, the evaluation method provided by the embodiment of the application respectively arranges the identification test flow and the comparison flow in two independent environments, so that when the field data value is adjusted, the comparison flow arranged in the computing environment is only required to be executed again, thereby avoiding the operation of repeatedly executing the identification test flow and the comparison flow in the related technology, and further improving the evaluation efficiency of the policy identification test.

Referring to FIG. 3, in some embodiments, the classification fields include primary fields and secondary fields, each primary field corresponding to at least one secondary field, and each secondary field corresponding to one of the field data values. Wherein the first-level field carries a read identification. Step S130 includes, but is not limited to, substeps S131 through S132.

S131, reading a reading identification of a first-level field in the test file;

s132, performing policy identification test on the test file according to the reading identification and the reading address to obtain a test result data value corresponding to the first-level field.

Specifically, the classification fields include a primary field for characterizing a large class category and a secondary field for characterizing a small class category. For example, referring to FIG. 4, the primary fields include policy base information, insurance product information, applicant information, insured information and beneficiary information. Taking the policy basic information as an example, the secondary field corresponding to the policy basic information includes the name of the insurance company, the policy number, and the policy validation date. By analogy, the secondary fields corresponding to the other primary fields can be obtained from fig. 4. It is to be understood that the primary and secondary fields in fig. 4 are exemplary only and are not limited thereto.

It will be appreciated that a target policy may include multiple categories of risk, multiple insureds, and multiple beneficiaries, and thus a target policy may correspond to multiple primary fields of the same category, such as three primary fields corresponding to insurance product information a, insurance product information B, and insurance product information C, wherein insurance product information a, insurance product information B, and insurance product information C belong to the same category of primary fields.

It will be appreciated that to determine the data values read and returned for the policy identification test on the target policy as one or more sets, the primary field is provided with a read identification characterizing the number of reads, for example: a unique attribute identification only for characterizing unique attributes and a multi-attribute identification multi for characterizing multi-attributes are set. Taking the example that the first-level field comprises insurance product information A multi, insurance product information B multi and insurance product information C multi, calling the policy identification test interface to traverse and acquire the image corresponding to the target policy according to the read address in the test file. And determining that three groups of test result data values respectively corresponding to the three primary fields need to be identified and returned from the corresponding target policy image by reading the reading identification of the primary fields. And respectively adding the three groups of test result data values to the cells corresponding to the first-level fields according to the identification sequence. Specifically, according to the identification sequence, the returned test result data values are analyzed and identified, and the secondary field corresponding to each test result data value in each returned group is determined. And according to the column information corresponding to the secondary field and the row information corresponding to the target policy, obtaining the position information of the test result data value, and adding the test result data value to the corresponding cell in the test file according to the position information, thereby obtaining the test result file.

Referring to FIG. 5, in some embodiments, the secondary field carries alignment identifiers. Step S160 includes, but is not limited to, substeps S161 through S162.

S161, reading the comparison identification of the secondary field in the target file, and reading the reading identification of the primary field in the target file;

s162, comparing the data value of the test result with the data value of the field according to the comparison identifier, the reading identifier and the position information to obtain a comparison result; the comparison result comprises any one of accurate identification, dislocation identification, missing identification and multiple identification.

Specifically, in a computing environment, a target test file is called to obtain test result data values and field data values in the target test file. And reading the comparison identification of the secondary field corresponding to the test result data value according to the position information. It is understood that the alignment identifier is used to characterize the alignment of the corresponding test result data values, for example: according to the characteristics of the secondary fields, a match identifier for representing complete comparison and a similar identifier for representing similarity comparison are set for each secondary field. And comparing the test result data value with the same position information with the field data value according to the acquired comparison identification to obtain any one of four comparison results of identification preparation, identification error, missing identification and multiple identification.

Wherein, the complete comparison indicates that the test result data value and the field data value need to be completely matched. Taking the second-level field policy number as an example, when the policy number of the target policy 1 corresponding to the second-level field in the labeled data file is completely consistent with the policy number of the target policy 1 corresponding to the second-level field in the test result file, determining that the comparison result is accurate to identify; and when the two policy numbers are not completely consistent, determining that the comparison result is an identification error.

The similarity comparison indicates that the test result data value is allowed to be different from the field data value. Taking the name similar of the second-level field insurance product as an example, if the field data value of the target policy 1 corresponding to the second-level field in the labeled data file is 'safe major disease insurance (platinum version)', and the test result data value of the target policy 1 corresponding to the second-level field in the test result file is 'safe major disease insurance (platinum),', acquiring the similarity value between the field data value and the test result data value, and when the similarity value is greater than a preset threshold value, determining that the comparison result is accurate in identification; and when the similarity value is smaller than a preset threshold value, determining that the comparison result is an identification error.

It can be understood that, in the identification accurate representation labeling data file and the test result file, both the field data value and the test result data value corresponding to the position information are non-null, and belong to the identification accurate condition determined by complete comparison or similarity comparison. The identification error indicates that the field data value and the test result data value corresponding to the position information in the labeling data file and the test result file are both non-null, but belong to the identification error condition determined by complete comparison or similarity comparison. The missing identification indicates that the field data value exists in the label data file, but the cell having the same position information as the field data value in the test result file is empty. The multiple identifications indicate that a certain cell in the label data file is empty (i.e. no field data value belonging to the secondary field corresponding to the cell is included in the target policy), but the cell having the same location information as the cell in the test result file is not empty (i.e. the cell corresponding to the location information in the test result file has a test result data value).

It is understood that when the read identifier of the primary field is a multi-attribute identifier, it indicates that there are multiple primary fields of the same category in the target file. Therefore, in order to avoid the phenomenon that the test result data value is added to other first-level fields in the same category, the test result data values corresponding to the multi-attribute first-level fields are dynamically compared, so that the accuracy of the comparison result is improved, and the accuracy of the policy identification test evaluation is further improved. Hereinafter, the dynamic alignment will be specifically described with reference to the description of the above embodiments.

Referring to fig. 6A and 6B, taking the target file including three primary fields of insurance product information a, insurance product information B, and insurance product information C as an example, the insurance product information a corresponds to six secondary fields of insurance product name a, insurance period a, basic allowance a, premium a, payment mode a, and payment period a. Similarly, the insurance product information B and the insurance product information C correspond to six secondary fields respectively. In some embodiments, a plurality of field data values corresponding to the same primary field are used as a set of label group, a plurality of test result data values corresponding to the same primary field are used as a set of test group, that is, six field data values corresponding to the insurance product information a are used as a set of label group 110, and six test result data values corresponding to the insurance product information a are used as a set of test group 210. It is to be understood that the insurance product information B can be divided into a set of label groups 120 and a set of test groups 220 with reference to the above method, and the insurance product information C can be divided into a set of label groups 130 and a set of test groups 230 with reference to the above method. Therefore, in the target test file, the first-level field of the insurance product information category corresponds to three test groups and three label groups.

Referring to fig. 7, in some embodiments, step S162 includes, but is not limited to, sub-steps S201 through S206.

S201, if the read identifier is a multi-attribute identifier and the comparison identifier is a similarity comparison identifier, comparing a plurality of to-be-detected data values with corresponding field data values according to position information to obtain a plurality of comparison similarity values; taking a plurality of test result data values corresponding to the same type of secondary fields as to-be-tested data values;

specifically, in a computing environment, the target file obtained by the method is called, and the reading identifier of the primary field and the comparison identifier of the secondary field in the target file are read. And if the reading identification of the primary field is the multi-attribute identification and the comparison identification of the secondary field is the similarity comparison identification, reading a test result data value in each group of test groups and taking the test result data value as the data value to be tested. It can be understood that a plurality of data values to be tested correspond to the same type of secondary field, for example: the test result data value corresponding to the second-level field insurance product name a is read from the test group 210, the test result data value corresponding to the second-level field insurance product name B is read from the test group 220, the test result data value corresponding to the second-level field insurance product name C is read from the test group 230, and all the three test result data values are used as data values to be tested. And respectively comparing the similarity of the three data values to be detected with the field data values with the same position information to obtain three comparison similarity values.

S202, comparing the comparison similarity values to obtain a maximum similarity value;

s203, if the maximum similarity value is larger than a preset first threshold value, the comparison result is accurate in identification; taking a field data value corresponding to the maximum similarity value as a first target marking data value, and taking a to-be-tested data value corresponding to the maximum similarity value as a first target test data value;

specifically, a plurality of comparison similarity values obtained according to the above method are compared, a maximum value among the plurality of comparison similarity values is determined, and the maximum value is taken as a maximum similarity value. And if the maximum similarity value is larger than a preset first threshold value, determining that the comparison result of the data value to be detected and the field data value corresponding to the maximum similarity is accurate in identification.

For example, in the data values to be tested corresponding to the insurance product name a, the insurance product name B, and the insurance product name C, the comparison similarity between the data value to be tested corresponding to the insurance product name a and the field data value is the largest. The preset first threshold value is 0.9, when the maximum similarity value is larger than 0.9, the similarity of the data value to be detected and the field data value corresponding to the maximum similarity value is high, and at the moment, the corresponding comparison result is determined to be accurate in identification. The data value to be tested corresponding to the insurance product name A is used as a first target test data value, and the field data value corresponding to the insurance product name A is used as a first target marking data value.

S204, acquiring a first sub-annotation group corresponding to the first target annotation data value, and acquiring a second target annotation data value according to the first sub-annotation group and the first target annotation data value; the second target labeling data value and the first target labeling data value correspond to the same first sub-labeling group;

s205, acquiring a first sub-test group corresponding to the first target test data value, and acquiring a second target test data value according to the first sub-test group and the first target test data value; the second target test data value and the first target test data value correspond to the same first sub-test group;

s206, comparing the second target test data value with the second target labeling data value according to the position information to obtain a comparison result.

Specifically, a label group corresponding to a first target label data value is used as a first sub label group, a test group corresponding to a first target test data value is used as a first sub test group, and the first sub label group and the first sub test group are respectively obtained to obtain data values of other fields in the first sub label group except the first target label data value and to obtain data values to be tested in the first sub test group except the first target test data value. And taking the data values of the rest fields as second target marking data values, and taking the data values to be tested of the rest fields as second target test data values. And comparing the second target test data value with the same position information with the second target label data value according to the comparison identification of the secondary field corresponding to the second target test data value to obtain a comparison result. And traversing all the read addresses in the target file according to the method to complete the comparison process of all the target policy.

For example, when the data value to be tested corresponding to the insurance product name a is used as a first target test data value, and the field data value corresponding to the insurance product name a is used as a first target labeled data value, it is determined that the primary field corresponding to the insurance product name a is insurance product information a, then the label group 110 corresponding to the insurance product information a is used as a first sub-label group, and the test group 210 corresponding to the insurance product information a is used as a first sub-test group. And acquiring the data values to be tested corresponding to the insurance period A, the basic insurance amount A, the insurance premium A, the payment mode A and the payment period A in the first sub-test group in the test result file, and taking the five data values to be tested as second target test data values. And acquiring field data values corresponding to the insurance period A, the basic insurance amount A, the insurance premium A, the payment mode A and the payment period A in the first sub-annotation group in the annotation data file, and taking the five field data values as second target annotation data values. And comparing the second target test data value with the corresponding second target marking data value according to the position information, thereby obtaining the comparison result of all the data values to be tested in the first sub-test group and the corresponding field data value.

It can be understood that, by the above comparison method, all field data values in the first sub-labeling group corresponding to the insurance product information a have participated in the comparison process, so that in the subsequent comparison process in which the insurance product information B and the insurance product information C correspond to the data values to be detected, the field data in the first sub-labeling group will not be acquired any more. The method for comparing the insurance product information B and the insurance product information C can refer to the above-mentioned comparison method, that is, comparing a plurality of data values to be tested in the insurance product information B with corresponding field data values, comparing a plurality of data values to be tested in the insurance product information C with corresponding field data values, and when it is determined that the data value to be tested corresponding to the maximum similarity is the test result data value in the insurance product information B (or the insurance product information C), completing the comparison process of all the test result data values corresponding to the insurance product information B (or the insurance product information C) with reference to the above-mentioned method.

Referring to fig. 8, in some embodiments, step S162 further includes, but is not limited to, sub-steps S207-S210.

S207, if the maximum similarity value is larger than a preset second threshold value, the comparison result is an identification error; the second threshold is smaller than the first threshold, the field data value corresponding to the maximum similarity value is used as a third target marking data value, and the data value to be tested corresponding to the maximum similarity value is used as a third target test data value;

specifically, a second threshold value with a value smaller than the first threshold value is set, and if the maximum similarity value among the multiple comparison similarity values obtained by the method is smaller than the first threshold value and larger than the second threshold value, the comparison result of the to-be-measured data value and the field data value corresponding to the maximum similarity is determined to be an identification error.

For example, in the data values to be measured corresponding to the insurance product name a, the insurance product name B, and the insurance product name C, the comparison similarity between the data value to be measured corresponding to the insurance product name a and the field data value is the largest. The first threshold value is preset to be 0.9, the second threshold value is preset to be 0.5, when the maximum similarity value is smaller than 0.9 and larger than 0.5, the similarity of the data value to be detected and the field data value corresponding to the maximum similarity value is low, and at the moment, the corresponding comparison result is determined to be an identification error. And taking the data value to be tested corresponding to the insurance product name A as a third target test data value, and taking the field data value corresponding to the insurance product name A as a third target marking data value.

S208, acquiring a second sub-annotation group corresponding to the third target annotation data value, and acquiring a fourth target annotation data value according to the second sub-annotation group and the third target annotation data value; the fourth target annotation data value and the third target annotation data value correspond to the same second sub-annotation group;

s209, acquiring a second sub-test group corresponding to the third target test data value, and acquiring a fourth target test data value according to the second sub-test group and the third target test data value; the fourth target test data value and the third target test data value correspond to the same second sub-test group;

s210, comparing the fourth target test data value with the fourth target marking data value according to the position information to obtain a comparison result.

Specifically, a label group corresponding to a third target label data value is used as a second sub label group, a test group corresponding to a third target test data value is used as a second sub test group, and the second sub label group and the second sub test group are respectively obtained to obtain data values of other fields in the second sub label group except the third target label data value and to obtain data values to be tested in the second sub test group except the third target test data value. And taking the data values of the rest fields as fourth target marking data values, and taking the data values to be tested of the rest fields as fourth target test data values. And comparing the fourth target test data value with the same position information with the fourth target marking data value according to the comparison identification of the secondary field corresponding to the third target test data value to obtain a comparison result. And traversing all the reading addresses in the target file according to the method to finish the comparison process of all the target insurance policies.

For example, when the data value to be tested corresponding to the insurance product name a is used as the third target test data value and the field data value corresponding to the insurance product name a is used as the third target labeled data value, the primary field corresponding to the insurance product name a is determined to be the insurance product information a, the labeled group 110 corresponding to the insurance product information a is used as the second sub-labeled group, and the test group 210 corresponding to the insurance product information a is used as the second sub-test group. And acquiring the data values to be tested corresponding to the insurance period A, the basic insurance amount A, the premium A, the payment mode A and the payment period A in the second sub-test group in the test result file, and taking the five data values to be tested as the fourth target test data value. And acquiring field data values corresponding to the insurance period A, the basic insurance amount A, the insurance premium A, the payment mode A and the payment period A in the second sub-annotation group in the annotation data file, and taking the five field data values as fourth target annotation data values. And comparing the fourth target test data value with the corresponding fourth target marking data value according to the position information, thereby obtaining the comparison result of all the data values to be tested in the second sub-test group and the corresponding field data value.

It can be understood that, by the above comparison method, all field data values in the second sub-labeling group corresponding to the insurance product information a have participated in the comparison process, so that in the subsequent comparison process in which the insurance product information B and the insurance product information C correspond to the data values to be detected, the field data in the second sub-labeling group will not be acquired any more. The method for comparing the insurance product information B and the insurance product information C may refer to the above-mentioned comparison method, that is, comparing a plurality of data values to be tested in the insurance product information B with corresponding field data values, comparing a plurality of data values to be tested in the insurance product information C with corresponding field data values, and when it is determined that the data value to be tested corresponding to the maximum similarity is the test result data value in the insurance product information B (or the insurance product information C), completing the comparison process of all the test result data values corresponding to the insurance product information B (or the insurance product information C) with reference to the above-mentioned methods of steps S201 to S206, or the methods of steps S207 to S210.

Referring to fig. 9, in some embodiments, step S162 further includes, but is not limited to, sub-steps S211-S215.

S211, if the maximum similarity value is smaller than a second threshold value, the comparison result is an identification error; taking the field data value corresponding to the maximum similarity value as a fifth target marking data value, and taking the data value to be tested corresponding to the maximum similarity value as a fifth target test data value;

specifically, if the maximum similarity value among the comparison similarity values obtained by the above method is smaller than the second threshold, the comparison result of the to-be-measured data value and the field data value corresponding to the maximum similarity is determined as an identification error.

For example, in the data values to be tested corresponding to the insurance product name a, the insurance product name B, and the insurance product name C, the comparison similarity between the data value to be tested corresponding to the insurance product name a and the field data value is the largest. And when the maximum similarity value is less than 0.5, indicating that the similarity of the data value to be detected and the field data value corresponding to the maximum similarity value is low, and determining that the corresponding comparison result is an identification error. And taking the data value to be tested corresponding to the insurance product name A as a fifth target test data value, and taking the field data value corresponding to the insurance product name A as a fifth target marking data value.

S212, a third sub-annotation group corresponding to the fifth target test data value is obtained, and an annotation group to be tested is obtained according to the third sub-annotation group; wherein, the to-be-detected labeling group and the third sub-labeling group correspond to the same class of first-level fields;

s213, acquiring a third sub-test group corresponding to the fifth target test data value, and acquiring a sixth target test data value according to the third sub-test group and the fifth target test data value; the sixth target test data value and the fifth target test data value correspond to the same third subtest group;

s214, traversing and comparing the sixth target test data value with the field data values in the to-be-tested labeling group according to the position information to obtain a maximum traversal similarity value;

and S215, obtaining a comparison result according to the maximum traversal similarity value.

Specifically, a label group corresponding to a fifth target label data value is used as a third sub label group, a test group corresponding to a fifth target test data value is used as a third sub test group, and the third sub label group and the third sub test group are respectively obtained to obtain the other label groups of the same class of first-class fields corresponding to the third sub label group and the other to-be-tested data values except the fifth target test data value in the third sub test group. And taking the rest of the marking groups as marking groups to be tested, and taking the rest of data values to be tested as sixth target test data values. And comparing the sixth target test data value with the field data values with the same position information in the to-be-detected labeling groups respectively according to the comparison identification of the secondary field corresponding to the third target test data value, taking a group of to-be-detected labeling groups with the highest comprehensive comparison similarity value as a target labeling group, and taking the comparison result of the sixth target test data value and the corresponding field data value in the target labeling group as a final comparison result. And traversing all the read addresses in the target file according to the method to complete the comparison process of all the target policy.

For example, when the data value to be tested corresponding to the insurance product name a is used as a fifth target test data value, and the field data value corresponding to the insurance product name a is used as a fifth target labeled data value, it is determined that the primary field corresponding to the insurance product name a is insurance product information a, the label group 110 corresponding to the insurance product information a is used as a third sub-label group, and the test group 210 corresponding to the insurance product information a is used as a third sub-test group. And acquiring the data values to be tested corresponding to the insurance period A, the basic insurance amount A, the premium A, the payment mode A and the payment period A in the third sub-test group in the test result file, and taking the five data values to be tested as the sixth target test data value. And acquiring other labeling groups (including insurance product information B and insurance product information C) which are of the same class of first-level fields as the insurance product information A, and taking the labeling group 120 corresponding to the insurance product information B and the labeling group 130 corresponding to the insurance product information C as to-be-detected labeling groups. According to the position information, comparing the sixth target test data value with the field data value in the corresponding label group 120 of the insurance product information B to obtain a comprehensive comparison similarity value 1; and comparing the sixth target test data value with the field data value in the corresponding label group 130 of the insurance product information C to obtain a comprehensive comparison similarity value 2. And if the comprehensive comparison similarity value 1 is greater than the comprehensive comparison similarity value 2, taking the to-be-detected labeling group corresponding to the insurance product information B as a target labeling group. And taking the comparison result of the sixth target test data value and the corresponding field data value in the target labeling group as the final comparison result of the sixth target test data value.

It can be understood that, by the above comparison method, in the third sub-label group corresponding to the insurance product information B, except for the field data value corresponding to the insurance product name B, the remaining field data values (including the insurance period B, the basic premium B, the payment method B, and the payment period B) all participate in the comparison process, so that in the subsequent comparison process of the data values to be tested corresponding to the insurance product information B and the insurance product information C, the field data corresponding to the insurance period B, the basic premium B, the payment method B, and the payment period B will not be acquired any more. The comparison method of the test result data values corresponding to the insurance product information B and the insurance product information C may refer to the comparison method described above, that is, comparing a plurality of data values to be tested in the insurance product information B with corresponding field data values, comparing a plurality of data values to be tested in the insurance product information C with corresponding field data values, and when it is determined that the data value to be tested corresponding to the maximum similarity is the test result data value in the insurance product information B (or the insurance product information C), completing the comparison process of all the test result data values corresponding to the insurance product information B (or the insurance product information C) with reference to the methods of steps S201 to S206, or steps S207 to S210, or steps S211 to S215.

It will be appreciated that the first and second thresholds are float values of either [0,1 ]. If the first threshold is set to 1, it indicates that the comparison mode between the corresponding test result data value and the field data value is a complete comparison.

Referring to fig. 10, in some embodiments, step S170 includes, but is not limited to, sub-steps S171 through S173.

S171, acquiring the accurate identification number with accurate identification as the comparison result, the identification error number with identification error as the comparison result, the missing identification number with missing identification as the comparison result and the multi-identification number with multi-identification as the identification result;

s172, calculating according to the accurate identification number, the error identification number and the missed identification number to obtain a recall rate;

and S173, calculating the accuracy rate according to the identification accurate number, the identification error number and the multi-identification number.

Specifically, the numbers of the comparison results in the target file, such as accurate identification, wrong identification, missed identification and multiple identification, are counted respectively, so that the recall rate is calculated according to the formula (1) and the accuracy rate is calculated according to the formula (2). Wherein TP represents the recognition accuracy number, FN represents the sum of the recognition error number and the recognition missing data, and FP represents the sum of the recognized error number and the recognition missing data.

According to the evaluation method provided by the embodiment of the application, the policy identification test flow and the comparison flow are respectively arranged in two independent environments, so that the problem that the whole flow needs to be repeatedly executed when the label field is frequently modified in the related technology is solved, and the evaluation efficiency of the evaluation method is improved; the accuracy of two indexes, namely the recall rate and the accuracy rate, is improved by setting a dynamic comparison mode and a similarity comparison mode; through the formed label data file, the target file and the test report file, process data in the policy identification test flow and the comparison flow can be intuitively known, and further, the subsequent policy identification test algorithm can be conveniently modified or adjusted.

Referring to fig. 11, an embodiment of the present application further provides an evaluation apparatus, where the evaluation apparatus includes:

a test execution module 300 for executing in a test environment;

reading original marking information of a marking data file; the system comprises a labeling data file, a data processing unit and a data processing unit, wherein the labeling data file is labeled with original labeling information of at least two target insurance policies, the original labeling information comprises a reading address, a classification field and at least two labeling fields of each target insurance policy, and each labeling field comprises position information and a field data value;

creating a test file according to the labeled data file; the test file comprises test marking information, and the test marking information is obtained by removing field data values from original marking information;

merging the marked data file and the test result file to obtain a target file;

a computing module 400 for execution in a computing environment;

acquiring a test result data value and a field data value in a target file, and comparing the acquired test result data value with the field data value according to the position information to obtain a comparison result;

calculating according to the comparison result to obtain the recall rate and the accuracy rate; and the recall rate and the accuracy rate are used for evaluating the recognition result of the target policy.

It can be seen that the content in the above evaluation method embodiment is all applicable to the embodiment of the evaluation apparatus, the function specifically implemented by the embodiment of the evaluation apparatus is the same as that of the above evaluation method embodiment, and the beneficial effect achieved by the embodiment of the evaluation method is also the same as that achieved by the above evaluation method embodiment.

An embodiment of the present application further provides an electronic device, where the electronic device includes: the evaluation system comprises a memory, a processor, a program stored on the memory and capable of running on the processor, and a data bus for realizing connection communication between the processor and the memory, wherein the program realizes the evaluation method when being executed by the processor. The electronic equipment can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

Referring to fig. 12, fig. 12 illustrates a hardware structure of an electronic apparatus of another embodiment, the electronic apparatus including:

the processor 501 may be implemented by a general-purpose CPU (central processing unit), a microprocessor, an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits, and is configured to execute a relevant program to implement the technical solution provided in the embodiment of the present application;

the memory 502 may be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM). The memory 502 may store an operating system and other application programs, and when the technical solution provided by the embodiments of the present disclosure is implemented by software or firmware, the relevant program codes are stored in the memory 502, and the processor 501 calls to execute the evaluation method according to the embodiments of the present disclosure;

an input/output interface 503 for implementing information input and output;

the communication interface 504 is used for realizing communication interaction between the device and other devices, and can realize communication in a wired manner (for example, USB, network cable, etc.) or in a wireless manner (for example, mobile network, WIFI, bluetooth, etc.);

a bus 505 that transfers information between various components of the device (e.g., the processor 501, the memory 502, the input/output interface 503, and the communication interface 504);

wherein the processor 501, the memory 502, the input/output interface 503 and the communication interface 504 are communicatively connected to each other within the device via a bus 505.

The embodiment of the application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above evaluation method.

The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and these remote memories may be connected to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The embodiments described in the embodiments of the present application are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute limitations on the technical solutions provided in the embodiments of the present application, and it is obvious to those skilled in the art that the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems with the evolution of technologies and the emergence of new application scenarios.

It will be appreciated by those skilled in the art that the solutions shown in fig. 1 to 12 do not constitute a limitation of the embodiments of the present application, and may comprise more or less steps than those shown, or some steps may be combined, or different steps.

The above-described embodiments of the apparatus are merely illustrative, wherein the units illustrated as separate components may or may not be physically separate, i.e. may be located in one place, or may also be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.

One of ordinary skill in the art will appreciate that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof.

The terms "first," "second," "third," "fourth," and the like in the description of the application and the above-described figures, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the application described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" for describing an association relationship of associated objects, indicating that there may be three relationships, e.g., "a and/or B" may indicate: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.

In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the above-described units is only one type of logical functional division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes multiple instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method of the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing programs, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The preferred embodiments of the present application have been described above with reference to the accompanying drawings, and the scope of the claims of the embodiments of the present application is not limited thereto. Any modifications, equivalents, and improvements that may occur to those skilled in the art without departing from the scope and spirit of the embodiments of the present application are intended to be within the scope of the claims of the embodiments of the present application.

Claims

1. An evaluation method, comprising:

reading original marking information of a marking data file in a test environment; the marked data file is marked with original marked information of at least two target insurance policies, the original marked information comprises a reading address, a classification field and at least two marked fields of each target insurance policy, and each marked field comprises position information and a field data value;

obtaining a test result data value according to the reading address in the test file; the test result data value is obtained by carrying out policy identification test on the test file, and the test result data value is used for representing the identification result of the target policy;

merging the marked data file and the test result file to obtain a target file;

2. An evaluation method according to claim 1, wherein said classification fields comprise primary fields and secondary fields, each of said primary fields corresponding to at least one of said secondary fields, each of said secondary fields corresponding to one of said field data values; the first-level field carries a reading identifier;

reading the reading identification of the primary field in the test file;

3. The evaluation method according to claim 2, wherein the secondary field carries a comparison identifier;

4. The evaluation method according to claim 3, wherein the reading identifier comprises a multi-attribute identifier, and the comparison identifier comprises a similarity comparison identifier; taking a plurality of field data values corresponding to the same primary field as a group of label groups, and taking a plurality of test result data values corresponding to the same primary field as a group of test groups;

5. The evaluating method according to claim 4, wherein the comparing the test result data value with the field data value according to the comparison flag, the reading flag, and the position information to obtain the comparison result, further comprises:

and comparing the fourth target test data value with the fourth target marking data value according to the position information to obtain the comparison result.

6. The evaluating method according to claim 5, wherein the comparing the test result data value with the field data value according to the comparison flag, the reading flag, and the position information to obtain the comparison result, further comprises:

acquiring a third sub-annotation group corresponding to the fifth target test data value, and acquiring an annotation group to be tested according to the third sub-annotation group; the to-be-detected labeling group and the third sub-labeling group correspond to the same class of the primary field;

7. The evaluation method according to any one of claims 3 to 6, wherein the calculating a recall ratio and an accuracy ratio based on the comparison result comprises:

8. An evaluation device, comprising:

the test execution module is used for executing in a test environment;

merging the marked data file and the test result file to obtain a target file;

a computing module for execution in a computing environment;

9. A computer device, characterized in that the computer device comprises a memory and a processor, wherein the memory has stored therein a computer program, and the processor is adapted to perform, when the computer program is executed by the processor:

evaluation method according to any of claims 1 to 7.

10. A storage medium which is a computer-readable storage medium, wherein the computer-readable storage stores a computer program, and when the computer program is executed by a computer, the computer is configured to perform:

evaluation method according to any of claims 1 to 7.