CN113268596A - Verification method, device and equipment of department classification model and storage medium - Google Patents
Verification method, device and equipment of department classification model and storage medium Download PDFInfo
- Publication number
- CN113268596A CN113268596A CN202110564888.1A CN202110564888A CN113268596A CN 113268596 A CN113268596 A CN 113268596A CN 202110564888 A CN202110564888 A CN 202110564888A CN 113268596 A CN113268596 A CN 113268596A
- Authority
- CN
- China
- Prior art keywords
- text data
- preset
- department
- classification model
- department classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/35—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3325—Reformulation based on results of preceding query
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/335—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/20—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the management or administration of healthcare resources or facilities, e.g. managing hospital staff or surgery rooms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- Business, Economics & Management (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Epidemiology (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Primary Health Care (AREA)
- Public Health (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of artificial intelligence, and discloses a method, a device, equipment and a storage medium for verifying a department classification model, which are used for improving the identification accuracy of the department classification model and reducing the referral rate of departments. The method comprises the following steps: acquiring initial text data, wherein the initial text data comprises a plurality of pieces of inquiry information; marking the initial text data according to a preset rule to obtain marked text data, wherein the marked text data comprises positive text data and negative text data; screening the labeled text data based on the preset data category to obtain screened text data; calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model; and when the accuracy rate exceeds a preset value, determining that the preset department classification model passes verification.
Description
Technical Field
The invention relates to the field of scoring models, in particular to a verification method, a verification device, verification equipment and a storage medium of a department classification model.
Background
With the rapid development of internet technology, various intelligent devices and intelligent technologies are applied to various fields in life, and the requirements of people on life quality are higher and higher, wherein for the medical field, patients hope to find professional doctors quickly, and the guiding and diagnosing of the patients is an important link for seeing a doctor.
In the field of intelligent inquiry, a department classification model for intelligent diagnosis guidance is a more critical loop, so that the performance of the department classification model is a key of quality, in the existing scheme, online inquiry data is firstly pulled, then doctors label the data to obtain a standard data set, wherein 90% of the data is used as a training model, and 10% of the data is used as a verification model, but the verified department classification model has low accuracy, so that subsequent doctor allocation is influenced, higher referral rate is caused, the patient visit time and experience are influenced, and the service efficiency of doctors is reduced.
Disclosure of Invention
The invention provides a method, a device, equipment and a storage medium for verifying a department classification model, which are used for more accurately evaluating the department classification model and improving the identification accuracy of the department classification model, so that the referral rate of departments is reduced, and the time consumed by department classification is shortened.
The first aspect of the embodiments of the present invention provides a verification method for a department classification model, including: acquiring initial text data, wherein the initial text data comprises a plurality of pieces of inquiry information, and each piece of inquiry information comprises main complaint content, gender information and age information, wherein the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient; marking the initial text data according to a preset rule to obtain marked text data, wherein the marked text data comprises positive text data and negative text data; screening the labeled text data based on preset data categories to obtain screened text data; calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model; and when the accuracy exceeds a preset value, determining that the preset department classification model passes verification.
Optionally, in a first implementation manner of the first aspect of the embodiment of the present invention, the labeling the initial text data according to a preset rule to obtain labeled text data, where the labeled text data includes positive text data and negative text data, and the labeling includes: filtering the learned inquiry information in the initial text data to obtain transition text data, wherein the transition text data comprises the unlearned inquiry information; inputting the transition text data into a preset model test platform; and calling the preset model test platform to label according to a preset rule to obtain labeled text data, wherein the labeled text data comprises positive text data and negative text data.
Optionally, in a second implementation manner of the first aspect of the embodiment of the present invention, the invoking the preset model test platform to label according to a preset rule to obtain labeled text data, where the labeled text data includes positive text data and negative text data, and the method includes: calling the preset model test platform to label the subcategories and standard departments according to preset rules, and determining the subcategories corresponding to each piece of inquiry information and the corresponding department categories; determining a general category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the general category comprises a positive type and a negative type, and positive text data and negative text data are obtained; and taking the positive text data and the negative text data as the marked text data.
Optionally, in a third implementation manner of the first aspect of the embodiment of the present invention, the screening the labeled text data based on a preset data category to obtain screened text data includes: determining the text quantity of each seed type in the labeled text data based on the preset data type; and respectively selecting a preset number of inquiry information from the inquiry information corresponding to each seed type according to a preset proportion to obtain screened text data.
Optionally, in a fourth implementation manner of the first aspect of the embodiment of the present invention, the invoking a preset department classification model to process the screened text data to obtain an accuracy of the preset department classification model includes: inputting the screened text data into a preset department classification model, and determining index parameters corresponding to the screened text data, wherein the index parameters comprise recall ratio, precision ratio, accuracy ratio, false alarm probability and false alarm probability; reading a plurality of preset candidate departments and generating a plurality of confusion matrixes, wherein each confusion matrix corresponds to two different departments; calculating the macro-check precision ratio or the micro-check precision ratio corresponding to the screened text data according to the plurality of confusion matrices; and determining the macro or micro-checking precision as the accuracy of the preset department classification model.
Optionally, in a fifth implementation manner of the first aspect of the embodiment of the present invention, after determining that the preset department classification model passes verification, the method for verifying the department classification model further includes: calculating the F value of the preset department classification model; and judging whether the F value meets a preset requirement, and if so, determining that the preset department classification model passes secondary verification.
Optionally, in a sixth implementation manner of the first aspect of the embodiment of the present invention, after determining that the preset department classification model passes verification, the method for verifying the department classification model further includes: when the preset department classification model passes verification, calling the preset department classification model to identify target text data and generating candidate departments, wherein the candidate departments are predicted by the department classification model; judging whether the candidate department is matched with the preset standard department or not based on a preset standard department, determining the candidate department as a target department when the candidate department is matched with the preset standard department, and determining the preset standard department as the target department when the candidate department is not matched with the preset standard department.
A second aspect of the embodiments of the present invention provides a verification apparatus for department classification models, including: the system comprises an acquisition module, a query module and a query module, wherein the acquisition module is used for acquiring initial text data, the initial text data comprises a plurality of pieces of inquiry information, and each piece of inquiry information comprises main complaint content, gender information and age information, the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient; the marking module is used for marking the initial text data according to a preset rule to obtain marked text data, and the marked text data comprises positive text data and negative text data; the screening module is used for screening the labeled text data based on preset data categories to obtain screened text data; the processing module is used for calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model; and the determining module is used for determining that the preset department classification model passes verification when the accuracy exceeds a preset value.
Optionally, in a first implementation manner of the second aspect of the embodiment of the present invention, the tagging module includes: the filtering unit is used for filtering the inquiry information which is learned in the initial text data to obtain transition text data, and the transition text data comprises the inquiry information which is not learned; the input unit is used for inputting the transition text data into a preset model test platform; and the marking unit is used for calling the preset model test platform to mark according to a preset rule to obtain marked text data, and the marked text data comprises positive text data and negative text data.
Optionally, in a second implementation manner of the second aspect of the embodiment of the present invention, the labeling unit is specifically configured to: calling the preset model test platform to label the subcategories and standard departments according to preset rules, and determining the subcategories corresponding to each piece of inquiry information and the corresponding department categories; determining a general category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the general category comprises a positive type and a negative type, and positive text data and negative text data are obtained; and taking the positive text data and the negative text data as the marked text data.
Optionally, in a third implementation manner of the second aspect of the embodiment of the present invention, the screening module is specifically configured to:
determining the text quantity of each seed type in the labeled text data based on the preset data type;
and respectively selecting a preset number of inquiry information from the inquiry information corresponding to each seed type according to a preset proportion to obtain screened text data.
Optionally, in a fourth implementation manner of the second aspect of the embodiment of the present invention, the processing module is specifically configured to:
inputting the screened text data into a preset department classification model, and determining index parameters corresponding to the screened text data, wherein the index parameters comprise recall ratio, precision ratio, accuracy ratio, false alarm probability and false alarm probability;
reading a plurality of preset candidate departments and generating a plurality of confusion matrixes, wherein each confusion matrix corresponds to two different departments;
calculating the macro-check precision ratio or the micro-check precision ratio corresponding to the screened text data according to the plurality of confusion matrices;
and determining the macro or micro-checking precision as the accuracy of the preset department classification model.
Optionally, in a fifth implementation manner of the second aspect of the embodiment of the present invention, the verification apparatus for a department classification model further includes:
the calculation module is used for calculating the F value of the preset department classification model;
and the first judgment and determination module is used for judging whether the F value meets a preset requirement, and if so, determining that the preset department classification model passes secondary verification.
Optionally, in a sixth implementation manner of the second aspect of the embodiment of the present invention, the verification apparatus for a department classification model further includes:
the identification module is used for calling the preset department classification model to identify target text data when the preset department classification model passes verification so as to generate a candidate department, wherein the candidate department is predicted by the department classification model;
the second judgment and determination module is used for judging whether the candidate department is matched with the preset standard department or not based on the preset standard department, determining the candidate department as a target department when the candidate department is matched with the preset standard department, and determining the preset standard department as the target department when the candidate department is not matched with the preset standard department.
A third aspect of the embodiments of the present invention provides a verification device for department classification models, comprising a memory and at least one processor, wherein the memory stores instructions, and the memory and the at least one processor are interconnected through a line; the at least one processor invokes the instructions in the memory to cause the verification device of the department classification model to perform the verification method of the department classification model described above.
A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores instructions that, when executed by a processor, implement the steps of the verification method of a department classification model according to any one of the above embodiments.
According to the technical scheme provided by the embodiment of the invention, initial text data is obtained, the initial text data comprises a plurality of pieces of inquiry information, each piece of inquiry information comprises main complaint content, gender information and age information, wherein the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient; marking the initial text data according to a preset rule to obtain marked text data, wherein the marked text data comprises positive text data and negative text data; screening the labeled text data based on the preset data category to obtain screened text data; calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model; and when the accuracy rate exceeds a preset value, determining that the preset department classification model passes verification. According to the embodiment of the invention, the positive text data and the negative text data are screened according to a certain proportion to obtain the screened text data, the department classification model is verified based on the screened text data to obtain the verification result, the data accuracy of the training department classification model is improved, the identification accuracy of the produced department classification model is further improved, the referral rate of departments is reduced, the time consumed by department classification is reduced, and the doctor service efficiency is improved.
Drawings
FIG. 1 is a diagram of an embodiment of a verification method for a department classification model according to an embodiment of the present invention;
FIG. 2 is a diagram of another embodiment of a verification method for a department classification model according to an embodiment of the present invention;
FIG. 3 is a diagram of an embodiment of a verification apparatus for department classification models according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another embodiment of a verification apparatus for department classification models in an embodiment of the present invention;
fig. 5 is a schematic diagram of an embodiment of a verification device for a department classification model in an embodiment of the present invention.
Detailed Description
The invention provides a method, a device, equipment and a storage medium for verifying a department classification model, which are used for more accurately evaluating the department classification model and improving the identification accuracy of the department classification model, so that the referral rate of departments is reduced, and the time consumed by department classification is shortened.
In order to make the technical field of the invention better understand the scheme of the invention, the embodiment of the invention will be described in conjunction with the attached drawings in the embodiment of the invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Referring to fig. 1, a flowchart of a verification method for a department classification model according to an embodiment of the present invention specifically includes:
101. the method comprises the steps of obtaining initial text data, wherein the initial text data comprise a plurality of pieces of inquiry information, each piece of inquiry information comprises main complaint content, gender information and age information, the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient.
The server acquires initial text data, wherein the initial text data comprises a plurality of pieces of inquiry information, each piece of inquiry information comprises main complaint content, gender information and age information, the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient. For example: the main complaints in an inquiry message are "seasonal rhinitis, what are the effective treatment methods? ", and the inquiry information includes male information and age information of 28; the main complaints in the other inquiry information are "how high triglycerides are, how do they get back? ", gender information is female, age information is 29; the other inquiry information includes the main complaints of chronic pharyngitis and long-term hoarseness. "sex information is male and age information is 41. For another example, the other inquiry message includes "baby yellow stool, foam" sex information is female, and age information is 40.
The main content in the inquiry information may be a sentence which has definite semantics and is matched with the gender information and the age information, for example, the main content in one inquiry information is "baby repeatedly burns for more than 10 months", the gender information is male, and the age information is 1; or sentences with ambiguous semantics, for example, a piece of inquiry information contains a main complaint of "how do i have a disease? ", gender information is female, age information is 10; or a sentence that does not match the sex information and the age information, for example: the main complaints in one piece of inquiry information are "i am irregular menstruation in this month", sex information is male, and age information is 30.
It is understood that the execution subject of the present invention may be a verification apparatus of a department classification model, and may also be a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
102. And marking the initial text data according to a preset rule to obtain marked text data, wherein the marked text data comprises positive text data and negative text data.
Specifically, the server filters the learned inquiry information in the initial text data to obtain transition text data, wherein the transition text data comprises the unlearned inquiry information; the server inputs the transition text data into a preset model test platform; and the server calls a preset model test platform to label according to a preset rule to obtain labeled text data, wherein the labeled text data comprises positive text data and negative text data.
The server calls a preset model test platform to label according to preset rules to obtain labeled text data, wherein the labeled text data comprise positive text data and negative text data, and the method specifically comprises the following steps: the server calls a preset model test platform to label the subcategories and the standard departments according to preset rules, and determines the subcategories corresponding to each piece of inquiry information and the corresponding department categories; the server determines a total category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the total category comprises a positive type and a negative type, and positive text data and negative text data are obtained; and the server takes the positive text data and the negative text data as the labeled text data.
103. And screening the labeled text data based on the preset data category to obtain the screened text data.
The server determines the text quantity of each seed type in the labeled text data based on the preset data type; and the server selects a preset number of inquiry information from the inquiry information corresponding to each seed category according to a preset proportion to obtain the screened text data.
For example, if there are 10 pieces of data in total in the initial text data, it is determined that the total number of required texts is 6 ten thousand according to a ratio of 60%, where the positive text data and the negative text data need to be filtered according to a ratio of 5:1, then it is determined that the number of the positive text data is 5 ten thousand, and the number of the negative text data is 1 ten thousand. The ratio of data in the algorithm capacity to data outside the algorithm capacity in the positive text data is 3:2, the ratio of data in the negative text data, which are not related to the main complaint content and are unclear, to data in the age information main complaint combination error is 2:3, so that the number of data in the algorithm capacity is 3 ten thousand, the number of data outside the algorithm capacity is 2 ten thousand, the number of data in the main complaint content and are not related to the main complaint content and are unclear is 4000, and the number of data in the age information main complaint combination error is 6000.
104. And calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model.
Specifically, the server inputs the screened text data into a preset department classification model, and determines index parameters corresponding to the screened text data, wherein the index parameters comprise recall ratio, precision ratio, accuracy ratio, false alarm probability and false alarm probability; the server reads a plurality of preset candidate departments and generates a plurality of confusion matrixes, wherein each confusion matrix corresponds to two different departments; the server calculates the macro-check precision ratio and the micro-check precision ratio corresponding to the screened text data according to the confusion matrixes; and the server determines the macro or micro-check precision as the accuracy of the preset department classification model.
The method comprises the steps of obtaining a labeling result and a prediction result of specific certain data in a preset department classification model, and comparing and calculating results, wherein the labeling result and the prediction result are multiple, and the algorithm prediction is also multiple, but the service is only the first one, so that the comparison logic is that the labeling result comprises the first result of the algorithm prediction and is passed.
105. And when the accuracy rate exceeds a preset value, determining that the preset department classification model passes verification.
And when the accuracy rate exceeds a preset value, the server determines that the preset department classification model passes verification.
It should be noted that, in the two classification indexes, the recall rate (i.e., recall ratio) and the precision rate (i.e., precision ratio) occur in pairs, and the false alarm occur in pairs, if the comparison of the weighted success number is successful, the recall rate and the precision rate are more convenient for the analysis of the algorithm research personnel, and the precision rate is more suitable for the measurement of the service.
According to the embodiment of the invention, the positive text data and the negative text data are screened according to a certain proportion to obtain the screened text data, the department classification model is verified based on the screened text data to obtain the verification result, the data accuracy of the training department classification model is improved, the identification accuracy of the produced department classification model is further improved, the referral rate of departments is reduced, and the time consumed by department classification is reduced.
Referring to fig. 2, another flowchart of the verification method of the department classification model according to the embodiment of the present invention specifically includes:
201. the method comprises the steps of obtaining initial text data, wherein the initial text data comprise a plurality of pieces of inquiry information, each piece of inquiry information comprises main complaint content, gender information and age information, the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient.
The server acquires initial text data, wherein the initial text data comprises a plurality of pieces of inquiry information, each piece of inquiry information comprises main complaint content, gender information and age information, the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient. For example: the main complaints in an inquiry message are "seasonal rhinitis, what are the effective treatment methods? ", and the inquiry information includes male information and age information of 28; the main complaints in the other inquiry information are "how high triglycerides are, how do they get back? ", gender information is female, age information is 29; the other inquiry information includes the main complaints of chronic pharyngitis and long-term hoarseness. "sex information is male and age information is 41. For another example, the other inquiry message includes "baby yellow stool, foam" sex information is female, and age information is 40.
The main content in the inquiry information may be a sentence which has definite semantics and is matched with the gender information and the age information, for example, the main content in one inquiry information is "baby repeatedly burns for more than 10 months", the gender information is male, and the age information is 1; or sentences with ambiguous semantics, for example, a piece of inquiry information contains a main complaint of "how do i have a disease? ", gender information is female, age information is 10; or a sentence that does not match the sex information and the age information, for example: the main complaints in one piece of inquiry information are "i am irregular menstruation in this month", sex information is male, and age information is 30.
It is understood that the execution subject of the present invention may be a verification apparatus of a department classification model, and may also be a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
202. And marking the initial text data according to a preset rule to obtain marked text data, wherein the marked text data comprises positive text data and negative text data.
Specifically, the server filters the learned inquiry information in the initial text data to obtain transition text data, wherein the transition text data comprises the unlearned inquiry information; the server inputs the transition text data into a preset model test platform; and the server calls a preset model test platform to label according to a preset rule to obtain labeled text data, wherein the labeled text data comprises positive text data and negative text data.
The server calls a preset model test platform to label according to preset rules to obtain labeled text data, wherein the labeled text data comprise positive text data and negative text data, and the method specifically comprises the following steps: the server calls a preset model test platform to label the subcategories and the standard departments according to preset rules, and determines the subcategories corresponding to each piece of inquiry information and the corresponding department categories; the server determines a total category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the total category comprises a positive type and a negative type, and positive text data and negative text data are obtained; and the server takes the positive text data and the negative text data as the labeled text data.
Optionally, the server calls a preset model test platform to label according to a preset rule to obtain labeled text data, where the labeled text data includes positive text data and negative text data, and the method includes:
(1) the server calls a preset model test platform to label the subcategories and the standard departments according to preset rules, and determines the subcategories corresponding to each piece of inquiry information and the corresponding department categories;
(2) the server determines a total category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the total category comprises a positive type and a negative type, and positive text data and negative text data are obtained;
specifically, the server performs feature extraction on each piece of inquiry information to obtain key data features of each piece of inquiry information, wherein the key data features comprise a prediction department, an actual standard department, chief complaint content, age and gender; the server determines corresponding sub-categories based on key data features of each piece of inquiry information, wherein the sub-categories comprise unclear main complaint content independent expression, incorrect combination of main complaint content of age and gender, inner algorithm capability and outer algorithm capability; the server determines that the general category to which the inquiry information corresponding to the irrelevant expression of the main complaint content or the inquiry information corresponding to the combination of the main complaint content and the errors belongs is negative text data, and determines that the inquiry information corresponding to the features in the algorithm capacity or the inquiry information corresponding to the outside of the algorithm capacity belongs is positive text data.
(3) And the server takes the positive text data and the negative text data as the labeled text data.
The server evaluates all text data based on a preset standard, and specifically, the server determines the obtained matching degree value as the support degree of the inquiry information by calculating the comprehensive matching degree of each piece of inquiry information with the predicted department, age and gender, determines the inquiry information with the support degree greater than a threshold value (namely high support degree) as positive text data, and determines the inquiry information with the support degree less than the threshold value (namely low support degree) as negative data. For example: the main complaint content is that the baby repeatedly burns for more than 10 months, the sex information is male, the age information is 1, the expression of the main complaint content needs deep semantic understanding, for example, that hallucinations often occur, insomnia and dreamful sleep occur, dreams come to heavy bleeding of uterus, the sex information is female, the age information is 21, and the server classifies the inquiry information as forward data. The data with low support degree generates referral, the data are analyzed to have the following conditions that the main complaint content is irrelevant to or unclear with the inquiry, for example, "just fall off, not good meaning", "the medicine can not be eaten when what is done", age and gender combine with the main complaint to see that the error exists, "baby yellow stool, has foam" gender women, age 40, and the data are classified as negative text data.
For example, the questionnaire ID number is 1354333502, the main complaint is "you good, what I want to consult is: asking for a question, the person who eat the collagen can also soak feet, the age is 10 years old, the gender is women, the forecast department is the traditional Chinese medicine health preserving department, the actually labeled department is the department of cardiology, and the subcategory of the inquiry information is within the algorithm capability; the inquiry bill has the ID number of 1354333448, the main complaint content is 'how to get back the pimple at the canthus', the age is 10 years old, the gender is female, the predicted department is ophthalmology, the actually marked department is dermatology, and the sub-category of the inquiry information is in the algorithm capability; the ID number of the inquiry sheet is 1354333993, the main complaint is 'vomiting and diarrhea and belly pain in lactation', the age is 29 years old, the sex is female, the forecast department is obstetrics and gynecology department, the actually labeled department is internal department, and the subcategory of the inquiry information is beyond algorithm capability; the ID number of the inquiry bill is 1354333392, the main complaint content is 'I just got out of line', the age is 21 years old, the gender is female, the forecast department is gynaecology and obstetrics, the actually marked department is the whole department, and the subcategory of the inquiry information is that the main complaint content is irrelevant and unclear; the ID number of the inquiry list is 1354333394, the main complaint content is 'I irregular menstruation in the month', the age is 30 years old, the gender is male, the forecast department is the whole department, the actual marked department is the whole department, and the sub-category of the inquiry information is the age-gender main complaint content combination error.
203. And screening the labeled text data based on the preset data category to obtain the screened text data.
The server determines the text quantity of each seed type in the labeled text data based on the preset data type; and the server selects a preset number of inquiry information from the inquiry information corresponding to each seed category according to a preset proportion to obtain the screened text data.
For example, if there are 10 pieces of data in total in the initial text data, it is determined that the total number of required texts is 6 ten thousand according to a ratio of 60%, where the positive text data and the negative text data need to be filtered according to a ratio of 5:1, then it is determined that the number of the positive text data is 5 ten thousand, and the number of the negative text data is 1 ten thousand. The ratio of data in the algorithm capacity to data outside the algorithm capacity in the positive text data is 3:2, the ratio of data in the negative text data, which are not related to the main complaint content and are unclear, to data in the age information main complaint combination error is 2:3, so that the number of data in the algorithm capacity is 3 ten thousand, the number of data outside the algorithm capacity is 2 ten thousand, the number of data in the main complaint content and are not related to the main complaint content and are unclear is 4000, and the number of data in the age information main complaint combination error is 6000.
204. And calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model.
Specifically, the server inputs the screened text data into a preset department classification model, and determines index parameters corresponding to the screened text data, wherein the index parameters comprise recall ratio, precision ratio, accuracy ratio, false alarm probability and false alarm probability; the server reads a plurality of preset candidate departments and generates a plurality of confusion matrixes, wherein each confusion matrix corresponds to two different departments; the server calculates the macro-check precision ratio or the micro-check precision ratio corresponding to the screened text data according to the confusion matrixes; and the server determines the macro or micro-check precision as the accuracy of the preset department classification model.
The method comprises the steps of obtaining a labeling result and a prediction result of specific certain data in a preset department classification model, and comparing and calculating results, wherein the labeling result and the prediction result are multiple, and the algorithm prediction is also multiple, but the service is only the first one, so that the comparison logic is that the labeling result comprises the first result of the algorithm prediction and is passed.
Optionally, the server calculates a macro-check precision ratio or a micro-check precision ratio corresponding to the screened text data according to the plurality of confusion matrices, and specifically includes:
the server calculates the recall ratio and precision ratio corresponding to each confusion matrix to obtain a plurality of recall ratios and a plurality of precision ratios; the server respectively averages the plurality of recall ratios and the plurality of precision ratios to obtain a recall ratio average value and a precision ratio average value; the server calculates according to a preset weight value, a recall ratio mean value and an accuracy ratio mean value, and determines the weighted recall ratio and the weighted accuracy ratio obtained through calculation as a macro recall ratio;
or the like, or, alternatively,
the server averages the quantity of each labeling type of the confusion matrixes to obtain the quantity of each labeling type, wherein the labeling types comprise positive case TP, negative case FP, negative case TN and positive case FN; and the server respectively calculates according to a preset recall ratio formula and a preset precision ratio formula based on the quantity of each labeling category, and determines the computed recall ratio and precision ratio as the micro-recall precision ratio.
It should be noted that, the macro audit precision calculation needs to obtain the weight of each two-class department, needs to provide the weight of the department, and is generally obtained by integrating the business weight visibility or the online data amount, and the default weight is "1/classification number". The macro-recall precision can analyze which classification has a problem, and for business, the weight can strengthen the distribution of the data set to the department level. And the micro-check precision rate is also regarded as the same weight for each piece of data in the data set (namely the initial text data), and the overall support effect of the data set is counted.
Wherein the recall ratio is used to indicate that all proportions marked as positive are truly correct, R ═ TP/(TP + FN); precision ratio (i.e., precision ratio) is used to indicate the true correct proportion of all algorithm predictions that are positive, P ═ TP/(TP + FP); f-value is used to indicate recall and fine weighted harmonic averaging, the coefficient is adjustable, typically recall fine is equally important using F1 value, F1 ═ 2PR/(R + P); accuracy is used to indicate the total weight of all predictions correct (positive-class negative-class), a ═ TP + TN)/(TP + TN + FN + FP); the false alarm probability is used for reflecting how many positive examples are missed, and MA is FN/(TP + FN) ═ 1-R, the false alarm probability is used for indicating false alarm, and FA is FP/(TP + FP) ═ 1-P.
205. And when the accuracy rate exceeds a preset value, determining that the preset department classification model passes verification.
And when the accuracy rate exceeds a preset value, the server determines that the preset department classification model passes verification.
It should be noted that, in the two classification indexes, the recall rate (i.e., recall ratio) and the precision rate (i.e., precision ratio) occur in pairs, and the false alarm occur in pairs, if the comparison of the weighted success number is successful, the recall rate and the precision rate are more convenient for the analysis of the algorithm research personnel, and the precision rate is more suitable for the measurement of the service.
206. And when the preset department classification model passes verification, calling the preset department classification model to identify the target text data to generate candidate departments, wherein the candidate departments are predicted by the department classification model.
And when the preset department classification model passes the verification, the server calls the preset department classification model to identify the target text data to generate a candidate department, wherein the candidate department is the department predicted by the department classification model.
207. And judging whether the candidate department is matched with the preset standard department or not based on the preset standard department, determining the candidate department as a target department when the candidate department is matched with the preset standard department, and determining the preset standard department as the target department when the candidate department is not matched with the preset standard department.
The server judges whether the candidate department is matched with the preset standard department or not based on the preset standard department, determines the candidate department as a target department when the candidate department is matched with the preset standard department, and determines the preset standard department as the target department when the candidate department is not matched with the preset standard department.
It should be noted that, the classification evaluation index and the business judgment criterion may be combined to determine whether the department classification model meets the requirement (i.e., matches). Whether the multi-category index is used for macro precision or micro precision determination or both is clear.
In the embodiment, the new evaluation strategy is more fit with the service index, the optimization effect of the model is objectively measured, whether the model can be on line is objectively evaluated, and the service support level after the model is released is ensured. It is to be appreciated that the validity of the optimization scheme can also be verified, continuing to optimize the data sets and the selection scheme of the data. It is understood that the successful verification of the model and the online verification may further include the following steps:
1. the branch office business requirements consider the requirements on online backflow, so that the backflow analysis of online data is facilitated after the functions are online; 2. before the functions are on-line, the statistical indexes and the backflow data fields are integrated, the BI report requirements are submitted, and the on-line report of the branch office business mainly counts the referral rate; 3. after the model is online, the online data is pulled in time for analysis, and the analysis content mainly includes the positive and negative behavior conditions of the user and marks whether the branch rooms are correct or not; 4. analyzing data of referral rate caused by department errors, filing the data into a test set, further analyzing data characteristics and classifying the data, and thinking about omission conditions in the data set to optimize a data set selection scheme; 5. data of referral rate caused by non-department misallocation are provided for an inquiry product team, the reason of the referral generation of the department is analyzed, and optimization such as business flow, interactive operation and the like in the aspect of products is considered.
Optionally, step 205 may be followed by:
the server calculates the F value of a preset department classification model; and the server judges whether the F value meets the preset requirement, and if so, the server determines that the preset department classification model passes secondary verification.
According to the embodiment of the invention, the forward text data and the negative text data are screened according to a certain proportion based on the data subcategory of the forward data and the data subcategory of the negative data to obtain the screened text data, and the department classification model is verified based on the screened text data to obtain the verification result, so that the data accuracy of the training department classification model is improved, the identification accuracy of the produced department classification model is further improved, the referral rate of departments is reduced, and the time consumed by department classification is reduced.
With reference to fig. 3, the method for verifying a department classification model in an embodiment of the present invention is described above, and a verification apparatus for a department classification model in an embodiment of the present invention is described below, where an embodiment of the verification apparatus for a department classification model in an embodiment of the present invention includes:
an obtaining module 301, configured to obtain initial text data, where the initial text data includes a plurality of pieces of inquiry information, and each piece of inquiry information includes a main complaint content, gender information, and age information, where the main complaint content is a question sentence queried by a user, and the inquiry information is historical visit data of a patient;
a labeling module 302, configured to label the initial text data according to a preset rule to obtain labeled text data, where the labeled text data includes positive text data and negative text data;
the screening module 303 is configured to screen the labeled text data based on a preset data category to obtain screened text data;
the processing module 304 is configured to invoke a preset department classification model to process the screened text data, so as to obtain an accuracy of the preset department classification model;
a determining module 305, configured to determine that the preset department classification model passes verification when the accuracy exceeds a preset value.
According to the embodiment of the invention, the positive text data and the negative text data are screened according to a certain proportion to obtain the screened text data, the department classification model is verified based on the screened text data to obtain the verification result, the data accuracy of the training department classification model is improved, the identification accuracy of the produced department classification model is further improved, the referral rate of departments is reduced, and the time consumed by department classification is reduced.
Referring to fig. 4, another embodiment of the verification apparatus for department classification model in the embodiment of the present invention includes:
an obtaining module 301, configured to obtain initial text data, where the initial text data includes a plurality of pieces of inquiry information, and each piece of inquiry information includes a main complaint content, gender information, and age information, where the main complaint content is a question sentence queried by a user, and the inquiry information is historical visit data of a patient;
a labeling module 302, configured to label the initial text data according to a preset rule to obtain labeled text data, where the labeled text data includes positive text data and negative text data;
the screening module 303 is configured to screen the labeled text data based on a preset data category to obtain screened text data;
the processing module 304 is configured to invoke a preset department classification model to process the screened text data, so as to obtain an accuracy of the preset department classification model;
a determining module 305, configured to determine that the preset department classification model passes verification when the accuracy exceeds a preset value.
Optionally, the labeling module 302 includes:
a filtering unit 3021, configured to filter the inquiry information that has been learned in the initial text data to obtain transition text data, where the transition text data includes the inquiry information that has not been learned;
an input unit 3022, configured to input the transition text data into a preset model test platform;
and the labeling unit 3023 is configured to call the preset model test platform to label according to a preset rule, so as to obtain labeled text data, where the labeled text data includes positive text data and negative text data.
Optionally, the labeling unit 3023 is specifically configured to:
calling the preset model test platform to label the subcategories and standard departments according to preset rules, and determining the subcategories corresponding to each piece of inquiry information and the corresponding department categories;
determining a general category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the general category comprises a positive type and a negative type, and positive text data and negative text data are obtained;
and taking the positive text data and the negative text data as the marked text data.
Optionally, the screening module 303 is specifically configured to:
determining the text quantity of each seed type in the labeled text data based on the preset data type;
and respectively selecting a preset number of inquiry information from the inquiry information corresponding to each seed type according to a preset proportion to obtain screened text data.
Optionally, the processing module 304 is specifically configured to:
inputting the screened text data into a preset department classification model, and determining index parameters corresponding to the screened text data, wherein the index parameters comprise recall ratio, precision ratio, accuracy ratio, false alarm probability and false alarm probability;
reading a plurality of preset candidate departments and generating a plurality of confusion matrixes, wherein each confusion matrix corresponds to two different departments;
calculating the macro-check precision ratio or the micro-check precision ratio corresponding to the screened text data according to the plurality of confusion matrices;
and determining the macro or micro-checking precision as the accuracy of the preset department classification model.
Optionally, the verification apparatus for department classification model further includes:
a calculating module 306, configured to calculate an F value of the preset department classification model;
a first determining module 307, configured to determine whether the F value meets a preset requirement, and if so, determine that the preset department classification model passes secondary verification.
Optionally, the verification apparatus for department classification model further includes:
the identification module 308 is configured to invoke the preset department classification model to identify target text data when the preset department classification model passes verification, so as to generate a candidate department, where the candidate department is a department predicted by the department classification model;
a second determination determining module 309, configured to determine whether the candidate department matches with the preset standard department based on a preset standard department, determine the candidate department as a target department when the candidate department matches with the preset standard department, and determine the preset standard department as a target department when the candidate department does not match with the preset standard department.
According to the embodiment of the invention, the forward text data and the negative text data are screened according to a certain proportion based on the data subcategory of the forward data and the data subcategory of the negative data to obtain the screened text data, and the department classification model is verified based on the screened text data to obtain the verification result, so that the data accuracy of the training department classification model is improved, the identification accuracy of the produced department classification model is further improved, the referral rate of departments is reduced, and the time consumed by department classification is reduced.
Fig. 3 to 4 describe the verification device of the department classification model in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the verification device of the department classification model in the embodiment of the present invention is described in detail from the perspective of hardware processing.
Fig. 5 is a schematic structural diagram of a verification device for department classification models, according to an embodiment of the present invention, the verification device 500 for department classification models may generate relatively large differences due to different configurations or performances, and may include one or more processors (CPUs) 510 (e.g., one or more processors) and a memory 520, one or more storage media 530 (e.g., one or more mass storage devices) storing applications 533 or data 532. Memory 520 and storage media 530 may be, among other things, transient or persistent storage. The program stored on the storage medium 530 may include one or more modules (not shown), each of which may include a series of instruction operations in the verification device 500 for a department classification model. Still further, the processor 510 may be configured to communicate with the storage medium 530 to execute a series of instruction operations in the storage medium 530 on the verification device 500 of the department classification model.
The verification device 500 for department classification models may also include one or more power supplies 540, one or more wired or wireless network interfaces 550, one or more input-output interfaces 560, and/or one or more operating systems 531, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the configuration of the verification device of the department classification model shown in fig. 5 does not constitute a limitation of the verification device of the department classification model, and may include more or fewer components than those shown, or some components in combination, or a different arrangement of components.
The present invention also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, and which may also be a volatile computer readable storage medium, having stored therein instructions, which, when run on a computer, cause the computer to perform the steps of the verification method of department classification models.
Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A verification method of a department classification model is characterized by comprising the following steps:
acquiring initial text data, wherein the initial text data comprises a plurality of pieces of inquiry information, and each piece of inquiry information comprises main complaint content, gender information and age information, wherein the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient;
marking the initial text data according to a preset rule to obtain marked text data, wherein the marked text data comprises positive text data and negative text data;
screening the labeled text data based on preset data categories to obtain screened text data;
calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model;
and when the accuracy exceeds a preset value, determining that the preset department classification model passes verification.
2. The method of claim 1, wherein the labeling the initial text data according to a preset rule to obtain labeled text data, the labeled text data including positive text data and negative text data, comprises:
filtering the learned inquiry information in the initial text data to obtain transition text data, wherein the transition text data comprises the unlearned inquiry information;
inputting the transition text data into a preset model test platform;
and calling the preset model test platform to label according to a preset rule to obtain labeled text data, wherein the labeled text data comprises positive text data and negative text data.
3. The method for verifying the department classification model according to claim 2, wherein the step of calling the preset model test platform to label the test platform according to a preset rule to obtain labeled text data, wherein the labeled text data comprises positive text data and negative text data, and comprises the steps of:
calling the preset model test platform to label the subcategories and standard departments according to preset rules, and determining the subcategories corresponding to each piece of inquiry information and the corresponding department categories;
determining a general category to which each piece of inquiry information belongs according to the sub-category corresponding to each piece of inquiry information and the corresponding department category, wherein the general category comprises a positive type and a negative type, and positive text data and negative text data are obtained;
and taking the positive text data and the negative text data as the marked text data.
4. The method for validating a department classification model according to claim 3, wherein the step of screening the labeled text data based on preset data categories to obtain screened text data comprises:
determining the text quantity of each seed type in the labeled text data based on the preset data type;
and respectively selecting a preset number of inquiry information from the inquiry information corresponding to each seed type according to a preset proportion to obtain screened text data.
5. The method for validating department classification model according to claim 1, wherein the step of calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model comprises the steps of:
inputting the screened text data into a preset department classification model, and determining index parameters corresponding to the screened text data, wherein the index parameters comprise recall ratio, precision ratio, accuracy ratio, false alarm probability and false alarm probability;
reading a plurality of preset candidate departments and generating a plurality of confusion matrixes, wherein each confusion matrix corresponds to two different departments;
calculating the macro-check precision ratio or the micro-check precision ratio corresponding to the screened text data according to the plurality of confusion matrices;
and determining the macro or micro-checking precision as the accuracy of the preset department classification model.
6. The method of validating department classification models according to claim 1, wherein after said determining that the preset department classification model is validated, the method further comprises:
calculating the F value of the preset department classification model;
and judging whether the F value meets a preset requirement, and if so, determining that the preset department classification model passes secondary verification.
7. The method for validating department classification models according to any one of claims 1-5, wherein after said determining that the preset department classification model is validated, the method for validating department classification models further comprises:
when the preset department classification model passes verification, calling the preset department classification model to identify target text data and generating candidate departments, wherein the candidate departments are predicted by the department classification model;
judging whether the candidate department is matched with the preset standard department or not based on a preset standard department, determining the candidate department as a target department when the candidate department is matched with the preset standard department, and determining the preset standard department as the target department when the candidate department is not matched with the preset standard department.
8. A verification apparatus for a department classification model, comprising:
the system comprises an acquisition module, a query module and a query module, wherein the acquisition module is used for acquiring initial text data, the initial text data comprises a plurality of pieces of inquiry information, and each piece of inquiry information comprises main complaint content, gender information and age information, the main complaint content is a question sentence inquired by a user, and the inquiry information is historical visit data of a patient;
the marking module is used for marking the initial text data according to a preset rule to obtain marked text data, and the marked text data comprises positive text data and negative text data;
the screening module is used for screening the labeled text data based on preset data categories to obtain screened text data;
the processing module is used for calling a preset department classification model to process the screened text data to obtain the accuracy of the preset department classification model;
and the determining module is used for determining that the preset department classification model passes verification when the accuracy exceeds a preset value.
9. A verification apparatus of a department classification model, comprising: a memory having instructions stored therein and at least one processor, the memory and the at least one processor interconnected by a line;
the at least one processor invoking the instructions in the memory to cause a verification device of the department classification model to perform a verification method of the department classification model of any of claims 1-7.
10. A computer-readable storage medium storing instructions which, when executed by a processor, implement a method of validating a department classification model according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110564888.1A CN113268596A (en) | 2021-05-24 | 2021-05-24 | Verification method, device and equipment of department classification model and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110564888.1A CN113268596A (en) | 2021-05-24 | 2021-05-24 | Verification method, device and equipment of department classification model and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113268596A true CN113268596A (en) | 2021-08-17 |
Family
ID=77232371
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110564888.1A Pending CN113268596A (en) | 2021-05-24 | 2021-05-24 | Verification method, device and equipment of department classification model and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113268596A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114240101A (en) * | 2021-12-02 | 2022-03-25 | 支付宝(杭州)信息技术有限公司 | Risk identification model verification method, device and equipment |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704869A (en) * | 2017-09-01 | 2018-02-16 | 厦门快商通科技股份有限公司 | A kind of corpus data methods of sampling and model training method |
| CN111126574A (en) * | 2019-12-30 | 2020-05-08 | 腾讯科技（深圳）有限公司 | Method and device for training machine learning model based on endoscopic image and storage medium |
| WO2020238053A1 (en) * | 2019-05-31 | 2020-12-03 | 平安科技（深圳）有限公司 | Neural grid model-based text data category recognition method and apparatus, nonvolatile readable storage medium, and computer device |
| CN112509682A (en) * | 2020-12-15 | 2021-03-16 | 康键信息技术（深圳）有限公司 | Text recognition-based inquiry method, device, equipment and storage medium |
-
2021
- 2021-05-24 CN CN202110564888.1A patent/CN113268596A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107704869A (en) * | 2017-09-01 | 2018-02-16 | 厦门快商通科技股份有限公司 | A kind of corpus data methods of sampling and model training method |
| WO2020238053A1 (en) * | 2019-05-31 | 2020-12-03 | 平安科技（深圳）有限公司 | Neural grid model-based text data category recognition method and apparatus, nonvolatile readable storage medium, and computer device |
| CN111126574A (en) * | 2019-12-30 | 2020-05-08 | 腾讯科技（深圳）有限公司 | Method and device for training machine learning model based on endoscopic image and storage medium |
| CN112509682A (en) * | 2020-12-15 | 2021-03-16 | 康键信息技术（深圳）有限公司 | Text recognition-based inquiry method, device, equipment and storage medium |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114240101A (en) * | 2021-12-02 | 2022-03-25 | 支付宝(杭州)信息技术有限公司 | Risk identification model verification method, device and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascular disease as exemplar | |
| Chandler et al. | Methodological standards for the conduct of new Cochrane Intervention Reviews | |
| Glare et al. | A systematic review of physicians' survival predictions in terminally ill cancer patients | |
| US8019624B2 (en) | Assessing athlete injuries | |
| Dakin et al. | “Yes”,“No” or “Yes, but”? Multinomial modelling of NICE decision-making | |
| Raymond | Missing data in evaluation research | |
| Fenton | Software measurement: A necessary scientific basis | |
| CN109346145B (en) | Method and system for actively monitoring adverse drug reactions | |
| US20120271612A1 (en) | Predictive modeling | |
| Nagendrababu et al. | Glossary for systematic reviews and meta‐analyses | |
| Grzybowski | Note on a new optimization based approach for estimating priority weights and related consistency index | |
| US20110166880A1 (en) | Method and System for Analyzing Health Related Data of Patients | |
| CN112927782B (en) | Heart health state early warning system based on text emotion analysis | |
| Hopkins et al. | Progressive statistics. | |
| Glance et al. | National Quality Forum guidelines for evaluating the scientific acceptability of risk-adjusted clinical outcome measures: a report from the National Quality Forum Scientific Methods Panel | |
| CN112509656A (en) | Grade evaluation method and device based on medical institution, computer equipment and medium | |
| CN115101160A (en) | Drug sales data mining and retrieving method and device | |
| CN113268596A (en) | Verification method, device and equipment of department classification model and storage medium | |
| CN113420946B (en) | News media evaluation method | |
| Conway et al. | Targets and the emergency medical system–intended and unintended consequences | |
| Tian et al. | An extended SECA-GDM method considering flexible linguistic scale optimization and its application in occupational health and safety risk assessment | |
| Bult et al. | Heterogeneity in the relationship between the standard-gamble utility measure and health-status dimensions | |
| CN111445993A (en) | Disease hierarchical relation processing method and device and electronic equipment | |
| CN111354456A (en) | Traditional Chinese medicine constitution type detection method and device | |
| CN110911014A (en) | Method and system for inquiring and analyzing repeated medication in outpatient service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |