US20190287685A1 - String classification apparatus and methods using artificial intelligence - Google Patents
String classification apparatus and methods using artificial intelligence Download PDFInfo
- Publication number
- US20190287685A1 US20190287685A1 US16/136,992 US201816136992A US2019287685A1 US 20190287685 A1 US20190287685 A1 US 20190287685A1 US 201816136992 A US201816136992 A US 201816136992A US 2019287685 A1 US2019287685 A1 US 2019287685A1
- Authority
- US
- United States
- Prior art keywords
- hpi
- string
- neural network
- examples
- hashes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G06F17/277—
-
- G06F17/278—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G06K9/6279—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G06N3/0481—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H10/00—ICT specially adapted for the handling or processing of patient-related medical or healthcare data
- G16H10/60—ICT specially adapted for the handling or processing of patient-related medical or healthcare data for patient-specific data, e.g. for electronic patient records
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients
Definitions
- This disclosure relates generally to managing electronic medical records and, more particularly, to methods and apparatus to classify medical data using artificial intelligence.
- a patient's electronic medical records are documentation of that patient's history of care and medical encounters stored in an electronic database.
- a healthcare professional usually takes a medical history of that patient.
- Aspects of a taking a medical history include asking questions to obtain a demographic information, chief complaint (CC), a history of the present illness (HPI), a review of systems (ROS) and past, family and/or social history (PFSH).
- the Center of Medicare and Medicaid (CMS) divides medical histories into four types, namely: problem focused, focus expanded problem, detailed and comprehensive. An amount of money reimbursed to a healthcare provider often depends on what type of medical history was taken during a medical encounter. A classification of medical history into these four types depends on information included in the gathered HPI, ROS and/or PFSH.
- CMS standards for medical history include:
- HPIs are narrative summaries compiled by a healthcare professional after identifying a patient's chief compliant (e.g., a reason for their visit). Classifying an HPI as “brief” or “extended” depends upon what information is obtained from the patient and recorded during the interview. The CMS has established a criterion such that an HPI is classified as “extended” when it contains four or more of the following elements: (1) location of problem, (2) quality of problem, (3) severity of problem, (4) duration of problem, (5) context of problem, (6) modifying factors, and (7) associated signs and symptoms. An HPI is classified as “brief” when it contains three or fewer elements. Historically, HPIs are classified by healthcare professionals during or after they are obtained or recorded.
- An example apparatus for classifying a string according to a binary classification system comprising: a processor to execute instructions to implement at least: a preprocessor to: tokenize the string into tokens based on a context associated with at least one section of the string; and convert the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context; a recurrent neural network including: an embedding layer to embed the hashes into dense vectors; a long short-term memory (LSTM) layer to convert the dense vectors into an activated output vector; and a fully connected layer to linearize the activated output vector into a binary output; and memory to store at least one of the string, the tensor or the binary output.
- a preprocessor to: tokenize the string into tokens based on a context associated with at least one section of the string; and convert the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context
- An example method includes receiving an HPI formatted as a string, the string including one or more words, the words organized in an order of sentences; tokenizing the one or more words into tokens based on a context associated with at least one of the one or more words; converting the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context; embedding each of the hashes into vectors; processing the vectors to classify the HPI as extended or brief based on a similarity to a set of classified HPIs; outputting a classification for the HPI; and modifying a medical support system with the HPI and the classification and to trigger an action with respect to the medical support system based on the classification.
- An example tangible machine readable medium comprising instructions, which when executed, cause a processor to at least tokenize a string into tokens based on a context associated with at least one section of the string; convert the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context; embed the hashes into dense vectors; convert the dense vectors into an activated output vector; linearize the activated output vector into a binary output; and store at least one of the string, the tensor or the binary output.
- FIG. 1 is an illustration of a medical data processing system in which the teachings of this disclosure may be implemented.
- FIG. 2 is a block diagram representative of the HPI classifier of FIG. 1 .
- FIG. 3 is an example data flow of an HPI through the HPI classifier of FIG. 2 .
- FIG. 4 is an example confusion table of a group of HPIs submitted to a trained HPI classifier.
- FIG. 5 is a process flow diagram that can be implemented to deploy a model to be used to process incoming patient data.
- FIG. 6 is an illustration of an example data flow and transformation of information flowing through components of the example system of FIG. 1 .
- FIG. 7 is a flowchart representative of machine readable instructions which can be executed to implement the HPI classifier of FIG. 2 .
- FIG. 8 is a flowchart representative of machine readable instructions which can be executed to preprocess input HPI data.
- FIG. 9 is a flowchart representative of machine readable instructions which can be executed to classify the preprocessed HPI data.
- FIG. 10 is a flowchart representative of machine readable instructions which can be executed to retrain the HPI model.
- FIG. 11 is a block diagram of an example processing platform structured to execute the instructions of FIGS. 7-10 to implement the systems and methods described herein including the HPI classifier and/or other components of FIG. 2 .
- automating the process for determining if an HPI is “extended” or “brief” becomes important to the maintenance and completeness of an EMR of a patient. Automating the process of classifying an HPI potentially allows an HPI to be gathered by a computer instead of a healthcare professional, thus allowing healthcare professionals to be performing more critical tasks. Additionally, empirical evidence suggests humans incorrectly classify HPIs at a relatively high and significant rate. Automating the classification process could potentially alleviate some of these potential issues. In some examples, this may lead to more accurate medical records for the patient and more accurate billing for the CMS and healthcare organizations. Furthermore, automatic classification of an HPI as “brief,” may encourage a healthcare professional to retake an HPI to ensure the HPI is “extended.”
- HPIs resist being easily classified by standard artificial intelligences (AI) and/or natural language processors (NLPs) for a number of reasons.
- AI artificial intelligences
- NLPs natural language processors
- HPIs are often recorded as a narrative which makes identifying which, if any, of the seven elements are present in a particular patient's HPI difficult.
- the writing style e.g., punctuation, abbreviations, word choice, sentence structure, etc.
- the narrative may contain misspelled words or incomplete sentences.
- the HPIs often contain high occurrences of medical terms, abbreviations and named entities which often have different meanings depending on the context.
- “pt” can refer to either “patient” or “physical therapy” depending on the context.
- Abbreviations such as “OSA” (obstructive sleep apnea), “PSA” (prostate specific antigen), etc. can be difficult to process.
- named entities such as “Dr. Smith”, “CPCA (California Primary Care Association)”, etc., can also be difficult to process.
- HPIs often contain extensive use of numbers with different semantic meanings. For examples, the phrases “last colonoscopy was 2009,” “the pain lasts 5 minutes,” and “Type 2 Diabetes” all contain numbers with different semantic meanings (e.g., a date, a duration and a classification of disease, respectively).
- the length (e.g., word count, number of sentences, etc.) of an HPI does not necessarily correlate with its classification.
- HPI is relatively long but would be classified as a brief HPI:
- the examples disclosed herein overcome the above obstacles and improve the technology of medical data processing by providing technologically improved systems and methods to normalize an input HPI and classify the HPI using a neural network tuned to process HPI information and generate a classification from the HPI information.
- the HPI is normalized with a natural language processor by tokenizing, lemmatizing, and replacing named entities and medical terms with standardized strings/predefined tags.
- the natural language processor randomly reorganizes the order of each sentence in the input to the HPI.
- the tokens are hashed into integers. In such examples, the integers are representative of an index of a sparse vector where each index represents a distinct word.
- the normalized HPI is classified with a neural network.
- the neural network is a three-layer neural network including an embedding layer, recurrent neural network layer, and fully connected layer.
- the recurrent neural network is a long short-term memory (LSTM) network.
- the three-layer neural network outputs a binary output (e.g., a binary classification, either “extended” or “brief” represented as 0 or 1, 1 or 0, etc.)
- the neural network outputs a vector including values corresponding to the presence of each HPI element in an input HPI.
- the output of the neural network can also include a determination of which bodily system(s) is/are discussed in the input HPI.
- the neural network is retrained when a certain number false labels and/or other feedback data are accrued.
- neural network refers to a computing system or other processor system that learns to perform a task by analyzing pre-classified training examples.
- Neural networks include a plurality of densely connected processing nodes inspired by the human brain.
- the nodes of a neural networks can be organized into layers in which data moves in the forward direction (e.g., data in the first layer moves into the second layer, data in the second layer moves into the third layer, etc.), for example, to drive one or more outputs based on one or more inputs via correlations (e.g., connections) represented by the nodes and their interconnections.
- Deep learning and/or machine learning can be implemented via a neural network to process incoming data to generate an output and benefit from feedback to improve its processing.
- a “recurrent neural network” or “RNN” is a type of neural network in which nodes or cells include loops to allow information to persist over time.
- the RNN can leverage reasoning about previous events to inform subsequent processing.
- a memory or other internal state is used to process input sequence(s) in an element-by-element process wherein an output for each element is dependent on the output of previous and/or other elements (e.g., a directed graph driving a sequence).
- LSTM networks are RNNs designed to handle long-term dependencies.
- LSTM networks are organized into cells and gates which interact to optimize the output of the network.
- Information from outside the processing of the current element e.g., information from previous elements
- gated cells These gates release information based on the weight of the gates, which are adjusted and optimized during the training phase of the AI.
- the nodes or cells in the network have storage and an associated stored state under control of the neural network to aid in establishing correlations and processing input data.
- FIG. 1 is an illustration of a system in which the teachings of this disclosure may be implemented.
- An example system 100 includes a data source 102 , an HPI classifier 104 , and medical support system 106 .
- the data source 102 outputs an unprocessed HPI 108 which is transferred to the both the HPI classifier 104 and the medical support system 106 .
- the HPI classifier 104 which is discussed below in greater detail in relation to FIG. 2 , classifies the HPI 108 and outputs an HPI classification 110 .
- the HPI classification 110 is a binary output of either “brief” or “extended.”
- the medical support system 106 links and stores the HPI 108 and the HPI classification 110 .
- the example data source 102 provides the HPI 108 to the HPI classifier 104 .
- the data source 102 may be database of previously collected and recorded HPIs.
- the data source 102 may be a text input (e.g., a keyboard, a speech to text processor, or a digital scanner with text recognition, etc.).
- the data source 102 is used by healthcare professionals, medical support staff, and/or patients to input the HPI.
- the data source 102 may be a computer terminal in which a healthcare professional records the patient's answers while conducting a medical history.
- the data source 102 may contain a user interface which issues prompts asking for particular inputs (e.g., “Where is the pain location?,” “How severe is the pain?,” “When do the symptoms occur?,” etc.).
- the data source 102 constructs the HPI 108 from the answers to the provided prompts.
- the example medical support system 106 is a digital database which contains the medical history of a patient and a summary of the medical encounters of the patient.
- the example medical support system 106 records both the HPI 108 and the HPI classification 110 .
- the example medical support system 106 can be any suitable medical system (e.g., an EMR, a medical billing system, etc.).
- the medical support system 106 may store the HPI 108 as a text string associated with the patient's related medical encounter.
- the HPI 108 may recorded in any suitable method (e.g., each word of the HPI is stored as separate string, etc.).
- the HPI classification 110 is stored as an associated binary value to the HPI 108 .
- the HPI classification 110 and the HPI 108 may be concatenated together, such that the HPI classification 110 is added to the text string of the HPI 108 .
- the HPI 108 and HPI classification 110 are stored as a single text string.
- the HPI classification 110 may be stored by any suitable method.
- FIG. 2 is a block diagram depicting the HPI classifier 104 of FIG. 1 in greater detail.
- the HPI classifier 104 includes an example preprocessor 202 and an example neural network 214 .
- the example preprocessor 202 includes an example natural language processor 204 and an example tensor generator 212 .
- the example natural language processor 204 includes an example tokenizer 206 , an example lemmatizer 208 , an example sentence reorderer 209 and an example named entity recognizer 210 .
- the example neural network 214 includes an embedding layer 216 , an example LSTM layer 218 and an example fully connected layer 220 .
- the neural network 214 has auxiliary elements to retrain including an example model evaluator 222 , an example model trainer 224 and an example model deployer 226 .
- the example HPI classifier 104 includes a medical system interface 228 .
- the HPI classifier 104 receives the unprocessed HPI 108 (e.g., from the data source 102 of FIG. 1 ).
- the HPI 108 is preprocessed by the preprocessor 202 beginning in the natural language processor 204 .
- the example sentence reorderer 209 shuffles the order of each sentence of the HPI 108 around into a random order.
- reshuffling the order of the sentences of the HPI 108 prevents the order of sentences from effecting the output of the neural network 214 .
- the classification of the HPI 108 is only dependent on the presence of the seven elements and not the ordering of the elements, this removes any potential unintentional effects that sentence ordering may have on the HPI classification 110 .
- the tokenizer 206 converts each word or group of words of the HPI 108 into a token.
- the tokenizer 206 breaks the input HPI 108 string into individual tokens. For example, if the tokenizer 206 encounters the sentence “the quick brown fox jumps over the lazy dog,” the tokenizer 206 would tokenizer the sentence into “the,” “quick,” “brown,” “fox,” “jumps,” “over,” “the,” “lazy,” and “dog.”
- the tokenizer 206 tokenizes the HPI 108 based on a “space” delimiter (e.g., “ ”). In other examples, the tokenizer 206 can tokenize the HPI 108 based on another character, rule, etc.
- the tokenizer 206 can have special-case rules which allow for certain types of phrases to be tokenized together. For example, if a date (e.g., “Mar. 12, 2018”) is encountered, the tokenizer 206 can tokenize the date into a single token. Additional examples include, the tokenizer 206 can tokenize names and titles together (e.g., “Dr. Smith”) and/or certain medical abbreviations (e.g., “obstructive sleep apnea,” “cardiac arrest,” and “Type 2 diabetes.”). Additionally or alternatively, the tokenizer 206 can tokenize short phrases together based on simple rules. For example, the tokenizer 206 can group together numbers and words following them together (e.g., “4 hours,” and “five ounces”). In some examples, multiword phrases indicating locations can similarly be tokenized together (e.g., “St. George's Hospital” and “Chicago, Ill.”).
- the example named entity recognizer 210 scans the tokenized HPI 108 for numbers, dates, named entities, medical terms, abbreviations, and/or misspelling and replaces these elements with standardized tokens. For example, if the named entity recognizer 210 identifies the token “Dr. Smith”, the named entity recognizer 210 replaces the token with a standardized indication token such as “PERSON.” For example, if the named entity recognizer 210 identifies the token “Mar. 12, 2018”, the named entity recognizer 210 replaces the token with a token saying “DATE.” Alternatively, the token “Mar. 12, 2018” is replaced with three tokens representing month, day and year, namely “DATE,” “DATE,” and “DATE,” respectively.
- the named entity recognizer 210 replaces the token with a standardized token such as “FACILITY.” For example, if the named entity recognizer 210 identifies the token “4 Hours”, the named entity recognizer 210 replaces the token with a standardized token such as “TIME.” For example, if the named entity recognizer 210 identifies the token “five ounces”, the named entity recognizer 210 replaces the token with a token such as “QUANTITY.” For example, if the named entity recognizer 210 identifies the token “Chicago, Ill.”, the named entity recognizer 210 replaces the specific token with a standardized token such as “LOC”, “LOCATION”, etc.
- the named entity recognizer 210 can replace medical abbreviations, abbreviations and misspellings with a standardized token representing words that are out of vocabulary (e.g., OOV, etc.). In some examples, out of vocabulary words are referenced to a dictionary. In other examples, the named entity recognizer 210 may have a separate token for medical terms and abbreviations (e.g., “MED.”). In this example, the example HPI classifier 104 includes a medical dictionary (e.g., Radlex, LOINC, SNOMED, CPT, ICD-10, etc.).
- a medical dictionary e.g., Radlex, LOINC, SNOMED, CPT, ICD-10, etc.
- the named entity recognizer 210 can replace medical terms and abbreviations with more specific tokens (e.g., separate tokens for medical procedures, medicines and diseases, etc.).
- the named entity recognizer 210 can replaced medical terms and abbreviations with tokens relating to specific bodily systems (e.g., “heart stent” could be replaced with a token reflecting a circulatory procedure (e.g., “CIR PRO,” etc.)).
- the example lemmatizer 208 receives the tokens from the named entity recognizer and replaces each token with a lemma associated with the respective token.
- a “lemma” is the dictionary form of a word.
- the lemmatizer 208 replaces inflected verbs with a related base verb. For example, if the lemmatizer 208 encounters token “am,” “are,” or “is,” the lemmatizer 208 can replace the token with “be.” Additionally or alternatively, the lemmatizer 208 can similarly replace inflected noun tokens (e.g., “cars,” “cars',” “car's,” etc.) with their related lemma (e.g., car).
- the lemmatizer 208 can have similarly functionality with other types of words.
- the lemmatizer 208 can use a word's context to determine its proper lemma. For example, the word “drawer” can have the lemma “drawer” if the word is user a noun or “draw” if the word is used as a verb.
- the lemmatizer 208 reduces the required complexity of the neural network by reducing the possible number of inputs the network can receive.
- the natural language processor 204 outputs a preprocessed HPI 211 .
- the lemmatizer 208 , sentence reorderer 209 , and the named entity recognizer 210 may not be components within the natural language processor 204 .
- the preprocessed HPI 211 may not be lemmatized, reordered or have its named tokens replaced with standardized or other predefined tags.
- any suitable type of preprocessing can be performed to generate the preprocessed HPI 211 .
- the example tensor generator 212 receives the tokenized HPI 211 .
- the example tensor generator 212 receives the tokenized HPI 211 and outputs a tensor 213 .
- the tensor generator 212 converts each token of the tokenized HPI 211 into a vector.
- the vector is a binary sparse vector in which one dimension (e.g., one index) has a value of “1” and each of the other dimensions are “0.”
- each dimension of the vector represents a different possible token. For example, if the tokenized HPI 211 can be composed from any number of 50,000 different tokens, each vector has 50,000 different dimensions.
- the tensor generator 212 vectorizes each of the one hundred tokens into a vector.
- the example tensor 213 includes each of these vectors concatenated (e.g., “stacked”, appended, etc.) together.
- the tensor generator 212 vectorizes each token into a scalar value representing the would-be index of sparse value of the associated vector.
- the tensor 213 is a vector of these scalar values.
- the example tensor 213 is input into the neural network 214 .
- the neural network 214 is an LSTM network.
- the neural network 214 can be implemented using a general RNN, recursive neural network, or any other suitable type of machine learning architecture.
- the neural network 214 can be a part of a larger and/or more complex neural network with additional functions (e.g., identifying the bodily system described in HPI, etc.).
- the neural network 214 outputs a binary output (e.g., the HPI classification 110 ).
- the output of the neural network 214 can indicate the presence of particular HPI elements (e.g., a location of problem, a quality of problem, etc.) in the input HPI 108 .
- the first layer of the neural network 214 is an embedding layer 216 to prepare tensor(s) 213 for processing by the layers of the neural network 214 .
- the embedding layer 216 converts each vectorized token of the tensor 213 into a dense vector corresponding to that token.
- the number of dimensions of the embedding layer 216 corresponds to the length of the dense vector created by the embedding layer 216 .
- adding more dimensions to the embedding layer 216 increases the accuracy and robustness of the neural network 214 .
- each unique sparse vector of the tensor 213 is embedded to a specific corresponding dense vector by the embedding layer 216 .
- the specific values of the dimensions of the embedded dense vectors are optimized during the training process of the neural network 214 .
- the embedded dense vectors 217 are input to the LSTM layer 218 of the neural network.
- a 2D tensor can be transformed into a 3D tensor via the embedding layer 216 as input to the RNN to determine a feature output (e.g., brief/extended, etc.).
- the example LSTM layer 218 receives the embedded dense vectors 217 output by the embedding layer 216 and outputs a single output vector 219 of a predetermined length.
- the dimensions of the LSTM layer 218 correspond to the length of the output vector 219 .
- the LSTM layer 218 uses a soft-sign activation function.
- any suitable activation function may be used (e.g., a hyperbolic tangent (tanh) activation function, etc.).
- the operations of the LSTM layer 218 are optimized during the training of the neural network 214 .
- the LSTM layer 218 leverages history or learned recognition of language, words, phrases, patterns, etc., in the input vectors 217 using information stored in recurrent gates from prior visible and/or hidden cells in the LSTM layer 218 to arrive at the output vector 219 based on the combination of information in the vector(s) 217 .
- An LSTM unit in the LSTM layer 218 receives input state, hidden state, and cell state information and processes the input information using one or more gates including sigmoid, hyperbolic tangent, etc., to apply weighted and/or unweighted element-wise addition and/or multiplication to the vector elements and produce an output state. Via the LSTM 218 , some information can be stored and/or conveyed from one cell to another via the output state and other information can be discarded or “forgotten” to rid the model of old or outdated information.
- the output vector 219 of the LSTM layer 218 is input into the fully connected layer 220 .
- the fully connected layer 220 has a single dimension with a binary output indicating if the HPI 108 is “brief” or “extended.”
- the neural network 214 has additional outputs (e.g., determining which bodily system, such as endocrine system, renal system, etc., is described in the HPI, the presence of particular HPI elements, etc.)
- the fully connected layer 220 can have additional dimensions.
- the fully connected layer 220 uses a sigmoid activation function.
- the output vector of the LSTM layer 218 is linearized by matrix multiplication.
- this scalar value is then rounded to either “0” or “1,” which are associated with either “brief” or “extended,” respectively.
- the binary output value generated by the fully connected layer 220 is the HPI classification 110 .
- the fully connected layer 220 can generate the HPI classification 110 by any other suitable function.
- the function of the fully connected layer 220 is optimized during the training of the neural network 214 .
- the neural network 214 can be periodically retrained (e.g., based on a threshold of feedback, at the discretion of an operator of the system, quarterly, etc.). In some examples, the neural network is automatically retrained after a certain threshold of incorrectly classified HPIs are accumulated (e.g., the a model evaluator 222 determines that too many HPIs have been mis-classified so the model should be updated to improve classification accuracy, etc.). To retrain the neural network 214 , the model evaluator 222 , a model trainer 224 and a model deployer 226 are used. The example model evaluator 222 monitors and evaluates the output HPI classifications 110 of the neural network 214 .
- the model evaluator 222 notes the error of the misclassified HPI 108 along with the correct HPI classification.
- another system and/or application such as a billing system, computer-aided diagnosis system, quality control processor, etc., flags and/or otherwise identifies an incorrect HPI classification 110 , which can be noted by the model evaluator 222 along with the correct classification.
- the model evaluator 222 can monitor a government and/or third-party process that rejects an HPI and/or associated medical record due to HPI misclassification.
- the model evaluator 222 when the model evaluator reaches a threshold in of feedback, the model evaluator 222 triggers or otherwise instructs the model trainer 224 to begin training a new neural network (e.g., to replace the deployed neural network model/construct 214 ). Additionally or alternatively, the model evaluator 222 can periodically (e.g., quarterly, yearly, etc.) trigger the model trainer 224 to begin training a new neural network. In some examples, the model evaluator 222 also monitors for positive feedback (e.g., a human, system, process, etc., verifying that an HPI was correctly classified and can be used).
- positive feedback e.g., a human, system, process, etc., verifying that an HPI was correctly classified and can be used.
- the example model trainer 224 trains a new, updated, or different neural network model/other construct to replace the currently deployed neural network 214 .
- the model trainer 224 can use the positive or/and negative feedback compiled by the model evaluator 222 to create a new data set of HPIs to train and/or test the new neural network.
- the model trainer 224 can use previous training/testing data (e.g., pre-classified HPIs used to train the neural network 214 ) in conjunction with the newly constructed training/testing data set (e.g., pre-classified HPIs not used to train the neural network 214 ).
- the model trainer 224 iteratively varies a strength of connection between the nodes/units of the neural network until the newly trained model set achieves a desired accuracy (e.g., the new neural network correctly classifies the previously incorrectly classified HPIs of the training set).
- the model train then uses a separate test set of HPIs to validate the accuracy of the newly trained neural network. If the result of this validation satisfies specified criterion(-ia), the model trainer 224 outputs the newly trained neural network to the model deployer 226 .
- the example model deployer 226 deploys the trained neural network model. For example, the model deployer 226 makes the strength of connections between nodes of the neural network rigid (e.g., not able to change). Once the model deployer 226 has made the newly trained neural network static, the model deployer 226 replaces the neural network 214 with the newly trained neural network as the deployed neural network 214 (e.g., the deployed model) to be used in classification of incoming medical data.
- the example medical system interface 228 modifies a medical support system (e.g., the medical support system 106 of FIG. 1 ) with the HPI classification 110 .
- the example medical system interface 228 may trigger a response from the medical support system 106 .
- the medical support system 106 is an EMR
- the medical system interface 228 may cause the HPI classification 110 to be associated with a patient's medical history.
- the medical support system 106 is a medical billing system
- the medical system interface 228 may trigger a medical bill to be generated based on the classified HPI (e.g., a more expensive bill may be issued if the HPI classification 110 is an extended).
- FIG. 2 While an example implementation of the HPI classifier 104 of FIG. 1 is illustrated in FIG. 2 , one or more of the elements, processes, and/or devices illustrated in FIG. 4 can be combined, divided, re-arranged, omitted, eliminated, and/or implemented in any other way.
- the example preprocessor 202 , the example natural language processor 204 , the example tokenizer 206 , the example lemmatizer 208 , the example sentence reorderer 209 , the example named entity recognizer 210 , the example tensor generator 212 , the example neural network 214 , the example embedding layer 216 , the example LSTM layer 218 , the example fully connected layer 220 , the medical system interface 228 and/or, more generally, the example HPI classifier 104 of FIG. 1 can be implemented by hardware, software, firmware, and/or any combination of hardware, software, and/or firmware.
- DVD digital versatile disk
- CD compact disk
- Blu-ray disk etc.
- the example HPI classifier 104 of FIG. 1 can include one or more elements, processes, and/or devices in addition to, or instead of, those illustrated in FIG. 2 , and/or may include more than one of any or all of the illustrated elements, processes, and devices.
- the phrase “in communication,” including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events.
- the machine readable instructions can be a program or portion of a program for execution by a processor such as the processor 1012 shown in the example processor platform 1100 discussed below in connection with FIG. 10 .
- the program can be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor 1012 , but the entire program and/or parts thereof can alternatively be executed by a device other than the processor 1012 and/or embodied in firmware or dedicated hardware.
- any or all of the blocks can be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.
- hardware circuits e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.
- FIGS. 6-9 can be implemented using executable instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory, and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information).
- a non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media.
- A, B, and/or C refers to any combination or subset of A, B, C such as (1) A alone, (2) B alone, (3) C alone, (4) A with B, (5) A with C, and (6) B with C.
- FIG. 3 depicts an example data flow diagram 300 of an HPI 302 through the HPI classifier 104 of FIGS. 1 and 2 .
- the example HPI 302 e.g., the narrative HPI 108 of FIG. 2
- the example HPI 302 begins with the phrase “THIS IS A 66 YEARS OLD FEMALE.”
- the HPI 302 is received by the preprocessor 202 .
- the preprocessor 202 converts the example HPI 302 into an example preprocessed HPI 304 .
- the preprocessor 202 can tokenize the HPI 302 (e.g., using the tokenizer 206 ), lemmatize the HPI 302 (e.g., using the lemmatizer 208 ), reorder the sentences of the HPI 302 (e.g., using the sentence reorderer 209 ), and/or replace named entities recognized in the HPI 302 (e.g., using the named entity recognizer 210 ).
- the beginning phrase “THIS IS A 66 YEARS OLD FEMALE” of the HPI 302 is tokenized into “THIS,” “IS,” “66” “A YEARS” “OLD,” and “FEMALE” by the tokenizer 206 .
- the lemmatizer 208 then replaces the “IS” token with the token's lemma, “BE.” Additionally, the named entity recognizer then replaces each of “66,” “A YEARS,” and “OLD” with a named entity token of “DATE.”
- the beginning phrase of the example HPI 302 has been preprocessed into the example preprocessed HPI 304 that includes the tokens “THIS,” “BE,” “DATE,” “DATE,” “DATE,” and “FEMALE.” The preprocessed HPI 304 is then sent to the tensor generator 212 .
- the preprocessed HPI 304 is converted into an example sparse tensor 306 .
- each token of the preprocessed HPI is one-hot encoded into a sparse vector. For example, if the number of possible tokens than included in the preprocessed HPI is 50,000, each sparse vector is 50,000 dimensions in length.
- each sparse vector is stored as a scalar value in the example tensor 306 where each scalar value represents the dimension of the sparse vector.
- the token “THIS” is associated with the 15,220th dimension of the sparse vector and is stored in the example tensor 306 as “15,220.”
- the tokens “BE” and “FEMALE” are similarly stored as “5,560” and “42,284” respectively.
- each of the “DATE” tokens is stored as “387” in the sparse tensor 306 .
- each element of the sparse tensor 306 is converted into an example dense tensor 308 .
- each element of the example sparse tensor 306 e.g., the sparse vectors
- the mathematical process of this conversion is optimized during the training of the model (e.g., the neural network 214 of FIG. 2 ).
- repeat elements of the sparse tensor 306 e.g., “387” are embedded as the same dense vector (e.g., [66, 37, 672, . . . , 31]).
- a length of each of the vectors in the dense vector 308 is equal to a number of dimensions in the embedding layer 216 .
- the example LSTM Layer 218 converts the dense vector 308 into an output vector 310 using a softsign activation function.
- the mathematical process of this conversion is optimized during the training of the model (e.g., the neural network 214 of FIG. 2 ).
- the length of the output vector 310 corresponds to the number of dimensions of the LSTM layer 218 .
- the example output vector 219 is then input into the fully connected layer 220 .
- the example fully connected layer 220 linearizes the example output vector 310 into an example binary output 312 using a sigmoid function.
- the binary output 312 corresponds to the HPI classification 110 .
- FIG. 4 shows an example confusion table 400 which includes example results of a group of human classified HPIs classified using by a deployed HPI classifier 104 of FIG. 1 .
- a confusion table is a metric used to visualize performance of an algorithm or neural network.
- a grading index 402 depicts a greyscale gradient of example percentages the results can have.
- a y-axis 404 represents the “True Label” (e.g., labeled by humans) of an HPI
- an x-axis 406 represents the “Predicted Label” of HPI as classified by the HPI classifier 104 .
- FIG. 5 is a process flow diagram of an example process 500 to deploy the HPI classifier 104 of FIG. 1 to be used to classify incoming patient HPIs.
- the process 500 begins at block 502 .
- the data source 102 ingests a set of preclassified HPIs to be used to train the neural network (e.g., the neural network 214 of FIG. 2 ) of the HPI classifier 104 .
- each HPI of the set of HPIs is classified by a healthcare professional prior to being ingested by the data source 102 .
- the HPI classification 110 is ingested with the example HPI 108 .
- the preprocessor 202 preprocesses the ingested data. Additional detail in the function of the preprocessor is provided below in conjunction with FIG. 7 . In some examples, the preprocessor 202 also preprocesses the HPI classification 110 .
- the model (e.g., the neural network 214 ) is trained using the preprocessed HPI(s) 108 and HPI classification(s) 110 (e.g., collectively referred to as the samples).
- the samples are processed iteratively in epochs until the model converges.
- the samples are divided such that are some of the samples are used for training and some are used for validation (e.g., confirming the model works after training).
- Known outcomes/results can be used to verify performance of the training model, which can also be validated with a test data set.
- a set of known, “gold standard”, “truthed”, or other reference data can be divided into a training data set to train the model and a test data set to test the trained network model to validate its correct operation. After the model has been trained and validated, the process 500 advances to block 508 .
- the model is deployed.
- the model is deployed as application within a medical support system (e.g., the medical support system 106 of FIG. 1 ), billing software, computer-aided diagnosis (CAD) application, etc.
- CAD computer-aided diagnosis
- the model is a standalone application.
- the process 500 is being used to retrain the model, the model is to replace a previously used model so a prior deployed model is replaced with the newly deployed model in the target system.
- the model evaluator 222 monitors the monitor for potential misclassifications.
- the model evaluator keeps a database of improperly classified HPIs. Classified HPIs can be confirmed as properly or improperly classified through user feedback, other system evaluation (e.g., a billing system determines that an HPI is not in fact extended, etc.), etc. Such feedback can be used to trigger a subsequent retraining of the model (e.g., when a number or percentage or improper classifications reaches or exceeds a threshold, criterion, etc.), for example.
- FIG. 6 is an illustration of an example data flow and transformation of information 600 flowing through the example data source 102 , the example HPI classifier 104 and the example medical support system 106 of the example system of FIG. 1 .
- an example transmission 602 e.g., a message, an instruction, a data packet, etc.
- the example transmission 602 includes an unprocessed HPI (e.g., the HPI 108 of FIG. 1 ). Additionally or alternatively, the example transmission can include any other suitable information (e.g., information about the data source 102 , about an associate patient, etc.).
- the example HPI classifier 104 can use the data included in the transmission 602 to execute an example action 604 .
- the example action 604 can include classifying the unprocessed HPI of the transmission 602 .
- the HPI classifier 104 can transmit an example transmission 606 including an HPI classification to medical support system 106 and/or an example HPI classification message 608 back to the data source 102 .
- the example transmission 606 and the example transmission 608 include an HPI classification.
- the transmission 606 and/or the example transmission 608 can further include the HPI 108 .
- the example transmission 608 can notify the data source 102 of the HPI classification generated by the HPI classifier 104 .
- the example HPI classifier 104 can also transmit an example transmission 610 .
- the example transmission 610 can include system instructions that can cause the medical support system 106 to undergo an example action 612 .
- the action 612 can include generating a bill and/or insurance claim.
- the medical system interface 220 can cause the medical support system 106 to generate, update or delete a medical record.
- the action 612 can include generating, updating or deleting a medical record. In some examples, the action 612 can include generating a request or reminder for an appointment. Additionally or alternatively, the action 612 can trigger any other suitable action from the medical support system 106 such as schedule an imaging exam, schedule a laboratory test session, trigger a reminder for clinician follow-up, configure an imaging workstation and/or other clinician computing device for patient data analysis, etc.
- the medical support system 106 can further send an example transmission 614 to the data source 102 .
- the example transmission 614 can include feedback (e.g., a notification of whether the HPI classification was correct, etc.) for the medical support system 106 .
- the example transmission 614 can further include a request to manually classify one or more HPIs that can also be included in the example transmission 614 .
- the example transmission 614 can trigger the data source 102 to send an example transmission 616 .
- the example transmission 616 can include, for example, a manual classification of an HPI included in the example transmission 614 .
- the medical support system 106 can further transmit an example transmission 618 .
- the example transmission 618 can include training data to be used to by the HPI classifier 104 .
- the example transmission 618 can include unclassified HPIs stored in the medical support system 106 and/or incorrectly classified HPIs (e.g., incorrectly classified by the HPI classifier 104 , incorrectly manually classified, etc.).
- the medical support system 106 can further transmit an example transmission 620 to the HPI classifier 104 .
- the example transmission 620 can include feedback from the medical support system 106 to the HPI classifier 620 .
- the example transmission 620 can trigger an example action 622 .
- the example action 622 can include retraining the neural network (e.g., the neural network 214 of FIG. 2 ) of the HPI classifier 104 and can, for example, include executing the process 500 of FIG. 5 .
- the HPI classifier 104 can be triggered to regenerate the network model 300 to be redeployed for further HPI classification, etc.
- FIG. 7 is a flowchart representative of machine readable instructions 700 which can be executed to implement the HPI classifier of FIG. 2 .
- the process 700 of FIG. 7 begins at block 702 .
- the preprocessor 202 receives the narrative HPI 108 from the data source 102 .
- the narrative HPI 108 is retrieved from a database of HPIs.
- the HPI 108 is input by a patient or healthcare professional after conducting a medical with the patient.
- the HPI 108 can be retrieved from any suitable source.
- the HPI 108 is formatted as one continuous text string.
- the HPI 108 can be formatted in any way readable by the preprocessor 202 .
- the process 700 advances to block 704 .
- the preprocessor 202 preprocesses the HPI 108 . Additional detail in the execution of block 704 is provided below in conjunction with FIG. 8 .
- the process advances to block 706 at which the tensor generator 212 generates a tensor 213 of vectorized tokens.
- the tensor generator 212 can map each of the tokens of the tokenized HPI 108 into a sparse vector and forms a tensor 213 by concatenating each of the sparse vectors.
- the tensor generator can map each the tokens of the tokenized HPI 211 into a scalar value, which corresponds to the dimension of the sparse vector that token would be mapped to.
- the tensor generator 212 generates a vector (e.g., the tensor 213 ) by stacking these values. Additionally or alternatively, any suitable method for generating the tensor 213 can be executed. Once the tensor 213 has been generated, the process 700 advances to block 708 .
- the neural network 214 classifies the HPI 108 . Additional detail in the execution of block 708 is provided below in conjunction with FIG. 8 .
- the process 700 advances to block 710 .
- the medical system interface 220 modifies a medical support system (e.g., the medical support system 106 ) based on the HPI 110 classified.
- the medical support interface 220 also modifies the medical support system 106 with unclassified HPI 108 .
- the medical system interface 220 may trigger an action from the medical support system 106 .
- the medical system interface 220 can trigger, facilitate and/or otherwise cause the medical support system 106 to schedule a lab test (e.g., bloodwork, etc.) and/or a medical procedure.
- the medical system interface 220 can trigger, facilitate, and/or otherwise cause the medical support system 106 to generate a bill and/or an insurance claim.
- the medical system interface 220 can trigger, facilitate, and/or otherwise cause the medical support system 106 to generate, update or delete a medical record.
- the medical support interface 220 can trigger, facilitate, and/or otherwise cause the medical support system 106 to generate request and/or reminder for an appointment.
- the medical support interface 220 can trigger, facilitate, and/or otherwise cause the medical support system 106 to issue a reminder to the data source 102 to retake the HPI 108 . Additionally or alternatively, the medical support system 106 can trigger facilitate, and/or otherwise cause any other suitable action from the medical support system 106 such as schedule an imaging exam, trigger a reminder for clinician follow-up, configure an imaging workstation and/or other clinician computing device for patient data analysis, etc.
- process control decides whether the neural network 214 needs to be retrained.
- the retraining decision is based on whether as many or more than a threshold of incorrectly labeled HPIs have been accrued.
- the decision to retrain the neural network 214 can instead base on a time interval (e.g., monthly, yearly, etc.).
- the neural network 214 can be retrained based on a user, application, and/or system trigger (e.g., by the by an administrator of the medical support system 106 by a billing system, etc.). If the neural network 214 is to be retrained, the process 700 advances to block 714 . If the neural network 214 is not to be retrained, the process 700 ends.
- the model trainer 224 retrains the neural network 214 . Additional detail in the execution of block 714 is provided below in conjunction with FIG. 10 .
- the subprocess 800 of FIG. 8 depicts the execution of block 704 of FIG. 7 in greater detail.
- Subprocess 800 begins at block 802 .
- the sentence reorderer 209 randomly reorders each sentence of the HPI 108 .
- the sentence reorderer 209 parses the input HPI 108 to determine sentences (e.g., by punctuation, capital, or any other suitable method to parse a text string into sentences).
- the sentence reorderer 209 then randomly shuffles the ordering of the sentences in the HPI 108 .
- reshuffling the HPI 108 prevents the neural network 214 from being trained to classify the HPI 108 on the ordering of elements instead of their presence.
- any other suitable method can be used to randomly reorder the sentences of the HPI 108 .
- the sentence reorderer 209 can then concatenate the parsed sentences into a single text string.
- the subprocess 800 then advances to block 804 .
- the tokenizer 206 tokenizes the HPI 108 .
- the tokenizer 206 can parse the HPI 108 into individual tokens.
- the tokenizer 206 tokenizes the HPI 108 by identifying a “space” or “ ” delimiter.
- the tokenizer 206 can tokenizer the HPI 108 by identifying other punctuation, sentence/phrase structure, related terms, etc.
- the tokenizer 206 can have special-case rules which allow for certain types of phrases (e.g., dates, Names, medical terms, etc.) to be tokenized together.
- the lemmatizer 208 lemmatizes the tokens of the HPI 108 .
- the lemmatizer scans each token and replaces each token with a lemma associated with that token.
- the lemmatizer 208 can leverage a database of words and their associated lemmas.
- the lemmatizer 208 utilizes a simple or a neural network to determine a context of a token.
- the context of a token can be used to determine its proper lemma (e.g., the word drawer has multiple lemmas).
- any suitable method can be used to replace tokens with their lemmas.
- the subprocess 800 then advances to block 808 .
- the named entity recognizer 210 replaces the tokens of named entities with predefined tags. For example, the named entity recognizer 210 parses the lemmatized tokens for any named entities and replaces each named entity with a tag from a database. In some examples, named entities, such as places, people and dates, are replaced with a predetermined tag. In some examples, the named entity recognizer 210 also replaces misspellings and other tokens that the named entity recognizer 210 does not recognize with a separate tag indicating the word is out of vocabulary (e.g., “OVV”).
- OOVV misspellings and other tokens that the named entity recognizer 210 does not recognize with a separate tag indicating the word is out of vocabulary
- the subprocess 900 of FIG. 9 depicts the execution of block 708 of FIG. 7 in greater detail.
- Subprocess 900 begins at block 902 .
- the embedding layer 216 embeds the tensor 213 into dense vectors.
- the embedding layer 216 converts each element (e.g., vector or scalar) of the tensor 213 into a dense vector.
- the length (e.g., the number of dimensions) of the dense vector is a fixed and predetermined quantity.
- the particular mathematical process to embed the tensor 213 into dense vectors is optimized and/or otherwise improved during the training of the neural network 214 .
- the subprocess 900 advances to block 904 .
- the LSTM layer 218 processes the dense vectors into an activated output vector.
- the LSTM layer 218 uses a soft-sign activation function.
- the LSTM layer 218 uses another suitable activation function (e.g., a hyperbolic tangent function, etc.).
- the particular mathematical process to generate the activated output vector is optimized and/or otherwise improved during the training of the neural network 214 .
- the subprocess 900 advances to block 906 .
- the fully connected layer 220 linearizes the output vector in a binary output.
- the fully connected layer 220 uses a sigmoid activation function and/or matrix multiplication to convert the output vector in binary output.
- the fully connected layer 220 linearizes the output vector and then rounds the output into a binary output (e.g., “0” or “1”).
- the binary output is the HPI classification 110 (e.g., “1” corresponds to brief and “0” corresponds to extended).
- the full connected layer can have multiple outputs which include the HPI classification 110 .
- process control decides whether the fully connected layer 220 is to classify the body system(s) described in the input HPI. If the fully connected layer 220 is classify the bodily system(s) described in the input HPI, the subprocess 900 advances to block 910 . If the fully connected layer 220 is not to be classified, the subprocess ends and returns to process 700 .
- the fully connected layer 220 amends the output to include a bodily system classification.
- the fully connected layer 220 can include a binary output for each notable bodily system (e.g., circulatory, endocrine, lymphatic, etc.).
- the fully connected layer 220 can include a probability that the input HPI pertains to a particular bodily system. Once the output has been amended to include a bodily system classification, the subprocess 900 ends and returns to process 700 .
- the subprocess 1000 of FIG. 10 depicts the execution of block 714 of FIG. 7 in greater detail.
- the model evaluator 222 evaluates model results (e.g., the output of the neural network 214 ).
- the model evaluator 222 can randomly sample the model results (e.g., the HPI classifications 110 ) and submit them to be independently evaluated.
- the model evaluator 222 can compiles feedback provided by healthcare professionals and/or third party entities (e.g., an insurance company, the CMS, etc.) on the sampled model results.
- the model evaluator 222 can create a new set of pre-classified training and validation HPIs.
- the model evaluator 222 can create a new set of training and validation HPI regardless of the accuracy the model results. In some examples, the model evaluator 222 continues to collect feedback until a threshold of feedback has been received. In some examples, the model evaluator 222 can evaluate the model periodically (e.g., quarterly, etc.). Once a suitable level of feedback has been collected, the subprocess 1000 advances to block 1004 .
- the model trainer 224 retrains the model using the evaluated results and/or new training set.
- the model trainer 224 can divide the collected feedback and/or new training set into a training set of HPIs and a validation set of HPIs.
- the model trainer 224 can add pre-classified HPIs from previous sets into the training set and/or validation set.
- the model trainer 224 starting with the current neural network 214 , begins using the training set to iteratively change the strength of connections between nodes in each layer (e.g., the embedding layer 216 , the LSTM layer 218 and/or fully connected layer 220 , etc.) until a deserved accuracy of classification is achieved.
- the validation set of HPIs is used to verify the fidelity of the newly trainer neural network.
- the model trainer 224 may change the activation functions used by neural network 214 (e.g., change the activation function of the LSTM layer 218 to a Tanh activation function, etc.).
- the model deployer 226 deploys the newly trained neural network.
- the model deployer 226 can replace the currently used neural network 214 with the newly trained neural network model/construct.
- the model deployer 226 makes the connections between nodes of the neural network rigid so they do not change when deployed in the HPI classifier 104 .
- the subprocess 1000 ends and returns to the process 700 .
- FIG. 11 is a block diagram of an example processor platform 1100 structured to execute the instructions of FIGS. 7-10 to implement the HPI classifier 104 of FIG. 2 .
- the processor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPadTM), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, a headset or other wearable device, or any other type of computing device.
- a self-learning machine e.g., a neural network
- a mobile device e.g., a cell phone, a smart phone, a tablet such as an iPadTM
- PDA personal digital assistant
- an Internet appliance e.g., a
- the processor platform 1100 of the illustrated example includes a processor 1112 .
- the processor 1112 of the illustrated example is hardware.
- the processor 1112 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer.
- the hardware processor may be a semiconductor based (e.g., silicon based) device.
- the processor 1112 implements the example preprocessor 202 , the example natural language processor 204 , the example tokenizer 206 , the example lemmatizer 208 , the example sentence reorderer 209 , the example named entity recognizer 210 , the example tensor generator 212 , the example neural network 214 , the example embedding layer 216 , the example LSTM layer 218 , and the example fully connected layer 220 .
- the processor 1112 of the illustrated example includes a local memory 1113 (e.g., a cache).
- the processor 1112 of the illustrated example is in communication with a main memory including a volatile memory 1114 and a non-volatile memory 1116 via a bus 1118 .
- the volatile memory 1114 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and/or any other type of random access memory device.
- the non-volatile memory 1116 may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory 1114 , 1116 is controlled by a memory controller.
- the processor platform 1100 of the illustrated example also includes an interface circuit 1120 .
- the interface circuit 1120 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), a Bluetooth® interface, a near field communication (NFC) interface, and/or a PCI express interface.
- one or more input devices 1122 are connected to the interface circuit 1120 .
- the input device(s) 1122 permit(s) a user to enter data and/or commands into the processor 1112 .
- the input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, an isopoint device, and/or a voice recognition system.
- One or more output devices 1124 are also connected to the interface circuit 1120 of the illustrated example.
- the output devices 1124 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer, and/or speaker.
- display devices e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touchscreen, etc.
- the interface circuit 1120 of the illustrated example thus, typically includes a graphics driver card, a graphics driver chip, and/or a graphics driver processor.
- the interface circuit 1120 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and/or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) via a network 1126 .
- the communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc.
- DSL digital subscriber line
- the processor platform 1100 of the illustrated example also includes one or more mass storage devices 1128 for storing software and/or data.
- mass storage devices 1128 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.
- the machine executable instructions 1132 of FIGS. 7-10 may be stored in the mass storage device 1128 , in the volatile memory 1114 , in the non-volatile memory 1116 , and/or on a removable non-transitory computer readable storage medium such as a CD or DVD.
- example methods, apparatus and articles of manufacture have been disclosed that classify medical data using an artificial intelligence.
- the disclosed examples offer several advantages over manually classified HPI.
- the disclosed examples improve medical data processing for computer-aided diagnosis, billing, treatment approval, and other patient safety and patient care.
- the disclosed examples improve operation of healthcare data processors by correctly and efficiently processing a variety of available information and generating a consistent, accurate result.
- the disclosed examples decrease the probability of denied reimbursement due to incorrect HPI classification.
Abstract
Description
- This patent arises from U.S. Provisional Patent Application Ser. No. 62/644,117, which was filed on Mar. 16, 2018. U.S. Patent Application Ser. No. 62/644,117 is hereby incorporated herein by reference in its entirety. Priority to U.S. Patent Application Ser. No. 62/644,117 is hereby claimed.
- This disclosure relates generally to managing electronic medical records and, more particularly, to methods and apparatus to classify medical data using artificial intelligence.
- A patient's electronic medical records (EMR) are documentation of that patient's history of care and medical encounters stored in an electronic database. During a medical encounter, a healthcare professional usually takes a medical history of that patient. Aspects of a taking a medical history include asking questions to obtain a demographic information, chief complaint (CC), a history of the present illness (HPI), a review of systems (ROS) and past, family and/or social history (PFSH). The Center of Medicare and Medicaid (CMS) divides medical histories into four types, namely: problem focused, focus expanded problem, detailed and comprehensive. An amount of money reimbursed to a healthcare provider often depends on what type of medical history was taken during a medical encounter. A classification of medical history into these four types depends on information included in the gathered HPI, ROS and/or PFSH. CMS standards for medical history include:
-
TABLE 1 CMS Standards for Medical History Type of History HPI ROS PFSH Problem Focused Brief N/A N/A Focused Expanded Problem Brief N/A N/A Detailed Extended Pertinent Pertinent Comprehensive Extended Complete Complete - HPIs are narrative summaries compiled by a healthcare professional after identifying a patient's chief compliant (e.g., a reason for their visit). Classifying an HPI as “brief” or “extended” depends upon what information is obtained from the patient and recorded during the interview. The CMS has established a criterion such that an HPI is classified as “extended” when it contains four or more of the following elements: (1) location of problem, (2) quality of problem, (3) severity of problem, (4) duration of problem, (5) context of problem, (6) modifying factors, and (7) associated signs and symptoms. An HPI is classified as “brief” when it contains three or fewer elements. Historically, HPIs are classified by healthcare professionals during or after they are obtained or recorded.
- An example apparatus for classifying a string according to a binary classification system, comprising: a processor to execute instructions to implement at least: a preprocessor to: tokenize the string into tokens based on a context associated with at least one section of the string; and convert the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context; a recurrent neural network including: an embedding layer to embed the hashes into dense vectors; a long short-term memory (LSTM) layer to convert the dense vectors into an activated output vector; and a fully connected layer to linearize the activated output vector into a binary output; and memory to store at least one of the string, the tensor or the binary output.
- An example method includes receiving an HPI formatted as a string, the string including one or more words, the words organized in an order of sentences; tokenizing the one or more words into tokens based on a context associated with at least one of the one or more words; converting the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context; embedding each of the hashes into vectors; processing the vectors to classify the HPI as extended or brief based on a similarity to a set of classified HPIs; outputting a classification for the HPI; and modifying a medical support system with the HPI and the classification and to trigger an action with respect to the medical support system based on the classification.
- An example tangible machine readable medium comprising instructions, which when executed, cause a processor to at least tokenize a string into tokens based on a context associated with at least one section of the string; convert the tokens into hashes, each of the hashes forming a dimension of a tensor based on the context; embed the hashes into dense vectors; convert the dense vectors into an activated output vector; linearize the activated output vector into a binary output; and store at least one of the string, the tensor or the binary output.
- The features and technical aspects of the system and method disclosed herein will become apparent in the following Detailed Description in conjunction with the drawings in which reference numerals indicate identical or functionally similar elements.
-
FIG. 1 is an illustration of a medical data processing system in which the teachings of this disclosure may be implemented. -
FIG. 2 is a block diagram representative of the HPI classifier ofFIG. 1 . -
FIG. 3 is an example data flow of an HPI through the HPI classifier ofFIG. 2 . -
FIG. 4 is an example confusion table of a group of HPIs submitted to a trained HPI classifier. -
FIG. 5 is a process flow diagram that can be implemented to deploy a model to be used to process incoming patient data. -
FIG. 6 is an illustration of an example data flow and transformation of information flowing through components of the example system ofFIG. 1 . -
FIG. 7 is a flowchart representative of machine readable instructions which can be executed to implement the HPI classifier ofFIG. 2 . -
FIG. 8 is a flowchart representative of machine readable instructions which can be executed to preprocess input HPI data. -
FIG. 9 is a flowchart representative of machine readable instructions which can be executed to classify the preprocessed HPI data. -
FIG. 10 is a flowchart representative of machine readable instructions which can be executed to retrain the HPI model. -
FIG. 11 is a block diagram of an example processing platform structured to execute the instructions ofFIGS. 7-10 to implement the systems and methods described herein including the HPI classifier and/or other components ofFIG. 2 . - In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. These examples are described in sufficient detail to enable one skilled in the art to practice the subject matter, and it is to be understood that other examples may be utilized and that logical, mechanical, electrical and other changes may be made without departing from the scope of the subject matter of this disclosure. The following detailed description is, therefore, provided to describe an exemplary implementation and not to be taken as limiting on the scope of the subject matter described in this disclosure. Certain features from different aspects of the following description may be combined to form yet new aspects of the subject matter discussed below.
- When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.
- Acquisition, analysis, classification and storage of information gathered while taking a medical history is important to the diagnosis and treatment of patient. Additionally, reimbursements from organizations like the CMS often depend on the quality of information gathered during while taking medical history. For example, generally more detailed medical histories (e.g., detailed or comprehensive histories, see table 1) garner high reimbursement fees from CMS. One determining factor in deciding if a medical history is detailed or comprehensive is determining if the taken HPI is “extended” or “brief” Historically, when an HPI is taken from a patient and recorded in the EMR of the patient, a healthcare professional would classify the HPI based the presence of seven elements.
- As computers continue to take on a greater role in patient care, automating the process for determining if an HPI is “extended” or “brief” becomes important to the maintenance and completeness of an EMR of a patient. Automating the process of classifying an HPI potentially allows an HPI to be gathered by a computer instead of a healthcare professional, thus allowing healthcare professionals to be performing more critical tasks. Additionally, empirical evidence suggests humans incorrectly classify HPIs at a relatively high and significant rate. Automating the classification process could potentially alleviate some of these potential issues. In some examples, this may lead to more accurate medical records for the patient and more accurate billing for the CMS and healthcare organizations. Furthermore, automatic classification of an HPI as “brief,” may encourage a healthcare professional to retake an HPI to ensure the HPI is “extended.”
- However, HPIs resist being easily classified by standard artificial intelligences (AI) and/or natural language processors (NLPs) for a number of reasons. First, HPIs are often recorded as a narrative which makes identifying which, if any, of the seven elements are present in a particular patient's HPI difficult. Because each HPI is recorded by a different healthcare professional, the writing style (e.g., punctuation, abbreviations, word choice, sentence structure, etc.) of each narrative can vary. Additionally, in some examples, the narrative may contain misspelled words or incomplete sentences. Second, the HPIs often contain high occurrences of medical terms, abbreviations and named entities which often have different meanings depending on the context. For example, “pt” can refer to either “patient” or “physical therapy” depending on the context. Abbreviations such as “OSA” (obstructive sleep apnea), “PSA” (prostate specific antigen), etc., can be difficult to process. Similarly, named entities such as “Dr. Smith”, “CPCA (California Primary Care Association)”, etc., can also be difficult to process. Third, HPIs often contain extensive use of numbers with different semantic meanings. For examples, the phrases “last colonoscopy was 2009,” “the pain lasts 5 minutes,” and “Type 2 Diabetes” all contain numbers with different semantic meanings (e.g., a date, a duration and a classification of disease, respectively). Fourth, the length (e.g., word count, number of sentences, etc.) of an HPI does not necessarily correlate with its classification.
- For example, the following HPI is relatively long but would be classified as a brief HPI:
-
- The patient comes in today for annual GYN exam. The patient has no history of no GYN complaints, abnormal period, pelvic pain, abnormal vaginal discharge, breast mass, breast pain, depression, anxiety, urinary symptoms, chest pain, palpations, shortness of breath, leg swelling, back pain, abdominal pain, headaches, bowel problems, menopausal issues, PMS, amenorrhea, and dysmenorrhea. Patient is a 58 YEARS Old G1PI who presents for an annual exam. Last exam was 2009 with Kaiser. Hysterectomy 2003 (supracerivcal). Mild hot flashes, when emotional stress is high. Pt needs mammogram. PT is taking calcium+D. Pt has colonoscopy 2007, polyps removed. I recommend that pt see PCP for referral to GI. Pt mentions urinary frequency. She attributes this to diuretic. No urgency. No dysuria. Pt gets regular exercise.
This HPI contains two elements, context (e.g., “mild hot flashes when emotional stress is high”) and modifying factors (“attributes hot flashes to diuretic”) and is therefore considered a “brief” HPI, rather than an “extended” HPI. Another example of an HPI that is likely to be identified as an “extended” HPI but is in fact brief because it only focuses on past medical history rather than the present illness is as follows (including typos): - xxx is a 21 month old boy who presents for initial pulmonary consult. The patient comes in today with his mother for evaluation of recurrent cough and wheezing. Mother reports that xxx was seen in the ER twice last year for an episode of increased coughing, wheezing and increased work fo breathing. He had a chest xray done once which did no reveal any pneumonia. He was placed on Albuterol nebs and oral steroids with improvement of the symptoms. Since then, mother reports that xxx has developed episodes of coughing and wheezing with viral URI. He had a several respiratory flare ups in the last 3 months. The most recent was 2 weeks ago. He presented with increased coughing and increased heart rate. He was brought to the ER and was advised to take the Albuterol and not need to be on the oral steroids. During this visit, mother reports that xxx has been doing better. He still has intermittent coughing but decreased in frequency. He has been taking the Pulmicort nebs twice a day in the last 2 weeks and also receiving the Albuterol nebs twice a day. Otherwise, he remains to be active with no fever, good appetite and weight gain. There were no episodes of pneumonia. There has been no history of hospitalization for respiratory exacerbation.
Alternatively, an HPI may be relatively short in length but “extended.” For example, the following excerpt is relatively short (e.g., shorter than the excerpt above) but includes enough elements to be an “extended” HPI: - RIH Urinary tract symptoms year old male complaining of pain, discomfort and the presence of an enlarging mass/bulge on his RIGHT inguinal region. He has noted more discomfort and was referred by his primary care physician for evaluation and advice Denies trauma, chills or fever. He has urinary tract symptoms, with penile discomfort, burning sensation and weaker urinary stream.
This HPI contains four elements: location (e.g., RIGHT inguinal region), quality (e.g., pain and discomfort), modifying factors (e.g., denies trauma, chills or fever) and associated signs and symptoms (e.g., urinary tract symptoms, with penile discomfort, burning sensations and weaker urinary stream) and, therefore, is considered an “extended” HPI. Other short in length but extended in content examples can include: - Patient returns for follow up after he underwent a surgical repair of an incarcerated ventral hernia in May 13, 2014. all drains are out now. He states that he is doing well, with minimal complaints.
- This is a 78 year old female who presents with venous ulceration. The patient denies pain, drainage, redness, streaking, malodor, fever, chills, night sweats, calf pain, and non-healing wound. It is located on the lateral leg. The ulceration has been present for 3-6 months. Prior care has included compression stockings.
Thus, the length of an HPI, while easy to determine, can be a poor indicator of whether or not an HPI is a “brief” or “extended.” An erroneous determination of “brief” or “extended” HPI can result in the HPI being processed incorrectly by an EMR, billing system, care plan, etc., with potential negative impact to patient health and health data processing, and can result in erroneously under-reimbursing (e.g., the HPI should qualify as “extended” but is mis-classified as “brief”) or over-reimbursing (e.g., the HPI is mis-classified as “extended” but does not include sufficient information to correctly do so) a provider.
- The patient comes in today for annual GYN exam. The patient has no history of no GYN complaints, abnormal period, pelvic pain, abnormal vaginal discharge, breast mass, breast pain, depression, anxiety, urinary symptoms, chest pain, palpations, shortness of breath, leg swelling, back pain, abdominal pain, headaches, bowel problems, menopausal issues, PMS, amenorrhea, and dysmenorrhea. Patient is a 58 YEARS Old G1PI who presents for an annual exam. Last exam was 2009 with Kaiser. Hysterectomy 2003 (supracerivcal). Mild hot flashes, when emotional stress is high. Pt needs mammogram. PT is taking calcium+D. Pt has colonoscopy 2007, polyps removed. I recommend that pt see PCP for referral to GI. Pt mentions urinary frequency. She attributes this to diuretic. No urgency. No dysuria. Pt gets regular exercise.
- The examples disclosed herein overcome the above obstacles and improve the technology of medical data processing by providing technologically improved systems and methods to normalize an input HPI and classify the HPI using a neural network tuned to process HPI information and generate a classification from the HPI information. In some examples disclosed herein, the HPI is normalized with a natural language processor by tokenizing, lemmatizing, and replacing named entities and medical terms with standardized strings/predefined tags. In some examples disclosed herein, the natural language processor randomly reorganizes the order of each sentence in the input to the HPI. In some examples disclosed herein, the tokens are hashed into integers. In such examples, the integers are representative of an index of a sparse vector where each index represents a distinct word. In examples disclosed herein, the normalized HPI is classified with a neural network. In some examples, the neural network is a three-layer neural network including an embedding layer, recurrent neural network layer, and fully connected layer. In some examples, the recurrent neural network is a long short-term memory (LSTM) network. In some examples, the three-layer neural network outputs a binary output (e.g., a binary classification, either “extended” or “brief” represented as 0 or 1, 1 or 0, etc.) In other examples, the neural network outputs a vector including values corresponding to the presence of each HPI element in an input HPI. In some examples, the output of the neural network can also include a determination of which bodily system(s) is/are discussed in the input HPI. In some examples, the neural network is retrained when a certain number false labels and/or other feedback data are accrued.
- Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The term “neural network” refers to a computing system or other processor system that learns to perform a task by analyzing pre-classified training examples. Neural networks include a plurality of densely connected processing nodes inspired by the human brain. In certain examples, the nodes of a neural networks can be organized into layers in which data moves in the forward direction (e.g., data in the first layer moves into the second layer, data in the second layer moves into the third layer, etc.), for example, to drive one or more outputs based on one or more inputs via correlations (e.g., connections) represented by the nodes and their interconnections. Deep learning and/or machine learning can be implemented via a neural network to process incoming data to generate an output and benefit from feedback to improve its processing. A “recurrent neural network” or “RNN” is a type of neural network in which nodes or cells include loops to allow information to persist over time. Thus, the RNN can leverage reasoning about previous events to inform subsequent processing. In an RNN, a memory or other internal state is used to process input sequence(s) in an element-by-element process wherein an output for each element is dependent on the output of previous and/or other elements (e.g., a directed graph driving a sequence).
- “Long short-term memory” networks or “LSTM” networks are RNNs designed to handle long-term dependencies. Generally, LSTM networks are organized into cells and gates which interact to optimize the output of the network. Information from outside the processing of the current element (e.g., information from previous elements) is stored in gated cells. These gates release information based on the weight of the gates, which are adjusted and optimized during the training phase of the AI. In an LSTM network (or its pared-down variant gated recurrent unit network), the nodes or cells in the network have storage and an associated stored state under control of the neural network to aid in establishing correlations and processing input data.
-
FIG. 1 is an illustration of a system in which the teachings of this disclosure may be implemented. Anexample system 100 includes adata source 102, anHPI classifier 104, andmedical support system 106. In the illustrated example, thedata source 102 outputs anunprocessed HPI 108 which is transferred to the both theHPI classifier 104 and themedical support system 106. TheHPI classifier 104, which is discussed below in greater detail in relation toFIG. 2 , classifies theHPI 108 and outputs anHPI classification 110. In some examples, theHPI classification 110 is a binary output of either “brief” or “extended.” In the illustrated example, themedical support system 106 links and stores theHPI 108 and theHPI classification 110. - The
example data source 102 provides theHPI 108 to theHPI classifier 104. For example, thedata source 102 may be database of previously collected and recorded HPIs. Alternatively, in some examples, thedata source 102 may be a text input (e.g., a keyboard, a speech to text processor, or a digital scanner with text recognition, etc.). In this example, thedata source 102 is used by healthcare professionals, medical support staff, and/or patients to input the HPI. For example, thedata source 102 may be a computer terminal in which a healthcare professional records the patient's answers while conducting a medical history. Additionally or alternatively, thedata source 102 may contain a user interface which issues prompts asking for particular inputs (e.g., “Where is the pain location?,” “How severe is the pain?,” “When do the symptoms occur?,” etc.). In this example, thedata source 102 constructs theHPI 108 from the answers to the provided prompts. - In the illustrated example, the example
medical support system 106 is a digital database which contains the medical history of a patient and a summary of the medical encounters of the patient. The examplemedical support system 106 records both theHPI 108 and theHPI classification 110. The examplemedical support system 106 can be any suitable medical system (e.g., an EMR, a medical billing system, etc.). For example, themedical support system 106 may store theHPI 108 as a text string associated with the patient's related medical encounter. Alternatively, theHPI 108 may recorded in any suitable method (e.g., each word of the HPI is stored as separate string, etc.). In some examples, theHPI classification 110 is stored as an associated binary value to theHPI 108. Alternatively, in some examples, theHPI classification 110 and theHPI 108 may be concatenated together, such that theHPI classification 110 is added to the text string of theHPI 108. In this example, theHPI 108 andHPI classification 110 are stored as a single text string. Alternatively, theHPI classification 110 may be stored by any suitable method. -
FIG. 2 is a block diagram depicting theHPI classifier 104 ofFIG. 1 in greater detail. TheHPI classifier 104 includes anexample preprocessor 202 and an exampleneural network 214. Theexample preprocessor 202 includes an examplenatural language processor 204 and anexample tensor generator 212. The examplenatural language processor 204 includes anexample tokenizer 206, anexample lemmatizer 208, anexample sentence reorderer 209 and an example namedentity recognizer 210. The exampleneural network 214 includes an embeddinglayer 216, anexample LSTM layer 218 and an example fully connectedlayer 220. Additionally, in some examples, theneural network 214 has auxiliary elements to retrain including anexample model evaluator 222, anexample model trainer 224 and anexample model deployer 226. In some examples, theexample HPI classifier 104 includes amedical system interface 228. - The
HPI classifier 104 receives the unprocessed HPI 108 (e.g., from thedata source 102 ofFIG. 1 ). In the illustrated example, theHPI 108 is preprocessed by thepreprocessor 202 beginning in thenatural language processor 204. For example, when theHPI 108 is received by thenatural language processor 204, theexample sentence reorderer 209 shuffles the order of each sentence of theHPI 108 around into a random order. In some examples, reshuffling the order of the sentences of theHPI 108 prevents the order of sentences from effecting the output of theneural network 214. In some examples, because the classification of theHPI 108 is only dependent on the presence of the seven elements and not the ordering of the elements, this removes any potential unintentional effects that sentence ordering may have on theHPI classification 110. - The
tokenizer 206 converts each word or group of words of theHPI 108 into a token. In some examples, thetokenizer 206 breaks theinput HPI 108 string into individual tokens. For example, if thetokenizer 206 encounters the sentence “the quick brown fox jumps over the lazy dog,” thetokenizer 206 would tokenizer the sentence into “the,” “quick,” “brown,” “fox,” “jumps,” “over,” “the,” “lazy,” and “dog.” In some examples, thetokenizer 206 tokenizes theHPI 108 based on a “space” delimiter (e.g., “ ”). In other examples, thetokenizer 206 can tokenize theHPI 108 based on another character, rule, etc. For example, thetokenizer 206 can have special-case rules which allow for certain types of phrases to be tokenized together. For example, if a date (e.g., “Mar. 12, 2018”) is encountered, thetokenizer 206 can tokenize the date into a single token. Additional examples include, thetokenizer 206 can tokenize names and titles together (e.g., “Dr. Smith”) and/or certain medical abbreviations (e.g., “obstructive sleep apnea,” “cardiac arrest,” and “Type 2 diabetes.”). Additionally or alternatively, thetokenizer 206 can tokenize short phrases together based on simple rules. For example, thetokenizer 206 can group together numbers and words following them together (e.g., “4 hours,” and “five ounces”). In some examples, multiword phrases indicating locations can similarly be tokenized together (e.g., “St. George's Hospital” and “Chicago, Ill.”). - The example named
entity recognizer 210 scans thetokenized HPI 108 for numbers, dates, named entities, medical terms, abbreviations, and/or misspelling and replaces these elements with standardized tokens. For example, if the namedentity recognizer 210 identifies the token “Dr. Smith”, the namedentity recognizer 210 replaces the token with a standardized indication token such as “PERSON.” For example, if the namedentity recognizer 210 identifies the token “Mar. 12, 2018”, the namedentity recognizer 210 replaces the token with a token saying “DATE.” Alternatively, the token “Mar. 12, 2018” is replaced with three tokens representing month, day and year, namely “DATE,” “DATE,” and “DATE,” respectively. For example, if the namedentity recognizer 210 identifies the token “St. George's Hospital”, the namedentity recognizer 210 replaces the token with a standardized token such as “FACILITY.” For example, if the namedentity recognizer 210 identifies the token “4 Hours”, the namedentity recognizer 210 replaces the token with a standardized token such as “TIME.” For example, if the namedentity recognizer 210 identifies the token “five ounces”, the namedentity recognizer 210 replaces the token with a token such as “QUANTITY.” For example, if the namedentity recognizer 210 identifies the token “Chicago, Ill.”, the namedentity recognizer 210 replaces the specific token with a standardized token such as “LOC”, “LOCATION”, etc. - In some examples, the named
entity recognizer 210 can replace medical abbreviations, abbreviations and misspellings with a standardized token representing words that are out of vocabulary (e.g., OOV, etc.). In some examples, out of vocabulary words are referenced to a dictionary. In other examples, the namedentity recognizer 210 may have a separate token for medical terms and abbreviations (e.g., “MED.”). In this example, theexample HPI classifier 104 includes a medical dictionary (e.g., Radlex, LOINC, SNOMED, CPT, ICD-10, etc.). In some examples, the namedentity recognizer 210 can replace medical terms and abbreviations with more specific tokens (e.g., separate tokens for medical procedures, medicines and diseases, etc.). For example, the namedentity recognizer 210 can replaced medical terms and abbreviations with tokens relating to specific bodily systems (e.g., “heart stent” could be replaced with a token reflecting a circulatory procedure (e.g., “CIR PRO,” etc.)). - The
example lemmatizer 208 receives the tokens from the named entity recognizer and replaces each token with a lemma associated with the respective token. As used herein, a “lemma” is the dictionary form of a word. In some examples, thelemmatizer 208 replaces inflected verbs with a related base verb. For example, if thelemmatizer 208 encounters token “am,” “are,” or “is,” thelemmatizer 208 can replace the token with “be.” Additionally or alternatively, thelemmatizer 208 can similarly replace inflected noun tokens (e.g., “cars,” “cars',” “car's,” etc.) with their related lemma (e.g., car). In some examples, thelemmatizer 208 can have similarly functionality with other types of words. In some additional examples, thelemmatizer 208 can use a word's context to determine its proper lemma. For example, the word “drawer” can have the lemma “drawer” if the word is user a noun or “draw” if the word is used as a verb. In some examples, thelemmatizer 208 reduces the required complexity of the neural network by reducing the possible number of inputs the network can receive. - In some examples, the
natural language processor 204 outputs a preprocessedHPI 211. In some examples, thelemmatizer 208,sentence reorderer 209, and the namedentity recognizer 210 may not be components within thenatural language processor 204. In these examples, the preprocessedHPI 211 may not be lemmatized, reordered or have its named tokens replaced with standardized or other predefined tags. Alternatively, any suitable type of preprocessing can be performed to generate the preprocessedHPI 211. - The
example tensor generator 212 receives thetokenized HPI 211. Theexample tensor generator 212 receives thetokenized HPI 211 and outputs atensor 213. In some examples, thetensor generator 212 converts each token of thetokenized HPI 211 into a vector. In some examples, the vector is a binary sparse vector in which one dimension (e.g., one index) has a value of “1” and each of the other dimensions are “0.” In some examples, each dimension of the vector represents a different possible token. For example, if thetokenized HPI 211 can be composed from any number of 50,000 different tokens, each vector has 50,000 different dimensions. In this example, if thetokenized HPI 211 is one hundred tokens in length, thetensor generator 212 vectorizes each of the one hundred tokens into a vector. In some examples, theexample tensor 213 includes each of these vectors concatenated (e.g., “stacked”, appended, etc.) together. In some examples, to save memory, thetensor generator 212 vectorizes each token into a scalar value representing the would-be index of sparse value of the associated vector. In this example, thetensor 213 is a vector of these scalar values. - In the illustrated example, the
example tensor 213 is input into theneural network 214. In the illustrated example, theneural network 214 is an LSTM network. Alternatively, theneural network 214 can be implemented using a general RNN, recursive neural network, or any other suitable type of machine learning architecture. In some examples, theneural network 214 can be a part of a larger and/or more complex neural network with additional functions (e.g., identifying the bodily system described in HPI, etc.). In the illustrated example, theneural network 214 outputs a binary output (e.g., the HPI classification 110). In other examples, the output of theneural network 214 can indicate the presence of particular HPI elements (e.g., a location of problem, a quality of problem, etc.) in theinput HPI 108. - In the illustrated example, the first layer of the
neural network 214 is an embeddinglayer 216 to prepare tensor(s) 213 for processing by the layers of theneural network 214. In the illustrated example, the embeddinglayer 216 converts each vectorized token of thetensor 213 into a dense vector corresponding to that token. In some examples, the number of dimensions of the embeddinglayer 216 corresponds to the length of the dense vector created by the embeddinglayer 216. In some examples, adding more dimensions to the embeddinglayer 216 increases the accuracy and robustness of theneural network 214. In some examples, each unique sparse vector of thetensor 213 is embedded to a specific corresponding dense vector by the embeddinglayer 216. For example, if the same vector (e.g., [522]) appears twice in the tensor, that sparse vector is mapped to the same dense vector. In some examples, the specific values of the dimensions of the embedded dense vectors are optimized during the training process of theneural network 214. The embeddeddense vectors 217 are input to theLSTM layer 218 of the neural network. Thus, for example, a 2D tensor can be transformed into a 3D tensor via the embeddinglayer 216 as input to the RNN to determine a feature output (e.g., brief/extended, etc.). - The
example LSTM layer 218 receives the embeddeddense vectors 217 output by the embeddinglayer 216 and outputs asingle output vector 219 of a predetermined length. In some examples, the dimensions of theLSTM layer 218 correspond to the length of theoutput vector 219. In some examples, theLSTM layer 218 uses a soft-sign activation function. Alternatively, any suitable activation function may be used (e.g., a hyperbolic tangent (tanh) activation function, etc.). In some examples, the operations of theLSTM layer 218 are optimized during the training of theneural network 214. TheLSTM layer 218 leverages history or learned recognition of language, words, phrases, patterns, etc., in theinput vectors 217 using information stored in recurrent gates from prior visible and/or hidden cells in theLSTM layer 218 to arrive at theoutput vector 219 based on the combination of information in the vector(s) 217. An LSTM unit in theLSTM layer 218 receives input state, hidden state, and cell state information and processes the input information using one or more gates including sigmoid, hyperbolic tangent, etc., to apply weighted and/or unweighted element-wise addition and/or multiplication to the vector elements and produce an output state. Via theLSTM 218, some information can be stored and/or conveyed from one cell to another via the output state and other information can be discarded or “forgotten” to rid the model of old or outdated information. - The
output vector 219 of theLSTM layer 218 is input into the fully connectedlayer 220. In the illustrated example, the fully connectedlayer 220 has a single dimension with a binary output indicating if theHPI 108 is “brief” or “extended.” Alternatively, if theneural network 214 has additional outputs (e.g., determining which bodily system, such as endocrine system, renal system, etc., is described in the HPI, the presence of particular HPI elements, etc.), the fully connectedlayer 220 can have additional dimensions. In the illustrated example, the fully connectedlayer 220 uses a sigmoid activation function. In some examples, the output vector of theLSTM layer 218 is linearized by matrix multiplication. In this example, this scalar value is then rounded to either “0” or “1,” which are associated with either “brief” or “extended,” respectively. In the illustrated example, the binary output value generated by the fully connectedlayer 220 is theHPI classification 110. In other examples, the fully connectedlayer 220 can generate theHPI classification 110 by any other suitable function. In some examples, the function of the fully connectedlayer 220 is optimized during the training of theneural network 214. - In the illustrated example, the
neural network 214 can be periodically retrained (e.g., based on a threshold of feedback, at the discretion of an operator of the system, quarterly, etc.). In some examples, the neural network is automatically retrained after a certain threshold of incorrectly classified HPIs are accumulated (e.g., the amodel evaluator 222 determines that too many HPIs have been mis-classified so the model should be updated to improve classification accuracy, etc.). To retrain theneural network 214, themodel evaluator 222, amodel trainer 224 and amodel deployer 226 are used. Theexample model evaluator 222 monitors and evaluates theoutput HPI classifications 110 of theneural network 214. In some examples, if a healthcare professional notices and records anincorrect HPI classification 110, themodel evaluator 222 notes the error of the misclassifiedHPI 108 along with the correct HPI classification. In some examples, another system and/or application, such as a billing system, computer-aided diagnosis system, quality control processor, etc., flags and/or otherwise identifies anincorrect HPI classification 110, which can be noted by themodel evaluator 222 along with the correct classification. In some examples, themodel evaluator 222 can monitor a government and/or third-party process that rejects an HPI and/or associated medical record due to HPI misclassification. In some examples, when the model evaluator reaches a threshold in of feedback, themodel evaluator 222 triggers or otherwise instructs themodel trainer 224 to begin training a new neural network (e.g., to replace the deployed neural network model/construct 214). Additionally or alternatively, themodel evaluator 222 can periodically (e.g., quarterly, yearly, etc.) trigger themodel trainer 224 to begin training a new neural network. In some examples, themodel evaluator 222 also monitors for positive feedback (e.g., a human, system, process, etc., verifying that an HPI was correctly classified and can be used). - The
example model trainer 224 trains a new, updated, or different neural network model/other construct to replace the currently deployedneural network 214. For example, themodel trainer 224 can use the positive or/and negative feedback compiled by themodel evaluator 222 to create a new data set of HPIs to train and/or test the new neural network. In some examples, themodel trainer 224 can use previous training/testing data (e.g., pre-classified HPIs used to train the neural network 214) in conjunction with the newly constructed training/testing data set (e.g., pre-classified HPIs not used to train the neural network 214). In some examples, themodel trainer 224 iteratively varies a strength of connection between the nodes/units of the neural network until the newly trained model set achieves a desired accuracy (e.g., the new neural network correctly classifies the previously incorrectly classified HPIs of the training set). In some examples, the model train then uses a separate test set of HPIs to validate the accuracy of the newly trained neural network. If the result of this validation satisfies specified criterion(-ia), themodel trainer 224 outputs the newly trained neural network to themodel deployer 226. Theexample model deployer 226 deploys the trained neural network model. For example, themodel deployer 226 makes the strength of connections between nodes of the neural network rigid (e.g., not able to change). Once themodel deployer 226 has made the newly trained neural network static, themodel deployer 226 replaces theneural network 214 with the newly trained neural network as the deployed neural network 214 (e.g., the deployed model) to be used in classification of incoming medical data. - The example
medical system interface 228 modifies a medical support system (e.g., themedical support system 106 ofFIG. 1 ) with theHPI classification 110. In some examples, the examplemedical system interface 228 may trigger a response from themedical support system 106. For example, if themedical support system 106 is an EMR, themedical system interface 228 may cause theHPI classification 110 to be associated with a patient's medical history. In other examples, if themedical support system 106 is a medical billing system, themedical system interface 228 may trigger a medical bill to be generated based on the classified HPI (e.g., a more expensive bill may be issued if theHPI classification 110 is an extended). - While an example implementation of the
HPI classifier 104 ofFIG. 1 is illustrated inFIG. 2 , one or more of the elements, processes, and/or devices illustrated inFIG. 4 can be combined, divided, re-arranged, omitted, eliminated, and/or implemented in any other way. Further, theexample preprocessor 202, the examplenatural language processor 204, theexample tokenizer 206, theexample lemmatizer 208, theexample sentence reorderer 209, the example namedentity recognizer 210, theexample tensor generator 212, the exampleneural network 214, theexample embedding layer 216, theexample LSTM layer 218, the example fully connectedlayer 220, themedical system interface 228 and/or, more generally, theexample HPI classifier 104 ofFIG. 1 can be implemented by hardware, software, firmware, and/or any combination of hardware, software, and/or firmware. Thus, for example, any of theexample preprocessor 202, the examplenatural language processor 204, theexample tokenizer 206, theexample lemmatizer 208, theexample sentence reorderer 209, the example namedentity recognizer 210, theexample tensor generator 212, the exampleneural network 214, theexample embedding layer 216, theexample LSTM layer 218, the example fully connectedlayer 220, and/or, more generally, the example AI can be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example,example preprocessor 202, the examplenatural language processor 204, theexample tokenizer 206, theexample lemmatizer 208, theexample sentence reorderer 209, the example namedentity recognizer 210, theexample tensor generator 212, the exampleneural network 214, theexample embedding layer 216, theexample LSTM layer 218, the example fully connectedlayer 220, themedical system interface 228 is/are hereby expressly defined to include a non-transitory computer readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including the software and/or firmware. Further still, theexample HPI classifier 104 ofFIG. 1 can include one or more elements, processes, and/or devices in addition to, or instead of, those illustrated inFIG. 2 , and/or may include more than one of any or all of the illustrated elements, processes, and devices. As used herein, the phrase “in communication,” including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events. - Flowcharts representative of example hardware logic or machine readable instructions for implementing the
HPI classifier 104 ofFIG. 1 are shown inFIGS. 6-9 . The machine readable instructions can be a program or portion of a program for execution by a processor such as the processor 1012 shown in theexample processor platform 1100 discussed below in connection withFIG. 10 . The program can be embodied in software stored on a non-transitory computer readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor 1012, but the entire program and/or parts thereof can alternatively be executed by a device other than the processor 1012 and/or embodied in firmware or dedicated hardware. Further, although the example programs are described with reference to the flowcharts illustrated inFIGS. 6-9 , many other methods of implementing theexample HPI classifier 104 can alternatively be used. For example, the order of execution of the blocks can be changed, and/or some of the blocks described can be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks can be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware. - As mentioned above, the example processes of
FIGS. 6-9 can be implemented using executable instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer and/or machine readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory, and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media. - “Including” and “comprising” (and all forms and tenses thereof) are used herein to be open ended terms. Thus, whenever a claim employs any form of “include” or “comprise” (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within a claim recitation of any kind, it is to be understood that additional elements, terms, etc. may be present without falling outside the scope of the corresponding claim or recitation. As used herein, when the phrase “at least” is used as the transition term in, for example, a preamble of a claim, it is open-ended in the same manner as the term “comprising” and “including” are open ended. The term “and/or” when used, for example, in a form such as A, B, and/or C refers to any combination or subset of A, B, C such as (1) A alone, (2) B alone, (3) C alone, (4) A with B, (5) A with C, and (6) B with C.
-
FIG. 3 depicts an example data flow diagram 300 of anHPI 302 through theHPI classifier 104 ofFIGS. 1 and 2 . In the illustrated example, the example HPI 302 (e.g., thenarrative HPI 108 ofFIG. 2 ) is input to theHPI classifier 104. In this example, theexample HPI 302 begins with the phrase “THIS IS A 66 YEARS OLD FEMALE.” Once received by theHPI classifier 104, theHPI 302 is received by thepreprocessor 202. Thepreprocessor 202 converts theexample HPI 302 into an example preprocessedHPI 304. To create the preprocessedHPI 304, thepreprocessor 202 can tokenize the HPI 302 (e.g., using the tokenizer 206), lemmatize the HPI 302 (e.g., using the lemmatizer 208), reorder the sentences of the HPI 302 (e.g., using the sentence reorderer 209), and/or replace named entities recognized in the HPI 302 (e.g., using the named entity recognizer 210). For example, the beginning phrase “THIS IS A 66 YEARS OLD FEMALE” of theHPI 302 is tokenized into “THIS,” “IS,” “66” “A YEARS” “OLD,” and “FEMALE” by thetokenizer 206. In the illustrated example, thelemmatizer 208 then replaces the “IS” token with the token's lemma, “BE.” Additionally, the named entity recognizer then replaces each of “66,” “A YEARS,” and “OLD” with a named entity token of “DATE.” The beginning phrase of theexample HPI 302 has been preprocessed into the example preprocessedHPI 304 that includes the tokens “THIS,” “BE,” “DATE,” “DATE,” “DATE,” and “FEMALE.” The preprocessedHPI 304 is then sent to thetensor generator 212. - At the
tensor generator 212, the preprocessedHPI 304 is converted into an examplesparse tensor 306. In the illustrated example, each token of the preprocessed HPI is one-hot encoded into a sparse vector. For example, if the number of possible tokens than included in the preprocessed HPI is 50,000, each sparse vector is 50,000 dimensions in length. In the illustrated example, to save memory, each sparse vector is stored as a scalar value in theexample tensor 306 where each scalar value represents the dimension of the sparse vector. For example, the token “THIS” is associated with the 15,220th dimension of the sparse vector and is stored in theexample tensor 306 as “15,220.” The tokens “BE” and “FEMALE” are similarly stored as “5,560” and “42,284” respectively. Additionally, because each possible dimension of the sparse vectors is associated with a specific token, each of the “DATE” tokens is stored as “387” in thesparse tensor 306. Once each token has been converted into a sparse vector and added to the examplesparse tensor 306, thesparse tensor 306 is input into the embeddinglayer 216. - At the embedding
layer 216, each element of thesparse tensor 306 is converted into an exampledense tensor 308. In the illustrated example, each element of the example sparse tensor 306 (e.g., the sparse vectors) is embedded into a corresponding dense vector. In some examples, the mathematical process of this conversion is optimized during the training of the model (e.g., theneural network 214 ofFIG. 2 ). In the illustrated example, repeat elements of the sparse tensor 306 (e.g., “387”) are embedded as the same dense vector (e.g., [66, 37, 672, . . . , 31]). In some examples, a length of each of the vectors in thedense vector 308 is equal to a number of dimensions in the embeddinglayer 216. Once thedense vector 308 has been generated, thedense vector 308 is input into theLSTM Layer 218. - The
example LSTM Layer 218 converts thedense vector 308 into anoutput vector 310 using a softsign activation function. In some examples, the mathematical process of this conversion is optimized during the training of the model (e.g., theneural network 214 ofFIG. 2 ). In some examples, the length of theoutput vector 310 corresponds to the number of dimensions of theLSTM layer 218. Theexample output vector 219 is then input into the fully connectedlayer 220. The example fully connectedlayer 220 linearizes theexample output vector 310 into an examplebinary output 312 using a sigmoid function. In some examples, thebinary output 312 corresponds to theHPI classification 110. -
FIG. 4 shows an example confusion table 400 which includes example results of a group of human classified HPIs classified using by a deployedHPI classifier 104 ofFIG. 1 . A confusion table is a metric used to visualize performance of an algorithm or neural network. In the illustrated example, agrading index 402 depicts a greyscale gradient of example percentages the results can have. A y-axis 404 represents the “True Label” (e.g., labeled by humans) of an HPI, and anx-axis 406 represents the “Predicted Label” of HPI as classified by theHPI classifier 104. In the illustrated example, 91% of “brief” HPIs were correctly classified as such by theHPI classifier 104 whereas 9% were incorrectly classified as “extended.” Similarly, in the illustrated example, 6% of “extended” HPIs were erroneous classified as “brief” by theHPI classifier 104 whereas 94% were correctly classified as “extended.” -
FIG. 5 is a process flow diagram of anexample process 500 to deploy theHPI classifier 104 ofFIG. 1 to be used to classify incoming patient HPIs. Theprocess 500 begins atblock 502. Atblock 502, thedata source 102 ingests a set of preclassified HPIs to be used to train the neural network (e.g., theneural network 214 ofFIG. 2 ) of theHPI classifier 104. In some examples, each HPI of the set of HPIs is classified by a healthcare professional prior to being ingested by thedata source 102. In this example, theHPI classification 110 is ingested with theexample HPI 108. Atblock 504, thepreprocessor 202 preprocesses the ingested data. Additional detail in the function of the preprocessor is provided below in conjunction withFIG. 7 . In some examples, thepreprocessor 202 also preprocesses theHPI classification 110. - At
block 506, the model (e.g., the neural network 214) is trained using the preprocessed HPI(s) 108 and HPI classification(s) 110 (e.g., collectively referred to as the samples). In some examples, the samples are processed iteratively in epochs until the model converges. In some examples, the samples are divided such that are some of the samples are used for training and some are used for validation (e.g., confirming the model works after training). Known outcomes/results can be used to verify performance of the training model, which can also be validated with a test data set. In some examples, a set of known, “gold standard”, “truthed”, or other reference data can be divided into a training data set to train the model and a test data set to test the trained network model to validate its correct operation. After the model has been trained and validated, theprocess 500 advances to block 508. - At
block 508, the model is deployed. In some examples, the model is deployed as application within a medical support system (e.g., themedical support system 106 ofFIG. 1 ), billing software, computer-aided diagnosis (CAD) application, etc. In other examples, the model is a standalone application. In some examples, when theprocess 500 is being used to retrain the model, the model is to replace a previously used model so a prior deployed model is replaced with the newly deployed model in the target system. - At
block 510, after the model has been deployed, themodel evaluator 222 monitors the monitor for potential misclassifications. In some examples, the model evaluator keeps a database of improperly classified HPIs. Classified HPIs can be confirmed as properly or improperly classified through user feedback, other system evaluation (e.g., a billing system determines that an HPI is not in fact extended, etc.), etc. Such feedback can be used to trigger a subsequent retraining of the model (e.g., when a number or percentage or improper classifications reaches or exceeds a threshold, criterion, etc.), for example. -
FIG. 6 is an illustration of an example data flow and transformation ofinformation 600 flowing through theexample data source 102, theexample HPI classifier 104 and the examplemedical support system 106 of the example system ofFIG. 1 . As shown in the illustrated example, an example transmission 602 (e.g., a message, an instruction, a data packet, etc.) is sent from thedata source 102 to the example 604. Theexample transmission 602 includes an unprocessed HPI (e.g., theHPI 108 ofFIG. 1 ). Additionally or alternatively, the example transmission can include any other suitable information (e.g., information about thedata source 102, about an associate patient, etc.). Theexample HPI classifier 104 can use the data included in thetransmission 602 to execute anexample action 604. Theexample action 604 can include classifying the unprocessed HPI of thetransmission 602. In response to theaction 604, theHPI classifier 104 can transmit anexample transmission 606 including an HPI classification tomedical support system 106 and/or an exampleHPI classification message 608 back to thedata source 102. - More specifically, as shown in the example of
FIG. 6 , theexample transmission 606 and theexample transmission 608 include an HPI classification. In some examples, thetransmission 606 and/or theexample transmission 608 can further include theHPI 108. Theexample transmission 608 can notify thedata source 102 of the HPI classification generated by theHPI classifier 104. Theexample HPI classifier 104 can also transmit anexample transmission 610. In the illustrated example, theexample transmission 610 can include system instructions that can cause themedical support system 106 to undergo anexample action 612. For example, theaction 612 can include generating a bill and/or insurance claim. Additionally or alternatively, themedical system interface 220 can cause themedical support system 106 to generate, update or delete a medical record. In some examples, theaction 612 can include generating, updating or deleting a medical record. In some examples, theaction 612 can include generating a request or reminder for an appointment. Additionally or alternatively, theaction 612 can trigger any other suitable action from themedical support system 106 such as schedule an imaging exam, schedule a laboratory test session, trigger a reminder for clinician follow-up, configure an imaging workstation and/or other clinician computing device for patient data analysis, etc. - The
medical support system 106 can further send anexample transmission 614 to thedata source 102. Theexample transmission 614 can include feedback (e.g., a notification of whether the HPI classification was correct, etc.) for themedical support system 106. Theexample transmission 614 can further include a request to manually classify one or more HPIs that can also be included in theexample transmission 614. Theexample transmission 614 can trigger thedata source 102 to send anexample transmission 616. Theexample transmission 616 can include, for example, a manual classification of an HPI included in theexample transmission 614. Themedical support system 106 can further transmit anexample transmission 618. In the illustrated example, theexample transmission 618 can include training data to be used to by theHPI classifier 104. For example, theexample transmission 618 can include unclassified HPIs stored in themedical support system 106 and/or incorrectly classified HPIs (e.g., incorrectly classified by theHPI classifier 104, incorrectly manually classified, etc.). - The
medical support system 106 can further transmit anexample transmission 620 to theHPI classifier 104. In the illustrated example, theexample transmission 620 can include feedback from themedical support system 106 to theHPI classifier 620. In some examples, theexample transmission 620 can trigger anexample action 622. Theexample action 622 can include retraining the neural network (e.g., theneural network 214 ofFIG. 2 ) of theHPI classifier 104 and can, for example, include executing theprocess 500 ofFIG. 5 . Thus, for example, based on feedback received from themedical support system 106 and/or the data source 102 (e.g., negative feedback regarding incorrect classification results, etc.), theHPI classifier 104 can be triggered to regenerate thenetwork model 300 to be redeployed for further HPI classification, etc. -
FIG. 7 is a flowchart representative of machinereadable instructions 700 which can be executed to implement the HPI classifier ofFIG. 2 . Theprocess 700 ofFIG. 7 begins atblock 702. Atblock 702, thepreprocessor 202 receives thenarrative HPI 108 from thedata source 102. In some examples, thenarrative HPI 108 is retrieved from a database of HPIs. In other examples, theHPI 108 is input by a patient or healthcare professional after conducting a medical with the patient. Alternatively, theHPI 108 can be retrieved from any suitable source. In some examples, theHPI 108 is formatted as one continuous text string. Alternatively, theHPI 108 can be formatted in any way readable by thepreprocessor 202. Once theHPI 108 has been retrieved, theprocess 700 advances to block 704. - At
block 704, thepreprocessor 202 preprocesses theHPI 108. Additional detail in the execution ofblock 704 is provided below in conjunction withFIG. 8 . After theHPI 108 has been preprocessed into thetokenized HPI 211, the process advances to block 706 at which thetensor generator 212 generates atensor 213 of vectorized tokens. For example, thetensor generator 212 can map each of the tokens of thetokenized HPI 108 into a sparse vector and forms atensor 213 by concatenating each of the sparse vectors. Alternatively, the tensor generator can map each the tokens of thetokenized HPI 211 into a scalar value, which corresponds to the dimension of the sparse vector that token would be mapped to. In this example, thetensor generator 212 generates a vector (e.g., the tensor 213) by stacking these values. Additionally or alternatively, any suitable method for generating thetensor 213 can be executed. Once thetensor 213 has been generated, theprocess 700 advances to block 708. - At
block 708, theneural network 214 classifies theHPI 108. Additional detail in the execution ofblock 708 is provided below in conjunction withFIG. 8 . Once theHPI classification 110 has been determined, theprocess 700 advances to block 710. Atblock 710, themedical system interface 220 modifies a medical support system (e.g., the medical support system 106) based on theHPI 110 classified. In some examples, themedical support interface 220 also modifies themedical support system 106 withunclassified HPI 108. In some examples, themedical system interface 220 may trigger an action from themedical support system 106. For example, themedical system interface 220 can trigger, facilitate and/or otherwise cause themedical support system 106 to schedule a lab test (e.g., bloodwork, etc.) and/or a medical procedure. In some examples, themedical system interface 220 can trigger, facilitate, and/or otherwise cause themedical support system 106 to generate a bill and/or an insurance claim. Additionally or alternatively, themedical system interface 220 can trigger, facilitate, and/or otherwise cause themedical support system 106 to generate, update or delete a medical record. In some examples, themedical support interface 220 can trigger, facilitate, and/or otherwise cause themedical support system 106 to generate request and/or reminder for an appointment. In some examples, themedical support interface 220 can trigger, facilitate, and/or otherwise cause themedical support system 106 to issue a reminder to thedata source 102 to retake theHPI 108. Additionally or alternatively, themedical support system 106 can trigger facilitate, and/or otherwise cause any other suitable action from themedical support system 106 such as schedule an imaging exam, trigger a reminder for clinician follow-up, configure an imaging workstation and/or other clinician computing device for patient data analysis, etc. - At
block 712, process control decides whether theneural network 214 needs to be retrained. In some examples, the retraining decision is based on whether as many or more than a threshold of incorrectly labeled HPIs have been accrued. Alternatively, the decision to retrain theneural network 214 can instead base on a time interval (e.g., monthly, yearly, etc.). Additionally or alternatively, theneural network 214 can be retrained based on a user, application, and/or system trigger (e.g., by the by an administrator of themedical support system 106 by a billing system, etc.). If theneural network 214 is to be retrained, theprocess 700 advances to block 714. If theneural network 214 is not to be retrained, theprocess 700 ends. Atblock 714, themodel trainer 224 retrains theneural network 214. Additional detail in the execution ofblock 714 is provided below in conjunction withFIG. 10 . - The
subprocess 800 ofFIG. 8 depicts the execution ofblock 704 ofFIG. 7 in greater detail.Subprocess 800 begins atblock 802. Atblock 802, thesentence reorderer 209 randomly reorders each sentence of theHPI 108. For example, thesentence reorderer 209 parses theinput HPI 108 to determine sentences (e.g., by punctuation, capital, or any other suitable method to parse a text string into sentences). In this example, thesentence reorderer 209 then randomly shuffles the ordering of the sentences in theHPI 108. In some examples, reshuffling theHPI 108 prevents theneural network 214 from being trained to classify theHPI 108 on the ordering of elements instead of their presence. Alternatively, any other suitable method can be used to randomly reorder the sentences of theHPI 108. In some examples, thesentence reorderer 209 can then concatenate the parsed sentences into a single text string. Thesubprocess 800 then advances to block 804. - At
block 804, thetokenizer 206 tokenizes theHPI 108. For example, thetokenizer 206 can parse theHPI 108 into individual tokens. In some examples, thetokenizer 206 tokenizes theHPI 108 by identifying a “space” or “ ” delimiter. In other examples, thetokenizer 206 can tokenizer theHPI 108 by identifying other punctuation, sentence/phrase structure, related terms, etc. In some examples, thetokenizer 206 can have special-case rules which allow for certain types of phrases (e.g., dates, Names, medical terms, etc.) to be tokenized together. Once theHPI 108 has been tokenized, thesubprocess 800 advances to block 806. - At
block 806, thelemmatizer 208 lemmatizes the tokens of theHPI 108. For example, the lemmatizer scans each token and replaces each token with a lemma associated with that token. In some examples, thelemmatizer 208 can leverage a database of words and their associated lemmas. In some examples, thelemmatizer 208 utilizes a simple or a neural network to determine a context of a token. In this example, the context of a token can be used to determine its proper lemma (e.g., the word drawer has multiple lemmas). Alternatively, any suitable method can be used to replace tokens with their lemmas. Thesubprocess 800 then advances to block 808. - At
block 808, the namedentity recognizer 210 replaces the tokens of named entities with predefined tags. For example, the namedentity recognizer 210 parses the lemmatized tokens for any named entities and replaces each named entity with a tag from a database. In some examples, named entities, such as places, people and dates, are replaced with a predetermined tag. In some examples, the namedentity recognizer 210 also replaces misspellings and other tokens that the namedentity recognizer 210 does not recognize with a separate tag indicating the word is out of vocabulary (e.g., “OVV”). Once theHPI 108 has been preprocessed into the preprocessedHPI 211, thesubprocess 800 returns to process 700. - The
subprocess 900 ofFIG. 9 depicts the execution ofblock 708 ofFIG. 7 in greater detail.Subprocess 900 begins atblock 902. Atblock 902, the embeddinglayer 216 embeds thetensor 213 into dense vectors. For example, the embeddinglayer 216 converts each element (e.g., vector or scalar) of thetensor 213 into a dense vector. In some examples, the length (e.g., the number of dimensions) of the dense vector is a fixed and predetermined quantity. In some examples, the particular mathematical process to embed thetensor 213 into dense vectors is optimized and/or otherwise improved during the training of theneural network 214. Once thetensor 213 has been embedded into dense vectors, thesubprocess 900 advances to block 904. - At
block 904, theLSTM layer 218 processes the dense vectors into an activated output vector. In some examples, theLSTM layer 218 uses a soft-sign activation function. In other examples, theLSTM layer 218 uses another suitable activation function (e.g., a hyperbolic tangent function, etc.). In some examples, the particular mathematical process to generate the activated output vector is optimized and/or otherwise improved during the training of theneural network 214. Once the output vector has been generated, thesubprocess 900 advances to block 906. - At
block 906, the fully connectedlayer 220 linearizes the output vector in a binary output. For example, the fully connectedlayer 220 uses a sigmoid activation function and/or matrix multiplication to convert the output vector in binary output. In some examples, the fully connectedlayer 220 linearizes the output vector and then rounds the output into a binary output (e.g., “0” or “1”). In some examples, the binary output is the HPI classification 110 (e.g., “1” corresponds to brief and “0” corresponds to extended). Alternatively, the full connected layer can have multiple outputs which include theHPI classification 110. Once theHPI classification 110 has been generated, thesubprocess 900 advances to block 908. - At
block 908, process control decides whether the fully connectedlayer 220 is to classify the body system(s) described in the input HPI. If the fully connectedlayer 220 is classify the bodily system(s) described in the input HPI, thesubprocess 900 advances to block 910. If the fully connectedlayer 220 is not to be classified, the subprocess ends and returns to process 700. Atblock 910, the fully connectedlayer 220 amends the output to include a bodily system classification. For example, the fully connectedlayer 220 can include a binary output for each notable bodily system (e.g., circulatory, endocrine, lymphatic, etc.). In other examples, the fully connectedlayer 220 can include a probability that the input HPI pertains to a particular bodily system. Once the output has been amended to include a bodily system classification, thesubprocess 900 ends and returns to process 700. - The
subprocess 1000 ofFIG. 10 depicts the execution ofblock 714 ofFIG. 7 in greater detail. Atblock 1002, themodel evaluator 222 evaluates model results (e.g., the output of the neural network 214). For example, themodel evaluator 222 can randomly sample the model results (e.g., the HPI classifications 110) and submit them to be independently evaluated. In this example, themodel evaluator 222 can compiles feedback provided by healthcare professionals and/or third party entities (e.g., an insurance company, the CMS, etc.) on the sampled model results. In some examples, if the independently evaluated model results do not satisfy an accuracy threshold, themodel evaluator 222 can create a new set of pre-classified training and validation HPIs. In other examples, themodel evaluator 222 can create a new set of training and validation HPI regardless of the accuracy the model results. In some examples, themodel evaluator 222 continues to collect feedback until a threshold of feedback has been received. In some examples, themodel evaluator 222 can evaluate the model periodically (e.g., quarterly, etc.). Once a suitable level of feedback has been collected, thesubprocess 1000 advances to block 1004. - At
block 1004, themodel trainer 224 retrains the model using the evaluated results and/or new training set. For example, themodel trainer 224 can divide the collected feedback and/or new training set into a training set of HPIs and a validation set of HPIs. In some examples, themodel trainer 224 can add pre-classified HPIs from previous sets into the training set and/or validation set. In some examples, themodel trainer 224, starting with the currentneural network 214, begins using the training set to iteratively change the strength of connections between nodes in each layer (e.g., the embeddinglayer 216, theLSTM layer 218 and/or fully connectedlayer 220, etc.) until a deserved accuracy of classification is achieved. In this example, after the deserved accuracy is achieved, the validation set of HPIs is used to verify the fidelity of the newly trainer neural network. In some examples, themodel trainer 224 may change the activation functions used by neural network 214 (e.g., change the activation function of theLSTM layer 218 to a Tanh activation function, etc.). Once the newly trained neural network has been deployed, thesubprocess 1000 advances to block 1006. - At
block 1006, themodel deployer 226 deploys the newly trained neural network. For example, themodel deployer 226 can replace the currently usedneural network 214 with the newly trained neural network model/construct. In some examples, themodel deployer 226 makes the connections between nodes of the neural network rigid so they do not change when deployed in theHPI classifier 104. Once the neural network has been replaced, thesubprocess 1000 ends and returns to theprocess 700. -
FIG. 11 is a block diagram of anexample processor platform 1100 structured to execute the instructions ofFIGS. 7-10 to implement theHPI classifier 104 ofFIG. 2 . Theprocessor platform 1100 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cell phone, a smart phone, a tablet such as an iPad™), a personal digital assistant (PDA), an Internet appliance, a DVD player, a CD player, a digital video recorder, a Blu-ray player, a gaming console, a personal video recorder, a set top box, a headset or other wearable device, or any other type of computing device. - The
processor platform 1100 of the illustrated example includes aprocessor 1112. Theprocessor 1112 of the illustrated example is hardware. For example, theprocessor 1112 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor based (e.g., silicon based) device. In this example, theprocessor 1112 implements theexample preprocessor 202, the examplenatural language processor 204, theexample tokenizer 206, theexample lemmatizer 208, theexample sentence reorderer 209, the example namedentity recognizer 210, theexample tensor generator 212, the exampleneural network 214, theexample embedding layer 216, theexample LSTM layer 218, and the example fully connectedlayer 220. - The
processor 1112 of the illustrated example includes a local memory 1113 (e.g., a cache). Theprocessor 1112 of the illustrated example is in communication with a main memory including avolatile memory 1114 and anon-volatile memory 1116 via abus 1118. Thevolatile memory 1114 may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®), and/or any other type of random access memory device. Thenon-volatile memory 1116 may be implemented by flash memory and/or any other desired type of memory device. Access to themain memory - The
processor platform 1100 of the illustrated example also includes aninterface circuit 1120. Theinterface circuit 1120 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), a Bluetooth® interface, a near field communication (NFC) interface, and/or a PCI express interface. - In the illustrated example, one or
more input devices 1122 are connected to theinterface circuit 1120. The input device(s) 1122 permit(s) a user to enter data and/or commands into theprocessor 1112. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, an isopoint device, and/or a voice recognition system. - One or
more output devices 1124 are also connected to theinterface circuit 1120 of the illustrated example. Theoutput devices 1124 can be implemented, for example, by display devices (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switching (IPS) display, a touchscreen, etc.), a tactile output device, a printer, and/or speaker. Theinterface circuit 1120 of the illustrated example, thus, typically includes a graphics driver card, a graphics driver chip, and/or a graphics driver processor. - The
interface circuit 1120 of the illustrated example also includes a communication device such as a transmitter, a receiver, a transceiver, a modem, a residential gateway, a wireless access point, and/or a network interface to facilitate exchange of data with external machines (e.g., computing devices of any kind) via anetwork 1126. The communication can be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular telephone system, etc. - The
processor platform 1100 of the illustrated example also includes one or moremass storage devices 1128 for storing software and/or data. Examples of suchmass storage devices 1128 include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives. - The machine
executable instructions 1132 ofFIGS. 7-10 may be stored in themass storage device 1128, in thevolatile memory 1114, in thenon-volatile memory 1116, and/or on a removable non-transitory computer readable storage medium such as a CD or DVD. - From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed that classify medical data using an artificial intelligence. The disclosed examples offer several advantages over manually classified HPI. The disclosed examples improve medical data processing for computer-aided diagnosis, billing, treatment approval, and other patient safety and patient care. The disclosed examples improve operation of healthcare data processors by correctly and efficiently processing a variety of available information and generating a consistent, accurate result. The disclosed examples decrease the probability of denied reimbursement due to incorrect HPI classification.
- On a broader scale, automatic HPI classification can be part of Clinical Documentation Improvement (CDI). Successful CDI programs facilitate the accurate representation of a patient's clinical status that translates into coded data. Coded data is then translated into quality reporting, physician report cards, reimbursement, public health data, patient care plan, and disease tracking and trending.
- This written description uses examples to disclose the invention, including the best mode, and also to enable any person skilled in the art to practice the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal language of the claims.
- Although certain example methods, apparatus and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/136,992 US20190287685A1 (en) | 2018-03-16 | 2018-09-20 | String classification apparatus and methods using artificial intelligence |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201862644117P | 2018-03-16 | 2018-03-16 | |
US16/136,992 US20190287685A1 (en) | 2018-03-16 | 2018-09-20 | String classification apparatus and methods using artificial intelligence |
Publications (1)
Publication Number | Publication Date |
---|---|
US20190287685A1 true US20190287685A1 (en) | 2019-09-19 |
Family
ID=67904560
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/136,984 Active 2039-03-12 US10784000B2 (en) | 2018-03-16 | 2018-09-20 | Medical system interface apparatus and methods to classify and provide medical data using artificial intelligence |
US16/136,992 Pending US20190287685A1 (en) | 2018-03-16 | 2018-09-20 | String classification apparatus and methods using artificial intelligence |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/136,984 Active 2039-03-12 US10784000B2 (en) | 2018-03-16 | 2018-09-20 | Medical system interface apparatus and methods to classify and provide medical data using artificial intelligence |
Country Status (1)
Country | Link |
---|---|
US (2) | US10784000B2 (en) |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10678830B2 (en) * | 2018-05-31 | 2020-06-09 | Fmr Llc | Automated computer text classification and routing using artificial intelligence transfer learning |
CN111259111A (en) * | 2020-01-13 | 2020-06-09 | 安徽科大讯飞医疗信息技术有限公司 | Medical record-based decision-making assisting method and device, electronic equipment and storage medium |
US10706534B2 (en) * | 2017-07-26 | 2020-07-07 | Scott Anderson Middlebrooks | Method and apparatus for classifying a data point in imaging data |
US20200234084A1 (en) * | 2018-08-06 | 2020-07-23 | Functionize, Inc. | Training a system to perform a task with multiple specific steps given a general natural language command |
CN111866192A (en) * | 2020-09-24 | 2020-10-30 | 汉桑(南京)科技有限公司 | Pet interaction method, system and device based on pet ball and storage medium |
US10860804B2 (en) * | 2018-05-16 | 2020-12-08 | Microsoft Technology Licensing, Llc | Quick text classification model |
US20200411200A1 (en) * | 2019-06-25 | 2020-12-31 | Fuji Xerox Co., Ltd. | Information processing apparatus and non-transitory computer readable medium |
US10898149B2 (en) * | 2018-12-11 | 2021-01-26 | International Business Machines Corporation | Standardizing breast density assessments |
US20210174380A1 (en) * | 2019-10-30 | 2021-06-10 | Veda Data Solutions, Inc. | Efficient data processing to identify information and reformant data files, and applications thereof |
US20210209425A1 (en) * | 2020-01-03 | 2021-07-08 | Mayachitra, Inc. | Deep learning methods for event verification and image re-purposing detection |
US11108645B2 (en) | 2019-11-04 | 2021-08-31 | Hewlett Packard Enterprise Development Lp | Device interface matching using an artificial neural network |
US20210365772A1 (en) * | 2020-05-21 | 2021-11-25 | Element Al Inc. | Method of and system for training machine learning algorithm for object classification |
US20210401295A1 (en) * | 2020-06-29 | 2021-12-30 | Aetna Inc. | System and methods utilizing artificial intelligence algorithms to analyze wearable activity tracker data |
US20220058511A1 (en) * | 2020-08-19 | 2022-02-24 | Bank Of America Corporation | Machine learning model builder |
US20220171937A1 (en) * | 2020-11-30 | 2022-06-02 | Industrial Technology Research Institute | Document sentence concept labeling system, training method and labeling method thereof |
US20220284190A1 (en) * | 2021-03-02 | 2022-09-08 | Microsoft Technology Licensing, Llc | Tokenizing alphanumeric text through use of finite state machines |
US11586929B2 (en) * | 2019-02-15 | 2023-02-21 | Wipro Limited | Method and system for optimizing memory requirement for training an artificial neural network model |
EP4210071A1 (en) * | 2022-01-07 | 2023-07-12 | Bayer AG | Machine learning architecture for medical coding |
US11727077B2 (en) | 2021-02-05 | 2023-08-15 | Microsoft Technology Licensing, Llc | Inferring information about a webpage based upon a uniform resource locator of the webpage |
US11748571B1 (en) * | 2019-05-21 | 2023-09-05 | Educational Testing Service | Text segmentation with two-level transformer and auxiliary coherence modeling |
US11748414B2 (en) * | 2018-06-19 | 2023-09-05 | Priyadarshini Mohanty | Methods and systems of operating computerized neural networks for modelling CSR-customer relationships |
JP7355303B2 (en) | 2020-08-20 | 2023-10-03 | 株式会社クオリティライフ・クリエイト | Receipt data significance determination program, receipt data significance determination method, and information processing device |
US11961622B1 (en) | 2022-10-21 | 2024-04-16 | Realyze Intelligence, Inc. | Application-specific processing of a disease-specific semantic model instance |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11714915B2 (en) | 2019-02-01 | 2023-08-01 | Health2047, Inc. | Data aggregation based on disparate local processing of requests |
US11322250B1 (en) | 2019-10-25 | 2022-05-03 | TNacity Blue Ocean LLC | Intelligent medical care path systems and methods |
CN110867231A (en) * | 2019-11-18 | 2020-03-06 | 中山大学 | Disease prediction method, device, computer equipment and medium based on text classification |
KR102429319B1 (en) * | 2020-05-20 | 2022-08-04 | 서울대학교병원 | Method and system for predicting patients needs for hospital resources |
CN111696675B (en) * | 2020-05-22 | 2023-09-19 | 深圳赛安特技术服务有限公司 | User data classification method and device based on Internet of things data and computer equipment |
US11461540B2 (en) | 2020-06-18 | 2022-10-04 | International Business Machines Corporation | Cross-document propagation of entity metadata |
CN111968741B (en) * | 2020-07-15 | 2023-07-18 | 华南理工大学 | Deep learning and integrated learning-based diabetes complication high-risk early warning system |
CN112201360B (en) * | 2020-10-09 | 2023-06-20 | 平安科技(深圳)有限公司 | Method, device, equipment and storage medium for collecting chronic disease follow-up record |
CN112509690B (en) * | 2020-11-30 | 2023-08-04 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for controlling quality |
CN116153452B (en) * | 2023-04-18 | 2023-06-30 | 济南科汛智能科技有限公司 | Medical electronic medical record storage system based on artificial intelligence |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9201869B2 (en) * | 2012-08-28 | 2015-12-01 | Oracle International Corporation | Contextually blind data conversion using indexed string matching |
US10495786B2 (en) * | 2007-02-12 | 2019-12-03 | Locus Energy, Inc. | Weather and satellite model for estimating solar irradiance |
US10891312B2 (en) * | 2012-10-22 | 2021-01-12 | Palantir Technologies Inc. | Sharing information between nexuses that use different classification schemes for information access control |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6915254B1 (en) | 1998-07-30 | 2005-07-05 | A-Life Medical, Inc. | Automatically assigning medical codes using natural language processing |
US7813937B1 (en) | 2002-02-15 | 2010-10-12 | Fair Isaac Corporation | Consistency modeling of healthcare claims to detect fraud and abuse |
US7610192B1 (en) | 2006-03-22 | 2009-10-27 | Patrick William Jamieson | Process and system for high precision coding of free text documents against a standard lexicon |
US10231077B2 (en) * | 2007-07-03 | 2019-03-12 | Eingot Llc | Records access and management |
US8260779B2 (en) * | 2009-09-17 | 2012-09-04 | General Electric Company | Systems, methods, and apparatus for automated mapping and integrated workflow of a controlled medical vocabulary |
US8898798B2 (en) * | 2010-09-01 | 2014-11-25 | Apixio, Inc. | Systems and methods for medical information analysis with deidentification and reidentification |
US8346804B2 (en) | 2010-11-03 | 2013-01-01 | General Electric Company | Systems, methods, and apparatus for computer-assisted full medical code scheme to code scheme mapping |
US10403391B2 (en) | 2012-09-28 | 2019-09-03 | Cerner Health Services, Inc. | Automated mapping of service codes in healthcare systems |
WO2014130749A1 (en) | 2013-02-20 | 2014-08-28 | Vitalware, Llc | Ontological medical coding method, system and apparatus |
EP2985711A1 (en) * | 2014-08-14 | 2016-02-17 | Accenture Global Services Limited | System for automated analysis of clinical text for pharmacovigilance |
US20170228500A1 (en) * | 2016-02-09 | 2017-08-10 | Justin Massengale | Process of generating medical records |
US10325020B2 (en) * | 2017-06-29 | 2019-06-18 | Accenture Global Solutions Limited | Contextual pharmacovigilance system |
-
2018
- 2018-09-20 US US16/136,984 patent/US10784000B2/en active Active
- 2018-09-20 US US16/136,992 patent/US20190287685A1/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10495786B2 (en) * | 2007-02-12 | 2019-12-03 | Locus Energy, Inc. | Weather and satellite model for estimating solar irradiance |
US9201869B2 (en) * | 2012-08-28 | 2015-12-01 | Oracle International Corporation | Contextually blind data conversion using indexed string matching |
US10891312B2 (en) * | 2012-10-22 | 2021-01-12 | Palantir Technologies Inc. | Sharing information between nexuses that use different classification schemes for information access control |
Non-Patent Citations (1)
Title |
---|
Banerjee, I., et al., "Comparative effectiveness of convulutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification," artificial intelligence in medicince 97 (2019) 79-88 (Year: 2019) * |
Cited By (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10706534B2 (en) * | 2017-07-26 | 2020-07-07 | Scott Anderson Middlebrooks | Method and apparatus for classifying a data point in imaging data |
US10860804B2 (en) * | 2018-05-16 | 2020-12-08 | Microsoft Technology Licensing, Llc | Quick text classification model |
US10678830B2 (en) * | 2018-05-31 | 2020-06-09 | Fmr Llc | Automated computer text classification and routing using artificial intelligence transfer learning |
US11748414B2 (en) * | 2018-06-19 | 2023-09-05 | Priyadarshini Mohanty | Methods and systems of operating computerized neural networks for modelling CSR-customer relationships |
US20200234084A1 (en) * | 2018-08-06 | 2020-07-23 | Functionize, Inc. | Training a system to perform a task with multiple specific steps given a general natural language command |
US11645467B2 (en) * | 2018-08-06 | 2023-05-09 | Functionize, Inc. | Training a system to perform a task with multiple specific steps given a general natural language command |
US10898149B2 (en) * | 2018-12-11 | 2021-01-26 | International Business Machines Corporation | Standardizing breast density assessments |
US11586929B2 (en) * | 2019-02-15 | 2023-02-21 | Wipro Limited | Method and system for optimizing memory requirement for training an artificial neural network model |
US11748571B1 (en) * | 2019-05-21 | 2023-09-05 | Educational Testing Service | Text segmentation with two-level transformer and auxiliary coherence modeling |
US20200411200A1 (en) * | 2019-06-25 | 2020-12-31 | Fuji Xerox Co., Ltd. | Information processing apparatus and non-transitory computer readable medium |
US20210174380A1 (en) * | 2019-10-30 | 2021-06-10 | Veda Data Solutions, Inc. | Efficient data processing to identify information and reformant data files, and applications thereof |
US11108645B2 (en) | 2019-11-04 | 2021-08-31 | Hewlett Packard Enterprise Development Lp | Device interface matching using an artificial neural network |
US20210209425A1 (en) * | 2020-01-03 | 2021-07-08 | Mayachitra, Inc. | Deep learning methods for event verification and image re-purposing detection |
US11854113B2 (en) * | 2020-01-03 | 2023-12-26 | Mayachitra, Inc. | Deep learning methods for event verification and image re-purposing detection |
CN111259111A (en) * | 2020-01-13 | 2020-06-09 | 安徽科大讯飞医疗信息技术有限公司 | Medical record-based decision-making assisting method and device, electronic equipment and storage medium |
US20210365772A1 (en) * | 2020-05-21 | 2021-11-25 | Element Al Inc. | Method of and system for training machine learning algorithm for object classification |
US11704558B2 (en) * | 2020-05-21 | 2023-07-18 | Servicenow Canada Inc. | Method of and system for training machine learning algorithm for object classification |
US20210401295A1 (en) * | 2020-06-29 | 2021-12-30 | Aetna Inc. | System and methods utilizing artificial intelligence algorithms to analyze wearable activity tracker data |
US20220058511A1 (en) * | 2020-08-19 | 2022-02-24 | Bank Of America Corporation | Machine learning model builder |
US11966821B2 (en) * | 2020-08-19 | 2024-04-23 | Bank Of America Corporation | Machine learning model builder |
JP7355303B2 (en) | 2020-08-20 | 2023-10-03 | 株式会社クオリティライフ・クリエイト | Receipt data significance determination program, receipt data significance determination method, and information processing device |
CN111866192A (en) * | 2020-09-24 | 2020-10-30 | 汉桑(南京)科技有限公司 | Pet interaction method, system and device based on pet ball and storage medium |
US20220171937A1 (en) * | 2020-11-30 | 2022-06-02 | Industrial Technology Research Institute | Document sentence concept labeling system, training method and labeling method thereof |
US11727077B2 (en) | 2021-02-05 | 2023-08-15 | Microsoft Technology Licensing, Llc | Inferring information about a webpage based upon a uniform resource locator of the webpage |
US20220284190A1 (en) * | 2021-03-02 | 2022-09-08 | Microsoft Technology Licensing, Llc | Tokenizing alphanumeric text through use of finite state machines |
EP4210071A1 (en) * | 2022-01-07 | 2023-07-12 | Bayer AG | Machine learning architecture for medical coding |
US11961622B1 (en) | 2022-10-21 | 2024-04-16 | Realyze Intelligence, Inc. | Application-specific processing of a disease-specific semantic model instance |
Also Published As
Publication number | Publication date |
---|---|
US10784000B2 (en) | 2020-09-22 |
US20190287684A1 (en) | 2019-09-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10784000B2 (en) | Medical system interface apparatus and methods to classify and provide medical data using artificial intelligence | |
US11810671B2 (en) | System and method for providing health information | |
US11288455B2 (en) | Ontologically driven procedure coding | |
US8612261B1 (en) | Automated learning for medical data processing system | |
US20200227175A1 (en) | Document improvement prioritization using automated generated codes | |
De Coster et al. | Comparison and validity of procedures coded with ICD-9-CM and ICD-10-CA/CCI | |
CN105190628B (en) | The method and apparatus for determining the intention of the subscription items of clinician | |
Banerjee et al. | Weakly supervised natural language processing for assessing patient-centered outcome following prostate cancer treatment | |
US20190370387A1 (en) | Automatic Processing of Ambiguously Labeled Data | |
US20230154575A1 (en) | Systems and Methods for Mental Health Care Delivery Via Artificial Intelligence | |
CN110534185A (en) | Labeled data acquisition methods divide and examine method, apparatus, storage medium and equipment | |
US20220293271A1 (en) | Method and system for the computer-assisted implementation of radiology recommendations | |
CN105956412A (en) | System and method for realizing coronary heart disease clinical data collection based on intelligent image-text identification | |
RU2699607C2 (en) | High efficiency and reduced frequency of subsequent radiation studies by predicting base for next study | |
CN112908452A (en) | Event data modeling | |
Bozkurt et al. | Phenotyping severity of patient‐centered outcomes using clinical notes: A prostate cancer use case | |
US11133091B2 (en) | Automated analysis system and method | |
Joo et al. | Neural machine translation–based automated current procedural terminology classification system using procedure text: Development and validation study | |
US11547345B2 (en) | Dynamic neuropsychological assessment tool | |
Solares et al. | Transfer learning in electronic health records through clinical concept embedding | |
Landes et al. | A new public corpus for clinical section identification: MedSecId | |
Wei et al. | Clinicallayoutlm: A pre-trained multi-modal model for understanding scanned document in electronic health records | |
Fowler et al. | Change in healthcare utilisation after surgical treatment: observational study of routinely collected patient data from primary and secondary care | |
CN113658688A (en) | Clinical decision support method based on word segmentation-free deep learning | |
Wu et al. | Adaptive generation of structured medical report using NER regarding deep learning |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: VVC HOLDING CORPORATION, WASHINGTON Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WU, ERIC;HUANG, WEI;SIGNING DATES FROM 20180912 TO 20180918;REEL/FRAME:047384/0447 |
|
AS | Assignment |
Owner name: JPMORGAN CHASE BANK, N.A., ILLINOIS Free format text: FIRST LIEN SECURITY AGREEMENT;ASSIGNORS:VVC HOLDING CORP.;ATHENAHEALTH, INC.;EPOCRATES, LLC;AND OTHERS;REEL/FRAME:048301/0890 Effective date: 20190211 |
|
AS | Assignment |
Owner name: ARES CAPITAL CORPORATION, NEW YORK Free format text: SECOND LIEN SECURITY AGREEMENT;ASSIGNORS:VVC HOLDING CORP.;ATHENAHEALTH, INC.;EPOCRATES, LLC;AND OTHERS;REEL/FRAME:048304/0161 Effective date: 20190211 |
|
AS | Assignment |
Owner name: VVC HOLDING CORP., WASHINGTON Free format text: RELEASE OF SECOND LIEN SECURITY INTEREST;ASSIGNOR:ARES CAPITAL CORPORATION;REEL/FRAME:055291/0421 Effective date: 20210212 Owner name: PRAXIFY TECHNOLOGIES, INC., MASSACHUSETTS Free format text: RELEASE OF SECOND LIEN SECURITY INTEREST;ASSIGNOR:ARES CAPITAL CORPORATION;REEL/FRAME:055291/0421 Effective date: 20210212 Owner name: ATHENAHEALTH, INC., MASSACHUSETTS Free format text: RELEASE OF SECOND LIEN SECURITY INTEREST;ASSIGNOR:ARES CAPITAL CORPORATION;REEL/FRAME:055291/0421 Effective date: 20210212 Owner name: EPOCRATES, LLC, MASSACHUSETTS Free format text: RELEASE OF SECOND LIEN SECURITY INTEREST;ASSIGNOR:ARES CAPITAL CORPORATION;REEL/FRAME:055291/0421 Effective date: 20210212 |
|
AS | Assignment |
Owner name: PRAXIFY TECHNOLOGIES, INC., MASSACHUSETTS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:059111/0757 Effective date: 20220215 Owner name: EPOCRATES, LLC, MASSACHUSETTS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:059111/0757 Effective date: 20220215 Owner name: ATHENAHEALTH, INC., MASSACHUSETTS Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:059111/0757 Effective date: 20220215 Owner name: VVC HOLDING LLC (F/K/A VVC HOLDING CORP.), WASHINGTON Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:JPMORGAN CHASE BANK, N.A.;REEL/FRAME:059111/0757 Effective date: 20220215 Owner name: JPMORGAN CHASE BANK, N.A., AS COLLATERAL AGENT, ILLINOIS Free format text: SECURITY INTEREST;ASSIGNORS:ATHENAHEALTH, INC.;VVC HOLDING LLC;EPOCRATES, LLC;AND OTHERS;REEL/FRAME:059016/0774 Effective date: 20220215 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |