Classifications

• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H15/00—ICT specially adapted for medical reports, e.g. generation or transmission thereof
• • G—PHYSICS
  • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
  • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
  • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
  • G16H50/70—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for mining of medical data, e.g. analysing previous cases of other patients

Definitions

• the disclosure relates generally to the field of natural language processing, and in particular to identifying which medical concepts, from a predefined set of medical concepts, are found in a medical report. More specifically, but not exclusively, the disclosure relates to a method for doing this with radiological reports.
• search algorithms exist for automatically determining whether or not a given concept is found in a given input text.
• the concept could be a search term, a string of words entered into a search engine, and the input text could be text found on one of a large number of web pages that the search engine is searching.
• the search term need not be found in exactly the same form in the input text, in order for the search algorithm to return a positive result, but some of the words in the search term might be missing from the input text, or the words of the search term might be found in a different order, and/or with other words between them, in the input text.
• the search algorithm may calculate a score for that search term and input text, indicating how good the match is, and when the search engine is finished searching the web pages that it is searching, it may provide a list of the web pages for which a positive result was found for that search term, ranked in order of the score.
• ElasticSearch is a commercially available search engine described at ⁇ https://www(dot)elastic(dot)co/products/elasticsearch>, with a user guide found at
• ElasticSearch analyzes text fields, such as the title fields of documents, by using different types of queries to calculate a relevance score of a document to the search term.
• a“match” query the text field is considered a match to the search term if more than a minimum number or more than a minimum percentage of words in the search term are found in the text field, and the“match” relevance score is higher if more of the words in the search term are found in the text field.
• The“match” query of ElasticSearch is an example of a“bag of words” method in which the order of words in the input text field does not matter.
• the text field is considered a match to the search term if all of the words of the search term are found in the text field, and if the relative positions of the words in the search term are not too far apart from the relative positions of the same words in the text field. How far apart the positions of the words are means how many changes must be made in the positions of the words in the search term, each time changing the position of a word by 1 in either direction, in order to make the words have the same positions as they have in the text field. For example, if the search term consists of two words, and those two words are found adjacent to each other but in reverse order in the text field, then the number of changes in position is 2, because each of the two words must undergo one change in position.
• the required number of changes in position is n.
• the required number of changes in position must be no greater than a maximum number, a parameter called“slop.”
• The“match_phrase” query may have a higher relevance score if the required number of changes in position is smaller.
• The“match” relevance score and the“match_phrase” relevance score may be added together to obtain an overall relevance score of the text field to the search term.
• ElasticSearch has an English stemming feature that does this for English text.
• the SNOMED CT medical dictionary found at ⁇ https://browser(dot)ihtsdotools(dot)org>, has a structured list of a large number of medical concepts, including body parts, diseases, and symptoms that can be used to diagnose diseases. Each concept is defined by a string of one or more words. In some cases multiple concepts refer to the same thing, in which case a preferred term and alternative terms are given. Concepts may have parents and children, referring to more general and more specific concepts, for example lung is a parent of lung part , which is a parent of lobe of lung. Disease concepts have a finding site relationship with a body part concept, for example lung cancer has a finding site relationship with lung.
• An aspect of some embodiments of the invention concerns a method for determining which of a set of medical concepts pertain to an input text from a medical report, in which a concept is considered a possible match to the input text if a sufficiently large number of pairs of words in the concept are both found in the input text sufficiently close to each other.
• a method of determining which concepts in a set of medical concepts pertain to an input text automatically by executing instructions on a computer, the method comprising:
• each query being a string of two words that is a sub-string of the words in the concept, in the same order as the words in the concept;
• the method also comprises calculating a match score at least for each concept in the selected sub-set that has two or more words, of which more than a minimum number of the words are in the input text, and for which more than the minimum number of the queries have all of their words in the input text sufficiently close together according to the criterion, the match score indicating a degree of matching between the concept and the input text according to a bag-of-words method, and wherein calculating the score for each concept comprises calculating the score depending on the match score for that concept as well as on the sub-scores of the queries for that concept.
• calculating the score for each concept comprises calculating a weighted sum of the match score and the sub-scores for the queries.
• the minimum number of words is 2 for concepts with two words, 2 or 3 for concepts with three words, 2, 3 or 4 for concepts with four words, 3 or 4 for concepts with five words, and 3, 4 or 5 for concepts with six words.
• the method also comprises assigning a score to the concepts that have only one word, when the one word is found in the input text, wherein the rules that depend on the score of the concept are applied both to the concepts with only one word and to the concepts with two or more words.
• the one or more rules specify that when two concepts for which scores have been calculated have sufficiently great overlap in their words, then the concept with a lower score does not pertain to the input text, and that a concept does pertain to the input text if it has a calculated score that is higher than the score calculated for any other concept with which it has sufficiently great overlap in its words.
• the first minimum number is between 35% and 65% of the number of queries created for that concept.
• calculating the score comprises calculating a weighted sum of the sub-scores of the queries, with lower weight given to queries with words that are further apart in the concept.
• the selected sub-set of concepts excludes at least those concepts for which one or more words defined as mandatory words for that concept are not found in the input text.
• the word in the concept that is rarest among all the words in the set of concepts is defined as a mandatory word.
• any singular word in the concept that is a name of a disease is defined as a mandatory word.
• the selected sub-set of concepts excludes at least those concepts which include a word for a body part and a word describing a location or direction of the body part, for which the word that describes the location or direction of a body part is more than one word away in the input text from the word for the body part.
• the criterion for words in the query being sufficiently close together in the input text specifies a maximum distance between the words that is lower for words that are not in the same order in the input text as they are in the query, and that is higher for words that are spaced further apart in the concept than for words that are spaced closer together in the concept.
• the maximum distance for words that are adjacent in the concept and are in the same order in the query and in the input text is between 10 and 25.
• the method also comprises preparing the set of concepts and preprocessing the input text, comprising:
• a computer storage product having at least one computer storage medium having instructions stored therein causing one or more computers to perform an exemplary method of the invention.
• a computer storage medium having instructions stored therein for causing a computer to perform an exemplary method of the invention.
• a computer product embodied in a computer readable medium for performing the steps of an exemplary method of the invention.
• a system for automatically identifying which concepts in a set of medical concepts are found in a medical report comprising:
• a processor module with access to the input texts and to the second database, configured, for each input text, to:
• each query being a string of two words that is a sub-string of the words in the concept, in the same order as the words in the concept;
• the method also includes calculating a score, for each concept that is a possible match to the input text, indicating a degree of matching between the concept and the input text, wherein at least some of the one or more rules depend on the score of the concept and the scores of any other concepts that are possible matches to the input text.
• a word of the concept that is least common among all the concepts in the set is specified to be a mandatory word for that concept.
• a word of a concept that is a one -word term for a disease is specified to be a mandatory word for that concept.
• both words are specified to be mandatory words for that concept, and the criteria further require that the words are adjacent to each other in the input text.
• a method of determining which concepts in a set of medical concepts pertain to an input text automatically by executing instructions on a computer, the method comprising:
• calculating the score comprises calculating a match score, calculating a match_phrase score, and combining the match score and the match_phrase score to obtain the score, the match score depending on how many of the words in the concept are found in the input text but not on the order of those words in the input text, and the match_phrase score depending both on how many of the words in the concept are found in the input text, and on the order of those words in the input text.
• a system for automatically identifying which concepts in a set of medical concepts are found in a medical report comprising:
• a processor module with access to the input texts and to the second database, configured, for each input text, to:
• a system for automatically identifying which concepts in a set of medical concepts are found in a medical report comprising:
• a processor module with access to the input texts and to the second database, configured, for each input text, to:
• Implementation of the method and/or system of embodiments of the invention can involve performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of embodiments of the method and/or system of the invention, several selected tasks could be implemented by hardware, by software or by firmware or by a combination thereof using an operating system.
• a data processor such as a computing platform for executing a plurality of instructions.
• the data processor includes a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard-disk and/or removable media, for storing instructions and/or data.
• a network connection is provided as well.
• a display and/or a user input device such as a keyboard or mouse are optionally provided as well.
• FIG. 1 A shows a block diagram for a system for identifying medical concepts in medical reports, according to an exemplary embodiment of the invention
• FIG. IB shows a high level flowchart of a method used by the processor module of the system of FIG. 1 A, according to any exemplary embodiment of the invention
• FIG. 2 shows a more detailed flowchart of the method used by the processor module of the system of FIG. 1A, according to an exemplary embodiment of the invention.
• FIGS. 3, 4 and 5 show high level flowcharts of methods used by the processor module of the system of FIG. 1 A, according to different exemplary embodiments of the invention.
• the disclosure relates generally to the field of natural language processing, and in particular to identifying which medical concepts, from a predefined set of medical concepts, are found in a medical report. More specifically, but not exclusively, the disclosure relates to a method for doing this with radiological reports.
• the systems including computer systems for processing medical reports) and/or methods described herein (e.g., code instructions executed by one or more processors) address the technical problem of automatically identifying the medical concepts that are mentioned in a medical report on a patient, with fewer missed concepts, and fewer false positives, than prior art methods of automatically identifying medical concepts in a medical report.
• the systems and methods solve this problem by one or more of: 1) not requiring all words of a concept to be present in an input text in order to consider the concept a possible match, even when doing a “match_phrase” test where word order matters; 2) requiring certain mandatory words in a concept to be present in the input text, in order to consider the concept a possible match, even if all words do not have to be present; and 3) calculating a score for each concept that is a possible match for the input text, and eliminating a concept as a possible match if there is another concept, that significantly overlaps that concept, that has a higher score.
• the systems including computer systems for processing medical reports
• methods described herein e.g., code instructions executed by one or more processors
• the systems and/or methods described herein improve the process of identifying medical concepts in a medical report, by one or more of 1) not requiring all words of a concept to be present in an input text in order to consider the concept a possible match; 2) requiring certain mandatory words to be present; and 3) calculating a score for each possible match and eliminating possible matches when they significantly overlap with a higher scoring possible concept.
• the systems including computer systems for processing medical reports) and/or methods described herein (e.g., code instructions executed by one or more processors) improve performance of a computing unit executing the code instructions that identify medical concepts in medical reports.
• the improvement in performance increases the number of medical concepts, actually present in each sentence of a medical report, that are successfully identified, while decreasing the number of false positives, medical concepts that are incorrectly identified as being found in sentences of a medical report.
• the improvement in performance is achieved at least in part by not requiring all words of a concept to be present in an input text, which increases the number of concepts successfully found, by requiring certain mandatory words of the concept to be present, which decreases false positives due to bad guessing of missing words, and by finding a score for each possible concept and eliminating possible concepts that significantly overlap with other higher scoring possible concepts, which decreases false positives due to identifying broader concepts when only narrower concepts are present.
• the systems including computer systems for processing medical reports) and/or methods described herein (e.g., code instructions executed by one or more processors) are tied to physical real-life components, because they process medical reports stored in data storage media, and because the medical reports are written by doctors to describe medical tests performed on real patients, using medical diagnostic equipment, for example medical imaging equipment such as CT scanning devices and MRI devices.
• medical diagnostic equipment for example medical imaging equipment such as CT scanning devices and MRI devices.
• the system and methods improve patient care, by making it possible to retrieve medical information from the medical reports much sooner than if the medical concepts had to be identified manually by a human reader of the medical reports, and more accurately than prior art computer systems that identify medical concepts in medical reports automatically.
• the systems including computer systems for processing medical reports) and/or methods described herein (e.g., code instructions executed by one or more processors) provide a unique, particular, and advanced technique of identifying medical concepts in medical reports.
• systems and/or methods described herein are inextricably tied to computer technology and physical hardware, to overcome an actual technical problem arising in natural language processing of medical reports.
• the present invention may be a system, a method, and/or a computer program product.
• the computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
• the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device.
• the computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
• a non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, and any suitable combination of the foregoing.
• RAM random access memory
• ROM read-only memory
• EPROM or Flash memory erasable programmable read-only memory
• SRAM static random access memory
• CD-ROM compact disc read-only memory
• DVD digital versatile disk
• memory stick a floppy disk, and any suitable combination of the foregoing.
• a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted through a wire.
• Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network.
• the network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers.
• a network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
• Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
• ISA instruction-set-architecture
• machine instructions machine dependent instructions
• microcode firmware instructions
• state-setting data or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.
• the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.
• the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).
• LAN local area network
• WAN wide area network
• Internet Service Provider for example, AT&T, MCI, Sprint, EarthLink, MSN, GTE, etc.
• electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
• FPGA field-programmable gate arrays
• PLA programmable logic arrays
• These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
• the computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
• each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s).
• each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration can be implemented by special purpose hardware -based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
• An object of some embodiments of the present disclosure is to automatically identify the medical concepts mentioned in the reports that doctors write up after examining patients, selected from a comprehensive database of medical concepts, using a computer system performing natural language processing.
• the computer has better performance than computers using existing methods for that purpose, both in identifying a higher percentage of the concepts that are mentioned in the report, and in not mistakenly identifying as many medical concepts that are not found in the report. Simultaneously achieving improved performance by both of these measures is challenging, since doctors often leave out words of a concept that they believe that another doctor reading the report will understand are implied. Finding those concepts requires guessing the words that are left out, but guessing words can also result in mistakenly identifying more concepts that are not really found in the report.
• a useful measure of success in achieving these objects is the parameter FI, which is the harmonic mean of the percent of concepts present that are identified, and the percent of identified concepts that are really present.
• Another object of some embodiments of the present disclosure is to achieve such an improved performance, for example as measured by FI, while processing medical reports quickly enough so that the method can serve as a first stage of a larger natural language processing system for radiological reports.
• the computer can process at least 100 reports per second, using an Intel ® CoreTM ⁇ 7-6700 CPU at 3.40 GHz with 4 cores and 8 logical processors.
• Another object of some embodiments of the present disclosure is to achieve such an improved performance without the algorithm having a deep AI-like understanding of each concept. Providing such a deep AI-like understanding of each concept could make development of the algorithm very labor intensive.
• An aspect of some embodiments of the invention concerns a computer system that determines which of a set of medical concepts is found in a medical report, for example a radiology report, automatically by executing instructions on the computer, with one or more of three features that the inventor has found are often effective at reducing false positives, and at not missing too many concepts that are really present. These features can be explained in terms of two approaches to the problem of automatically identifying concepts in a report, using a computer.
• the bag-of-words method as exemplified by the match test in ElasticSearch, considers a concept to match an input text, for example one sentence of the medical report, if all the words of the concept, or almost all the words, are found in the input text, regardless of word order.
• This method tends to be fairly efficient at finding the concepts that are present, but tends to produce a lot of false positives, finding concepts that are not really present.
• Another method used by the cTAKES algorithm and exemplified by the match_phrase test in ElasticSearch, considers a concept to match an input text if all the words of the concept are found in the input text in the same order, or almost the same order, as in the concept. This method tends to be somewhat better than the bag- of-words method at eliminating false positives, but somewhat worse at finding the concepts that are present.
• a method employing any one or two of the following features, or better yet all three of the following features can give better results both in eliminating false positives and in finding the concepts that are really present: 1) Not requiring all words to be present even in the match _phrase test, where word order matters. This allows additional concepts to be found, when the doctor who wrote the report omitted words from the concept that they thought would be obvious to human readers of the report.
• the first feature is optionally implemented as follows.
• Each concept is a string of words.
• the query need not be a proper sub-string of the words of the concept, but in the case of a 2-word concept, the query could be identical to the concept.
• the concept is optionally considered a possible match to the input text if at least a minimum number of the queries have both their words in the input text, sufficiently close together.
• the minimum number depends on the total number of queries for that concept, for example the minimum number is a percentage of the total number of queries, for example 35% of the queries, or 40%, 45%, 50%, 55%, 60%, or 65%.
• An overall match phrasc score for the concept based on the query sub scores, for example a weighted average of the query sub-scores, is then found, and a concept is not considered to pertain to the input text if it has sufficient overlap with another concept that has a higher score.
• the weighted average gives lower weight to queries for which the two words are further apart in the concept.
• the second feature requiring certain mandatory words to be present in the input text, is optionally implemented as follows.
• the word in the concept that is the least common of the words in the concept, among the words of all of the concepts in the set of concepts, is a mandatory word that must be found in the input text.
• any singular word that is the name of a disease is a mandatory word that must be found in the input text.
• any word referring to a direction or a location of a body part must be adjacent, in the input text, to the word for the body part that it refers to, in order to consider the concept as a possible match to the input text.
• a concept is considered a possible match if a sufficient number of words in the concept are found in the input text, where the minimum number of words may depend on the number of words in the concept. If a concept is considered a possible match to the input text according to this criterion, then optionally a match score is found for the concept, optionally depending only on the number of words of the concept that are present in the input text, regardless of the order of the words in the input text.
• the match score is combined with the match_phrase score described above, for example a weighted sum of the match score and the match_phrase score is found, to find a total score for the concept, and it is the total score that is used to decide which of two concepts to keep, when they have sufficient overlap.
• a concept is considered a possible match to the input text only if it is considered a possible match both according to the criterion of having enough of the queries match the input text (the“match_phrase test”), and according to the criterion of having enough of the words of the concept present in the input text (the “match test”).
• Requiring a concept to pass both the match test and the match_phrase test may make it more likely that a concept pertains to the input text even if some of the words of the concept are not found in the input text.
• only the match test is done, or only the match phrasc test is done, or both tests are done but the concept only has to pass one of the tests to be considered a possible match to the input text.
• the match test is applied also to one word concepts, for which a match _phrase test is not applied, and a match phrasc score is not calculated, because there are no two-word queries in a one word concept.
• a one word concept passes the match test if the one word is found in the input text, and optionally its match score, and its total score, is always the same.
• the match _phrase score is zero.
• one word concepts have their scores compared to other multi-word concepts that overlap them, for example to other concepts that also have that word, and that pass the match test and the match_phrase test.
• the score of the multi-word concept will always be higher in this case, and the one word concept will be eliminated from consideration.
• the set of concepts is preprocessed by stemming the words, i.e. replacing words by their stems.
• stop words are removed from the concepts, in the pre-processing.
• words in the concepts are replaced by a preferred synonym, when several medical terms have the same meaning.
• an adjective such as“hepatic” is replaced by a corresponding noun, in this case “liver.”
• acronyms in the concepts are expanded.
• vertebrae referred to by the letter-number designation have the words“vertebra” and“spine” added to them, and the letter is replaced by the region of the spine that it refers to, either“cervical,”“thoracic” or“lumbar”.
• any or all of the preprocessing procedures described herein for the concepts are applied to the input text.
• the methods described and claimed herein are not directed to abstract ideas, but to methods of improving computer technology for natural language processing, and the claims do not pre-empt all methods of identifying medical concepts in a medical report, but are limited to particular methods of achieving that result.
• at least some of the claims may be limited to a method where two-word queries are created from the concepts, and performing a match_phrase test for each concept by seeing how many of the two-word queries are a match to the input text.
• At least some of the claims may be limited to methods where certain mandatory words of the concept must be present in the input text, in order for the concept to be considered a possible match for the input text.
• FIG. 1A illustrates a block diagram 100 for a system that implements the method according to an exemplary embodiment of the invention.
• Each block in diagram 100 represents a software module, as stored in a computer that is programmed to implement the module, or a server that stores or provides access to data, for example in digital form.
• a report database 102 stores or provides access to medical reports that the system will process.
• a splitter module 104 divides each report into multiple input texts 106, each of which is processed separately to find the concepts it contains. Typically each sentence of a report is a separate input text.
• long sentences are divided into more than one input text, for example by a parser that uses grammatical rules to attempt to determine the boundaries of each component of the compound sentence.
• long sentences are divided by the splitter provided by the Natural Language Tool Kit in Python, but alternatively any other sentence splitter known in the art can be used.
• sentences are split at isolated commas, for example commas that have no other comma within 15 characters of them, but not at commas that do have another comma within 15 characters, because those commas are likely to be part of a list, such as in“lung, liver or throat cancer,” that bridge a single concept.
• a concept database 108 has a list of medical concepts, for example the medical concepts listed in the SNOMED database, or a selection of concepts listed in the SNOMED database.
• concept database 108 does not include all of the concepts listed in SNOMED or a similar database, but is limited to concepts describing disorders and morphological abnormalities, and is limited to concepts that are not longer than a maximum number of words.
• the inventors have generally used a database of concepts limited to concepts which have 6 words or less, after preprocessing, since the method tends to have a lower success rate for concepts with more words, and such concepts are found less frequently in medical reports. In the current implementation of the concept database, only concepts that describe disorders and morphological abnormalities are included.
• a processor module 110 uses an algorithm to identify the medical concepts found in each input text, as will be described in FIG. 2, though in general this process produces some errors.
• the results of this process a list of medical concepts found in each input text, are optionally sent to output module 112, which outputs them in a form that a user can read.
• output module does not separately list, in the output, the medical concepts found in each input text, but only collectively lists the medical concepts found in a set of input texts, for example in the entire medical report.
• processor module 110 in FIG. 1 is described in more detail by flowchart 200 of FIG. 2.
• Preprocessing of the concepts and the input texts need only be done once before processing many input texts, and optionally is only done again if the set of concepts is being revised.
• Preprocessing of the input text is optionally done for each new input text that is being processed, but could be done for a set of input texts, for example for all the input texts of a medical report, before processing any of them.
• Preprocessing of both concepts and input text includes, for example, stemming, removing stop words, expanding acronyms, and replacing words with synonyms.
• Preprocessing of concepts may also include, for example, generating queries for concepts of two or more words, finding mandatory words for each concept, and eliminating certain concepts, for example concepts that have more than a maximum number of words.
• Loop 120 includes, for each concept: 1) checking the input text for mandatory words at 122; 2) performing a match test and finding a match score at 124; and 3) performing a match_phrase test and finding a match_phrase score at 126. These three tests may be performed in series or in parallel, and if they are performed in series, they may be performed in any order. For the detailed flowchart shown in FIG.
• these tests are shown as performed in series, first the check for mandatory words, then the match test, and then the match_phrase test. Performing the tests in this order has the potential advantage that the check for mandatory words is generally faster than the match test which is generally faster than the match_phrase test, and the slower tests can be skipped, saving computer time, if the concept does not pass one of the faster tests.
• Performing the match_phrase test and finding the match_phrase score includes a loop 128 over the queries for that concept. At the end of loop 120, a total score for the concept is found at 130, for example by combining the match score and the match_phrase score.
• a concept has to pass all three tests, the check for mandatory words at 122, the match test at 124, and the match_phrase test at 126.
• the tests can be done in any order, or in parallel, but if the tests are done serially, then if a concept fails to pass one of the tests, the loop optionally ends for that concept and goes on to consider the next concept.
• the check for mandatory words is shown as done first, followed by the match_test, and the match phrasc test is shown as done last. A record is kept of which concepts pass all the tests and are thus possible matches for that input text, and of the total score for each concept that is a possible match.
• post-processing is done at 132.
• Post-processing includes a loop 134 over the possible matched concepts.
• loop 134 possible matched concepts are eliminated from consideration if they have significant overlap with another possible matched concept that has a higher score, for example because more words of the other concept are found in the input text, and/or the order of words in the input text is closer to the order of words in the other concept.
• next input text is considered, if there are any remaining input texts that have not been processed.
• Preprocessing optionally includes one or more of the following:
• the software optionally includes a list of acronyms used in medical reports, and their expanded forms. Such a list can be created manually, for example. From time to time the list of acronyms is optionally modified, for example by adding new acronyms that have started to appear in medical reports, or by removing an acronym that has been found to be ambiguous, and/or is no longer is common use.
• the concepts are preprocessed, optionally using the same procedures as described for the input texts.
• concepts that have more than a maximum number of words after preprocessing are removed from the list of concepts.
• concepts with more than six words are excluded.
• concepts with more than five words, or more than seven words, or more than eight words are excluded. Tests done by the inventor have shown that it is more difficult to determine if a concept is present in the input text, the more words the concept has, and furthermore concepts with a very large number of words are likely to be narrow in scope and not to occur very often, so in practice it is not so important to consider them.
• queries are generated for each concept.
• each query consists of a pair of words found in the (preprocessed) concept, in the same order as the words are found in the concept, and there is a query generated for each such pair of words.
• The“Percolator” feature of ElasticSearch provides a convenient way to generate the queries. It should be understood that the order of words in the query is important, not because the words of the input text are required to be in the same order, but because a concept for which the words in the input text are out of order will generally have a lower score than if they are in the same order as in the concept.
• queries with more than two words are also generated, for example queries with up to three words. However, the inventor has found that there is little to be gained by using queries of more than two words.
• each concept is examined to find the mandatory words it contains.
• the mandatory words for a concept are words that must appear in the input text, in order to consider that concept a match for the input text.
• the mandatory words optionally include any of the following:
• the word indicating direction must be found in the input text within one word of the body part that it describes, in order for the concept to be considered a match to the input text.
• procedures 204, 206 and 208 may be done once, before processing any medical reports, and they become part of the version of the algorithm that is being used. They need not be done again for each new report that is processed.
• procedures 204, 206 and 208 may be done over again, at least for the concepts that are new or have been changed.
• the first input text is considered.
• the first concept is considered, to decide whether it is a match for that input text.
• match test requires that a certain minimum number of words, out of all the words in the concept, be present in the input text, but their order does not matter, because the match test is a“bag of words” test. It should be understood that the number of words present in the input text, and the total number of words in the concept, optionally both refer to after the concept and input text have been preprocessed, for example removing stop words, and stemming. The inventor has found the best results when the minimum number of words in the concept that must be present in the input text to pass the match test is greater than 2/3 of all the words in the concept, but not greater than 3/4 of all the words in the concept.
• the minimum at any value greater than 1/2 and not greater than 5/6 of all the words in the concept For example, for two word concepts, optionally both words must be found in the input text. For three word concepts, optionally two words must be found, or alternatively all three words must be found in the input text. For four word concepts, the minimum number of words that must be found in the input text is optionally two, or three, or four. For five word concepts, the minimum number of words is optionally three, or four. For six word concepts, the minimum number of words is optionally three, four, or five. Optionally, the minimum number of words is 50%, or 60%, or 70%, or 80%, of the total number of words in the concept.
• the match score is found and recorded.
• the match score depends on how many words in the concept are found in the input text. If a greater number of words in the concept are found in the input text, then it is generally more likely that the concept is found in the input text, even if the percent of the words of the concept that are found in the input text is not greater.
• the match score depends only on the number of words of the concept found in the input text, without regard to how many words are in the concept. Alternatively, the total number of words in the concept is also taken into account, in calculating the match score. In an exemplary implementation of the method, the match score is the square root of the number of words of the concept that are found in the input text.
• the first query for that concept is considered, at the beginning of a loop that will look at all the queries for that concept, in order to determine whether the concept passes the match_phrase test, and in order to calculate the match_phrase score. It should be understood that, in the case of a one word concept, there are no queries, since each query contains two words from the concept, and so no match_phrase test can be done, and no match_phrase score can be calculated. In that case , although this is not explicitly shown in flowchart 200, control passes to 238, where the next concept is sought.
• the parameter specifying the maximum s that is allowed, in order to consider the query to be found in the input text is referred to as“slop.”
• the query sub-score is calculated and recorded.
• the query sub-score is optionally based on how different the positions of the words of the query are in the input score, from their positions in the query, with a lower query sub-score if the positions of the words are more different.
• the difference in positions s is defined as described above.
• the query sub-score is proportional to l/(s + 1), with s defined as described above.
• the query sub-score is V2/(s + 1).
• a concept passes the match_phrase test if at least a minimum number of the queries in that concept are found in the input text. For example, at least a minimum percentage of the queries for that concept are found in the input text, for example at least 35%, or at least 40%, or at least 45%, or at least 50%, or at least 55%, or at least 60%, or at least 65%, of the concepts are found in the input text.
• the match_phrase score is calculated for that concept.
• the match phrasc score is a weighted sum of all the query sub-scores for the queries of that concept that are found in the input text, divided by the number N of queries of that concept.
• a lower weight is used for queries whose words are further apart in the concept. For example, if the query consists of the z th and the f h words in the concept, then the weight used for that query is 1.2 - 0.15(/ - /).
• the constant 1.2 in this expression is replaced by another constant, for example between 0.9 and 1.5, and/or the constant 0.15 is replaced by another constant between 0.10 and 0.20.
• i -j is always between 1 and 5
• the weight 1.2 - 0.1 5(/ - /) can range from 0.45 to 1.05.
• the expression for weight is chosen so that, given the maximum number of words in the concept, the weight is never negative.
• the concept passes the match_phrase test, then at 236 the concept is recorded as being a possible match for the input text, and the total score for the concept is found and recorded.
• the total score for the concept depends on both the match score and the match_phrase score, for example the total score is a weighted sum of the match score and the match_phrase score.
• the match score is given a weight of 1.5 times the weight of the match_phrase score, with the match score defined as the square root of the number of words in the concept that are found in the input text, and the match_phrase score defined as
• MP(i,j) is the query sub-score for the query consisting of the z th word and the f h word in the concept, defined as 2 /(s + 1), with the sum in the expression for the match_phrase score being over all the queries in that concept, and N being the number of queries in the concept.
• the definition of total score will be unchanged if the match score is redefined to differ by a factor .v, the match_phrase score is defined to differ by a factor y, and the relative weights of the match score and the match_phrase score are redefined to differ by a factor of y/x.
• One -word concepts If the concept has only one word, then there are no queries associated with it, since queries each have two words from the concept. In this case, it is not possible to perform a match_phrase test or to calculate a match_phrase score as described above.
• a concept is considered a match as long as the match test is passed, which means that the one word of the concept is found in the input text.
• the total score depends only on the match score, which is always the same for a one word concept, if the match score depends only on the number of words of the concept that are found in the input text.
• match score is defined as the square root of the number of words of the concept found in the input text, then the match score would always be 1 for a one word concept. If the score is defined in general as the weighted sum of the match score and the match_phrase score, with a weight of 1.5 for the match score, and if the same definition is used for a one word concept with the match_phrase score set to zero, then for a one word concept the total score will always be 1.5, if the concept is a match for that input text. The inventor has found, in some cases, that treating one word concepts in this way results in good performance of the algorithm, finding more of the concepts that are really present, and reducing false positives. Alternatively, a different constant value is used for the total score for all one word concepts for which the one word is found in the input text, for example 1, or 2, or an intermediate value, or a value less than 1 or greater than 2.
• the second highest scoring possibly matching concept is considered.
• the score of the concept being considered is compared to the scores of each of the concepts that have a higher score than it has. The first time this is done, with the second highest scoring concept, there is only one concept with a higher score, that it has to be compared to. If the concept being considered has significant overlap with a concept that has a higher score, then the concept being considered is removed from the list of possibly matched concepts for this input text.
• “significant overlap” is defined as any of these three conditions being true:
• the number of words found in one of the two concepts but not in the other concept is sufficiently small relative to the number of words in the concept being considered.
• the number of words found in one of the concepts but not the other, divided by 1.7 is less than the greatest integer that is less than or equal to 0.5 times the number of words in the concept being considered.
• the set of concepts includes some groups of two or more concepts that are considered to have the same meaning, and one of the concepts in each group is designated as the more preferred form, and the other concepts in each group are designated as less preferred forms. The concepts are treated as separate concepts throughout the process of the algorithm, until the concepts in the input text have been identified. Then, when the identified concepts are listed in the output, any less preferred forms are replaced by the corresponding more preferred form.
• FIG. 3 shows flowcharts, similar to FIG. IB, for the cases respectively where only the first feature is used, where only the second feature is used, and where only the third feature is used. In all three cases, the flowchart begins with preprocessing 116, taking the first input text 118, and performing a loop over concepts 120, as described above for FIG. IB.
• the flowchart ends with outputting the results 136 for that input text, and for then looking at the next input text 138, as described above for FIG. IB.
• the flowcharts differ in what is included in the loop over concepts, and in whether or not they include post processing.
• flowchart 300 shows a method of identifying medical concepts in a medical report, where, in each input text, all the words of a concept do not have to be found in the input text, in a match_phrase test, in order for the concept to be considered a possible match to the input text.
• the loop over concepts 120 shows only a match_phrase test 128, though optionally a match test is also done, and optionally mandatory words are checked for. No calculation of scores is shown, and no post processing is shown, in flowchart 300.
• the match phrasc test optionally includes a loop over queries, with each query consisting of a pair of words from the concept, and all queries need not be found in the input text.
• the match_phrase only requires a certain number of the queries, for example 50% of the queries, to be found in the input text, in order for the concept to pass the match_phrase test.
• the match_phrase test does not include a loop over queries, but, for example, the match_phrase test requires that a certain number of words of the concept, less than 100% of the words, are found in the input text, in order to pass the match _phrase test.
• the computer system may find more of the concepts that are present in the input text, than in prior art where all words in the concept have to be present in the input text, in order to pass a match_phrase test.
• a flowchart 400 shows a method of identifying medical concepts in a medical report, where certain mandatory words, for each concept, have to be found in the input text, in order for the concept to be considered a possible match to the input text, even though all words in the concept do not have to be found in the input text.
• the loop 120 over concepts includes a check 122 that the mandatory words are present.
• Loop 120 also includes a test 402, for example a match test and/or a match phrasc test, which does not always require that all words of the concept be present in the input text, in order to pass the test, but does require that a certain number of words of the concept be present, in order to pass the test.
• no calculation of scores is shown, and no post-processing step is shown.
• a flowchart 500 shows a method of identifying medical concepts in a medical report, in which each possibly matching concept is given a score, indicating a degree of matching to the input text, and lower scoring concepts are excluded as possibly matching, when there is a higher scoring concept that has a significant overlap with the lower scoring concept.
• Loop 120 over concepts shows both match test 124, and a match_phrase test 126, with a score calculated for each one, if the concept passes the test and is found to be possibly matching concept.
• the match_phrase test includes a loop 128 over queries. The match score and the match_phrase score are combined, at 130 to obtain a total score, for example by taking a sum or a weighted sum of the match score and the match_phrase score.
• Post-processing is done at 132, by performing a loop 134 over possibly matched concepts. For each possibly matched concept, the concept is excluded as a possibly matched concept if there is a higher scoring possibly matched concept that significantly overlaps it. This post-processing prevents broader concepts from being identified in the input text, if only a narrower concept is found. For example in an input text that only mentions the left lung, the concept“left lung” will be identified, but the broader concept“lung” will not be.
• a computer program product may include one or more storage medium, for example; magnetic storage media such as magnetic disk (such as a floppy disk) or magnetic tape; optical storage media such as optical disk, optical tape, or machine readable bar code; solid-state electronic storage devices such as random access memory (RAM), or read-only memory (ROM); or any other physical device or media employed to store a computer program having instructions for controlling one or more computers to practice the method according to the present invention.
• magnetic storage media such as magnetic disk (such as a floppy disk) or magnetic tape
• optical storage media such as optical disk, optical tape, or machine readable bar code
• solid-state electronic storage devices such as random access memory (RAM), or read-only memory (ROM); or any other physical device or media employed to store a computer program having instructions for controlling one or more computers to practice the method according to the present invention.
• compositions, method or structure may include additional ingredients, steps and/or parts, but only if the additional ingredients, steps and/or parts do not materially alter the basic and novel characteristics of the claimed composition, method or structure.
• a compound or “at least one compound” may include a plurality of compounds, including mixtures thereof.
• range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.
• a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range.
• the phrases “ranging/ranges between” a first indicate number and a second indicate number and “ranging/ranges from” a first indicate number“to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.
• test data was prepared from 107 radiology reports. Each sentence of each report was manually examined to determine which concepts in a set of concepts, taken from the SNOMED CT medical dictionary, limited to concepts with no more than six words (after preprocessing as described above, to remove stop words, etc.), and limited to concepts that describe disorders and medical abnormalities. In this way, it was determined which concepts were really found in each sentence. A total of 860 concepts were found in the sentences of the 107 reports.
• the algorithm was used to identify which concepts in the set occurred in each sentence of the reports. A count was made of the total number of true concepts identified (true positives), the total number of concepts identified that were not really found in that sentence (false positives), the total number of concepts that were really present in each sentence that the algorithm failed to identify (false negatives), and the total number of concepts not present in each sentence, that the algorithm correctly did not identity (true negatives). Two quantities were then calculated. Precision is the ratio of true positives to total positivies. This is the fraction of all concepts identified by the algorithm that were really present. Recall is the ratio of true positives to true positives plus false negatives. This is the fraction of all concepts really present that the algorithm identified. The quantity FI is the harmonic mean of the precision and the recall, which may be defined as 2/(1 /precision + 1 /recall).
• Hybrid algorithm including both a match score and a match_phrase score (with some“slop”) in the score, with the match phrasc test requiring all words in the concept to be present in the input text.
• Hybrid algorithm including both a match score and a match_phrase score, where the match_phrase score and match_phrase test are based on a set of all the two- word queries in the concept, rather than requiring a match with all the words in the concept, but without using mandatory words.

Landscapes

• Engineering & Computer Science (AREA)
• Health & Medical Sciences (AREA)
• Medical Informatics (AREA)
• Public Health (AREA)
• Epidemiology (AREA)
• Data Mining & Analysis (AREA)
• Primary Health Care (AREA)
• General Health & Medical Sciences (AREA)
• Biomedical Technology (AREA)
• Pathology (AREA)
• Databases & Information Systems (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Machine Translation (AREA)
• Medical Treatment And Welfare Office Work (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=68766710

Family Applications (1)

Country Status (5)

Cited By (1)

Families Citing this family (2)

Citations (1)

Family Cites Families (15)

• 2019
  • 2019-11-22 US US17/296,688 patent/US20220028509A1/en not_active Abandoned
  • 2019-11-22 EP EP19813435.5A patent/EP3888096A1/en active Pending
  • 2019-11-22 WO PCT/EP2019/082288 patent/WO2020109177A1/en not_active Ceased
  • 2019-11-22 JP JP2021529772A patent/JP7550756B2/ja active Active
  • 2019-11-22 CN CN201980090257.1A patent/CN113348515B/zh active Active

Patent Citations (1)

Non-Patent Citations (4)

Also Published As

Similar Documents

Legal Events