KR101555114B1 - A decision-support application and system for problem solving using a question-answering system - Google Patents

Abstract

The decision-support system for problem solving includes software modules implemented on a computer-readable medium, the software modules including an input / output module and a query-response module. The method includes receiving problem case information using the input / output module, generating a query based on the problem case information, and generating a plurality of responses to the query using the query-response module. The method also includes calculating the values for the plurality of evidence dimensions from the evidence sources for each of the responses using the query-response module and using the query-response module to calculate And calculates a corresponding confidence value for each of the responses. The method also uses the input / output module to output values of each evidence dimension for the responses, the corresponding confidence values, and the one or more selected responses.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention < RTI ID = 0.0 > [0001] < / RTI > A decision-

The present invention relates to assisting a human expert in solving a problem in a specific area by using the question-answering system of the present application, and more particularly, And a decision-support application and system.

Decision-support systems exist in many different industries that require support from human experts to retrieve and analyze information. An example to be used throughout this application is a diagnostic system used in the medical industry.

Diagnostic systems can include systems that use structured knowledge, systems that use unstructured knowledge, and clinical decision formulas, rules, trees, , Or algorithms (algorithms). Early diagnostic systems used a structured knowledge base or a classical, manually constructed knowledge base. The Internist-I system, developed in the 1970s, has been used to assess disease-finding and disease-disease relationships associated with numbers such as sensitivity, the fraction of patients with disease finding (Myers, JD The background of INTERNIST-I and QMR. In Proceedings of ACM Conference on History of Medical Informatics (1987), 195-197).

The MYCIN system for the diagnosis of infectious diseases, developed in the 1970s, also includes production rules that specify that "if certain facts are true, they can conclude certain other facts with a given certainty factor" (BG and Shortliffe, Eds.) Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project (Addison-Wesley, Reading, MA, 1984). DXplain, developed in the early 1980s, uses a structured knowledge similar to that of Internist-I, but adds a hierarchical lexicon of findings (Barnett, GO, Cimino, JJ, Hupp, JA, Hoffer , EP DXplain: An Evolving Diagnostic Decision-Support System, JAMA 258, 1 (1987), 67-74).

Iliad, developed in the early 1990s, adds more sophisticated probabilistic reasoning. Each disease is associated with an associated ancestry of the disease, along with the proportion of patients with that disease (sensitivity) and the proportion of patients with that disease (1-specificity) a priori probability of the disease) and a list of findings (Warner, HR, Haug, P., Bouhaddou, O., Lincoln, M., Warner, H., Sorenson, D., Williamson, JW and Fan, C. ILIAD as an expert consultant to teach differential diagnosis.In Proc. Annu. Symp.Comp Appl. Med. Care. (1988), 371-376). DiagnosisPro (http://en.diagnosispro.com) is a structured knowledge base for online queries and searches.

In 2000, diagnostic systems using unstructured knowledge began to emerge. These systems also use some structuring of knowledge. For example, entities such as findings and disorders can be tagged in documents to facilitate searching. ISABEL uses Autonomy information retrieval software and a database of medical books to search for appropriate diagnoses of input findings (Ramnarayan, P., Tomlinson, A., Rao, A (2003), 408-413. In this paper, we present the results of an initial performance evaluation of a differential diagnosis aid for pediatrics. ).

Autonomy Auminence uses Autonomy's technology to search for diagnostics on input findings and organize the diagnosis by body system (http://www.autonomyhealth.com). First CONSULT allows you to search massive volumes of medical books, journals, and manuals by chief complaints and age groups to arrive at possible diagnoses (http: // www .firstconsult.com). PEPID DDX is PEPID's independent clinical content-based diagnostic generator (http://www.pepid.com/products/ddx/).

Clinical decision rules have been developed for many disorders and computer systems have been developed to help practitioners and patients to apply the rules. ACI-TIPI (Acute Cardiac Ischemia Time-Insensitive Predictive Instrument) takes clinical and ECG features as inputs and generates the output of acute cardiac ischemia as an output (Selker, HP, Beshansky, JR, Griffith, JL, Aufderheide, TP, Ballin, DS, Bernard, SA, Crespo, SG, Feldman, JA, Fish, SS, Gibler, WB, Kiez, DA, McNutt, RA, Moulton, AW, Ornato, JP, Use of the acute cardiac ischemia time-insensitive predictive instrument (ACI-TIPI) to assist with triage of patients with chest pain or other symptoms suggestive of acute Cardiac ischemia: A multicenter, controlled clinical trial. Annals of Internal Medicine 129, 11 (1998), 845-855). For example, ACI-TIPI has been introduced into the market for cardiac monitors / defibrillators.

The CaseWalker system uses a four-item questionnaire to diagnose major depressive disorder (Cannon, DS and Allen, SN A comparison of the effects of computer and manual reminders on a mental health clinical practice guideline. Journal of the American Medical Informatics Association 7, 2 (2000), 196-203). The PKC Advisor provides guidance on 98 patient problems, including abdominal pain and vomiting (http://www.pkc.com/software/advisor/).

The advantage of current diagnostic systems is that these systems improve the diagnostic hypothesis of clinicians (Friedman, CP, Elstein, AS, Wolf, FM, Murphy, GC, Franz, TM, Heckerling, JAMA 282, 19 (1999), 1851-1856), and clinical doctors who missed important diagnoses. (Ramnarayan, P., Roberts, GC, Coren, M., Nanduri, V., Tomlinson, A., Taylor, PM, Wyatt, JC and Britto, JF Diagnosis of the potential impact of a reminder system on the reduction of diagnostic errors: A quasi-experimental study BMC Med. Inform. Decis., 6, 22 (2006)).

Current diagnostic systems are not widely used (Berner, ES, Diagnostic Decision Support Systems: AMIA Annu. Symp. Proc. 2006 (2006), 1167-1168, This is due to limitations that prevent the use of these systems in the day-to-day operations of medical institutions (Coiera, E., Guide to Health Informatics (Second Edition), Hodder Arnold, 2003; Shortliffe, T. Medical thinking: What should we do? Refer to "What Do We Know? A Review Meeting (2006), http://www.openclinical.org/ medicalThinking2006Summary2.html, then Shortliffe, 2006 being).

Many other medical practitioners consult patients, and patient data can be distributed across many other computer systems in both structured and unstructured formats. In addition, these systems are difficult to interact with each other (Berner, 2006; Shortliffe, 2006). The entry of patient data is also difficult, the list of diagnostic suggestions is too long, and the reasoning behind diagnostic suggestions are not always clear. Moreover, the systems are not fully prepared for the next steps and do not help recall what the clinician should do to help the patient (Shortliffe, 2006). The systems also can not request missing information that can increase the reliability of the diagnosis to the physician, and the systems are not always based on up-to-date advanced medical evidence and are also difficult to keep up to date (Sim, I., Gorman, P., Greenes, RA, Haynes, RB, Kaplan, B., Lehmann, H. and Tang, J. Clinical decision support systems Inform. Assoc. 8, 6 (2001), 527-534).

In view of these problems, the disclosed embodiments of the present application provide an improved medical diagnostic system.

One exemplary method embodiment of the present application provides a decision-support system for problem solving. The system includes software modules implemented on a computer readable medium, the software modules including an input / output module and a query-response module. The method includes receiving problem case information using the input / output module, generating a query based on the problem case information, and using the query-response module, And generates a plurality of answers. The method also includes using the query-response module to calculate numerical values for multiple evidence dimensions from evidence sources for each of the responses, Module to calculate a corresponding confidence value for each of the responses based on the value of each of the evidence dimensions. The method also uses the input / output module to output the responses, the corresponding confidence values, and the values of each evidence dimension for the one or more selected responses.

One exemplary system embodiment of the present application includes a first repository for maintaining problem case information, a computer processor operatively connected to the first repository, and a processor operatively coupled to the computer processor, 2 repositories. The computer processor is configured to receive the problem case information from the first storage. The computer processor is also configured to generate a query based on the problem case information and to generate a plurality of responses to the query. Wherein the computer processor is further configured to calculate values for a plurality of evidence dimensions from evidence sources for each of the responses and to calculate corresponding confidence values for each of the responses based on the values of each evidence dimension do. The computer processor is also configured to output the queries, the responses, the corresponding confidence values, and the values of each evidence dimension to the second repository.

Another embodiment of the present application includes a computer program product comprising a computer readable storage medium storing computer readable program code comprising instructions executable by a computerized device . Wherein the computer program code comprises an input / output module for receiving problem case information, a problem case analysis module for analyzing the problem case information to identify semantic concepts, module, a question generation module for generating a query from the semantic concepts, and a question-answering module for generating a plurality of responses to the query. The query-response module computes values for a number of evidence dimensions from the evidence sources for each of the responses. The query-response module uses the query-response module to calculate corresponding confidence values for each of the responses based on the numerical value of the respective evidence dimension. The input / output module outputs the values for the queries, the responses, the corresponding confidence values, and the plurality of evidence dimensions.

1 is a schematic diagram illustrating a system architecture chart for one embodiment of the present application.
Figure 2 is a flow diagram illustrating the decision support process flow.
3 is a systematic diagram showing a semantic model for the medical domain.
Figure 4 shows the marginal contribution of evidence for the four disease responses according to the dimensions of current illness, family history, findings, and demographics. Fig.
Figure 5 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
Figure 6 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
7 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
Figure 8 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
9 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
10 is a systematic diagram showing one embodiment of the present application as applied to the medical field.
11 is a flow diagram illustrating one embodiment of the present application as applied to the medical field.
Figure 12 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
13 is a schematic diagram illustrating one embodiment of the present application as applied to the medical field.
14 is a schematic diagram showing one embodiment of the present application as applied to the medical field.
15 is a schematic diagram illustrating a computing node in accordance with one embodiment of the present application.
16 is a schematic diagram illustrating a cloud computing environment in accordance with one embodiment of the present application.
17 is a schematic diagram illustrating an abstract model hierarchy according to one embodiment of the present application.

The following is a description of a decision-support application for problem solving in a specific area. The area may be a specific area, such as, for example, a differential diagnosis of a medical area, as discussed below, or it may be a broader area. The purpose of a decision-support application is to inform the problem solving process based on contextual information related to a problem, as described in the target case. Such problem case entry information may be structured, unstructured, or otherwise formatted. Decision support is provided using a query-response system that accepts questions or queries and returns a list of answers and associated confidences.

When referring to a query-response system in a method, an input query expressed in many possible forms, including natural language, structured language, or many other means, And the like. Note that the response need not be limited to being a single "atomic" concept (such as a person, place, or thing), but can be a complex entity. Some examples of complex entities are sophisticated explanations that are reached through complex reasoning or a series of process steps necessary to achieve a user's intended goal. The embodiments of the present application can be applied to various areas including complex systems where human experts need to solve problems. For example, a detailed description (as one example of many areas) is provided for decision support applications for differential diagnosis and treatment in the medical domain. As one of ordinary skill in the art will appreciate, the system may be used for other complex systems as well.

One embodiment of the present application enables "mixed-initiative" conversation. For example, a user may receive queries and receive responses from the application. In addition, the application can automatically provide a "push" notification (eg, notification of some important changes) to the user or help to change the system reliability in the responses provided by the application Possible queries can be made to the user. In fact, the system can continue to monitor the relevant case input as well as take directed queries on a particular case.

Figure 1 is a schematic diagram illustrating a broader decision-making context in a system architecture chart of an embodiment of the present application. The decision-maker 108 can input information through a variety of devices including cell phones, tablets, computers, consumer electronics, and the like. This information can be entered in a variety of formats including spoken format, typed format, and a format constructed through a series of GUI interactions. The information may be problem case information or may be queried. The query may be in the form of a natural language, a structured language, or some other query format. The problem case information used by the system may be multimodal and may take the form of text, images, audio, or other media types.

In general, embodiments of the present application are configured to support an iterative refinement approach by causing interaction during a period of time. Thus, an aspect of the embodiments of the present application is a repository of all relevant analysis and decisions made to date. This repository 106 not only retains a representation of the reasoning and decision process as an efficiency mechanism but also allows the system to determine the assumptions and assumptions based on new evidence relevant to the case To be reevaluated. This allows users to interact with this expression, accepting, refusing, or modifying this expression as the user deems necessary in this interaction, which may or may not affect the validity or importance of the evidence or reasoning chain. And to explore alternative solutions based on the insights of the users. This repository 106 is not only useful in evolving decision making interactions but also can be used to track the provenance of decisions made in the past, It enables notification of actions to be taken based on newly arriving information. For example, if a new study reports a contraindication to a drug in a given situation, the system can use the prior analysis of this repository 106 to determine the conclusions of the analysis Reevaluate and provide relevant notification of alternative therapies to patients who have been on medication for many years.

Finally, all of the embodiments of the present application described herein generally provide that, upon notification of the decision making process, the decision maker 108 may view the reliability associated with the alternatives and proposed responses To retrieve the evidence used and the reasoning process used by the system to arrive at conclusions and consequently to obtain feedback as to whether any additional information, if provided, can result in altering the responses.

The term diagnostic used in the medical field can be generalized to mean "inform" in other areas. The medical examples illustrated in the present application illustrate diagnosis, including answers, confidences, dimensions of evidence, associated evidence passages, and evidence of this And through the reliability of the evidence source. In the medical domain, embodiments of the present application may be used as a clinical decision support tool by physicians to provide therapy to a patient. Examples of queries include: what clinical conditions characterize a set of symptoms ?; What is a "list of diseases" (a list of diseases that can potentially lead to a set of symptoms, conditions, and findings)? This can be coordinated by providing other patient-related information, such as active diseases, current medications, allergies, past medical history, family history, and patient demographics); What tests can increase or decrease confidence in a given disease hypothesis provided in differential diagnosis ?; And / or information about the patient, what treatments are recommended for the specified disease? . In the medical domain, problem case information can be electronic medical records.

The query-response system derives responses from the repository of 'domain knowledge'. Embodiments of the present application leave delineation of domain knowledge to the query-response system. In an exemplary medical implementation, a query-response system not only uses the natural language of medical textbooks, clinical guides, and other documents as domain knowledge, but also the structured sources of information provided in databases or ontologies And other structured sources of information provided in a potentially structured format.

In general, embodiments of the present application describe using the example of a medical diagnosis system a decision-support application 104 located between the source of the problem case information 102 and the query-response system 110. However, as those skilled in the art will appreciate, embodiments of the present application are not limited to medical diagnostic systems. Conversely, embodiments of the present application apply to diagnostic problem solving in other complex-system areas that require a query response with respect to unstructured material. Examples of these areas include aircraft (and other transports of similar complexity) maintenance and information technology support (call center). The system can be used for automobiles, submarines, or much less complex but more universally accessible applications, such as an application that finds responses to queries about "how to" Given the input specifications such as ingredients, cost requirements, time, complexity, elegance, etc., it may be used in applications looking for responses to suitable meal recipes. They all feature large quantities of structured and unstructured "problem case information" and domain knowledge, and require a deep question-answer type technology. Thus, while the examples of the present application use a medical diagnostic system, those of ordinary skill in the art will recognize that embodiments of the present application may be applied to all query-response systems and are not limited to merely exemplary medical diagnostic systems It will be appreciated that an exemplary medical diagnostic system is used as a platform to illustrate embodiments of the present application.

The system 104 includes an input / output module, a problem case analysis module, a question generation module, a hypotheses and evidence module, And the like (implemented on a computer readable medium). The QA system 110 includes a query-response module and the like. The objective of decision-making is to diagnose and solve problems arising in the complex system 100 specified in the domain. A decision is made by the human expert (user) 108 interacting with the decision-support application 104 (through items 100 and 102). A record of past decisions and associated information used to arrive at the decision is maintained in the repository 106. In some embodiments, the query-response module may match the query to at least one of the pre-generated medical diagnostic queries that are generated by the query generation module and stored in the repository.

The decision-support application 104 may be triggered in various ways. In one mode, the decision maker 108 queries for specific cases. The application 104 expands the query into relevant problem case information and submits it to a query-response (QA) The resulting responses or hypotheses are then provided to the decision maker 108 and the decision maker can continue to repeat this process to access an acceptable solution to the problem.

Another mode of operation assumes the existence of standing queries defined by the query generation module and / or the decision maker 108. When new case information comes in, these queries are automatically executed by the decision-support application 104 without the active intervention of the decision maker 108. The results can be either proactively sent to the decision maker 108 for subsequent scheduled interaction or stored in the store 106.

These modes of operation assume the existence of active problem cases that have not yet been satisfactorily resolved. Another mode of operation may be initiated by a change in the content of the domain knowledge used by the query-response system of item 112. In the medical field, new clinical literature and guidelines are being published and they describe new screening procedures, therapies, and treatment complications. The decision-support application 104 may use the repository 106 to store past analyzes and decisions to determine which of its previous instances will be sufficiently affected by this new knowledge, If such a decision is made, an alert is sent to the responsible decision maker (108).

Thus, Figure 1 also illustrates a computer system comprising a first storage for maintaining problem case information, a computer processor 104/110 operatively coupled to the first storage 102 (executing the modules), and a computer processor 104/110 As well as a second reservoir 106 operatively connected to the second reservoir 106. As shown in FIG. When items are directly or indirectly connected to one another (e.g., physically, functionally, wired, wirelessly, etc.), they are considered to be operatively connected to each other.

This "computer processor" 104/110 automatically analyzes the problem case information to identify semantic concepts, relations and data, and at least one of the semantic concepts, relationships and data And automatically generates a diagnosis query of the user. The computer processor 104/110 may also automatically generate a plurality of diagnostic responses for each diagnostic query and generate a response to the response based on the numerical values for several dimensions of evidence associated with the problem- Lt; RTI ID = 0.0 > confidence values. ≪ / RTI > The computer processor 104/110 then calculates corresponding reliability values for each of the diagnostic answers based on the value of each evidence dimension of the evidence sources of the confidence values It can be calculated automatically. In addition, the computer processor 104/110 may then automatically generate links for each item of evidence, which may include the passages justifying the answer in a given fact to the user, . For example, links can be generated for the source of the phrase, its type (textbook, handbook, journal article, web content, database or structured source). The computer processor 104/110 may also output to the decision maker 108 and / or the second repository 106 the values of the queries, the responses, the corresponding confidence values, the links to the evidence sources, and each evidence dimension do.

Figure 2 is a flow diagram illustrating the decision support process flow performed by the decision support application 104. 2 shows the flow of information from the problem case information 102 to the query-response system 110 and the flow of information returned from the query-response system 110 to the decision maker 108. In particular, Lt; / RTI > Thus, the method receives problem case information using an input / output module. Further, in the medical domain, the problem case information may include a disease symptom of a patient, family history of the patient, demographic characteristics of the patient, and the like. The output from query-response system 110 is used by decision maker 108 to make decisions and find additional information about the problem.

In item 202, the method receives input regarding the current problem. The method can receive a user query through an input / output module in a free format query format, a free format statement format, and / or a keyword search format. The input from the problem may be multi-mode, such as text, audio, image, video, and the like. The text may be unstructured, such as paragraphs of a problem description in a natural language, and structured as content from a database. For example, in the medical domain, the input may be clinical information related to a patient's "History of Present Illness" (HPI). This may be in the form of unstructured textual paragraphs describing all aspects of the patient's HPI recorded by the nurse or physician or may be in the form of shorter sentences or sculptural statements assigned to specific HPI categories and may be semi-structured in the form of snippets.

The input information may flow over time. The input may be initiated by a change in problem state, a result of an additional check or procedure being performed, or a response to a query for more information generated by the decision maker 108. In addition, the information in the domain knowledge content 102 may be used as the demographic changes evolve, as the medical discoveries evolve, as medication conflicts evolve, As side effect information evolves, it can change. This time-stamped information is recorded in the in-system storage 106.

In item 204, the method automatically analyzes problem case information 102 using a problem case analysis module to identify semantic concepts, relationships and other relevant knowledge (e.g., patient medical data) do. Therefore, when incoming information is not structured such as natural language text, audio or images and concepts, relations and other kinds of information related to the area are to be identified, semantic concepts, relationships and other pertinent knowledge. This is done by software components called "annotators". These annotators may use procedural code, rule based, programmed logic, or some other form to determine concepts, relationships and other pertinent information. Annotators may, for example, be based on machine learning using a set of training data that holds known concepts and relationships.

In the medical field, annotators are used to describe phrases related to clinical concepts such as patient symptoms, current medical conditions, clinical findings, medications, family history, demographic characteristics, Can be recognized. Annotators can also identify relationships between entities such as the location of symptoms, the severity of a condition, or the number of findings. The concepts and relationships are expressed in a domain-specific semantic model or type system. An example of this semantic model of the medical domain is shown in FIG. More specifically, in the example shown in FIG. 3, the various elements have different logical / causal relationships. For example, the substance 302 has a relationship of "is a" with the agent 304, which indicates that the substance 304 is "drug" 304. Similarly, the disease / syndrome 306 may be caused by the medication 304 and the disease / syndrome may be a complication of another disease / syndrome in item 306.

With regard to the disease / syndrome 306, it can be identified by a test 308, located at an anatomy location 310, and presented by a finding 318 And may be a complication of a certain treatment 312. The treatment 312 may be a procedure 314, a drug 314, ) 316, and the like. Similarly, in connection with the finding 318, it may be measured by the exam 308, located at the human position 310, treated by the treatment 312, , Or may specify a clinical attribute 320. In one embodiment, Clinical attribute 320 may also be affected by treatment 312. Thus, the semantic model illustrated in Figure 3 (which may be referred to as a factoid physiology definition guideline) illustrates various concepts and relationships of the domain-specific semantic model.

At item 206, the method may receive queries or may automatically generate queries from the semantic concepts, relationships and data using a query generation module. Thus, using the semantic concepts and relationships found in the previous step, queries for the query-response system can be automatically generated. Alternatively, it is possible for the decision maker 108 to enter queries in natural language or other ways, as described above.

When created automatically, a set of "standing" queries may be designed as a template. For example, in the medical field, fixed queries are "differential diagnosis". This is a list of potential hypotheses of diseases or other medical conditions that describe the patient's symptoms and abnormal findings. The diagnostic query templates referred to herein have blank slots for concepts such as symptoms, findings, past illnesses, current medications, allergies, and the like. Once the semantic concepts and relationships are identified, they fill the blanks in the template, resulting in a synthesized query. The concept of a template is a generic computational element for automatically constructing a set of related queries (queries) for a basic query-response system used to combine and return information related to specific information needed in the near future element.

There are many ways to implement templates. For example, queries may be automatically generated in item 206 based on what is known and unknown about the problem case. For example, in the medical domain, a diagnostic query can be generated if the symptoms and findings are identified in patient case information but no disease is found. The doctor may enter a query such as "What is the diagnosis?" And may rely on the rest of the context from the semantic concepts. Your doctor may fine-tune your query by specifying more restrictions, such as "Are you infected with these symptoms?"

In item 208, the method sends queries to the QA system 110. Thus, the method can automatically generate a plurality of responses for each query using a query-response module. Once the query is created, the query-response system 110 is called. To aid in the interpretation of future responses, a query can be transformed into multiple queries. Each query in this set may have a subset of the concepts found about the problem. For example, a clinical diagnostic query with symptoms, remarks, family history, and demographic characteristics information can generate a series of queries as follows - where the text in the <> sign matches the response found in the case text - What disease of condition can cause <symptom>? "; "What conditions can cause symptoms and symptoms?"; "What conditions can cause <symptom>, <symptoms>, and <family history>?"; "What conditions can cause symptoms, symptoms, family history, and demographic characteristics?" Etc. This build-up of information in the query allows the calculation of marginal contributions that contribute to the reliability of diagnoses, family history, and demographic characteristics information. Other strategies may be used to subdivide the query into a set of queries.

The method receives responses from a query-response system in item (210). For each query submitted, the query-response system 110 returns a list of responses, the reliability of the responses, the evidence dimensions, and the evidence sources. The reliability of each response may be a number between 0 and 1, for example. This reliability is constructed from various answer scorers in the query-response system and evaluates the accuracy of the response according to the various dimensions of the evidence sources. For example, a candidate response to a medical diagnostic query can be evaluated in terms of the semantic type of answer. If the response is a type of disease or medical condition that can be considered a diagnosis, the score for this dimension will be high. For all responses to the query, the passages of the domain knowledge from which the response was extracted are also available from the query-response system. This may be snippets of text that match the structure of the query or entire documents returned from the search components during the query-response process. For each phrase, citations to the original source of information are also recorded.

In item 212, the method further automatically calculates confidence values for each of the responses based on values for various evidence dimensions associated with the problem-solving area. Each evidence level may be based on various semantic concepts and relationships found in problem case information 102, as described by the method of item 204. For example, in the medical domain, these may be patient symptoms, findings, family history, demographic characteristics, and the like.

The processes described methods of making multiple queries that hold the concepts of the subsets found in the problem text. An estimate of the marginal contribution of these concepts can be generated by analyzing the responses and the reliability of their responses to these queries. For example, for the generated queries, the marginal effects of symptoms, findings, family history, and demographic characteristics are calculated. Other techniques can be used to achieve this.

At point 214, the method displays information for supporting decision making. The list of responses is displayed with the reliability of the response that the decision maker 108 will evaluate (see, e.g., FIG. 4). Therefore, the method uses the input / output module for user inquiry to output queries, responses, corresponding confidence values, links to evidence sources, and values for each evidence dimension. In addition, decision-makers can further investigate each evidence dimension by examining each piece of evidence, and can also investigate its associated sources. For example, the evidence may be a supporting passage, a reasoning chain, or a database fact. Similarly, examples of related sources include journal articles, textbooks, and databases. In addition, when outputting a value for each evidence dimension, this embodiment can show the amount that each evidence dimension contributes to the corresponding confidence value (e.g., on a magnitude or percentage basis) and the value of each evidence dimension Lt; RTI ID = 0.0 &gt; changes in corresponding confidence values. &Lt; / RTI &gt;

Embodiments of the present application also automatically and continuously update diagnostic responses, corresponding confidence values, and values for each evidence dimension based on revisions to the problem case information (query-response module, To generate revised queries, responses, corresponding confidence values, and the like. The method may also automatically output changed queries, responses, and / or corresponding confidence values when the difference threshold is exceeded. This "difference threshold" refers to a time period (e.g., ~ time, ~ week, ~ month), the amount by which one or more responses change (Yes / no) change, number of varying responses, etc.) and / or a change in the confidence value (a percent change in confidence, a change in confidence polarity, etc.).

Thus, the decision support application 104 may be able to provide the decision maker 108 with the most reliable responses and evolving semantic concepts, relationships and other related data (e.g., For example, patient medical data). &Lt; / RTI &gt; Given the same input (as done conventionally), rather than providing static applications that always provide the same responses, embodiments of the present application continue to update the values and relationships of the values of each evidence dimension, Thereby changing the reliability values. When the confidence values of the potential responses change, the highest highly recommended responses may also change, so that when the problem case information evolves over time, the decision maker dynamically sets the different best answers . &Lt; / RTI &gt;

Therefore, embodiments of the present application are more practical than systems that generate responses and confidence values based on predetermined, fixed criteria that are rarely changed (or only changed when periodic updates (e.g., software updates) Provide advantages. By taking action dynamically, for example, in the medical domain, it is possible to take a variety of actions based on changes in evolving demographic characteristics, evolving medical discovery, evolving drug conflicts, evolving side effects information, Previous responses and recommendations may change. Thus, embodiments of the present application may change the policy of medical treatment recommendations for a patient, since other data within the area knowledge content 112 evolves over time, even though the patient does not undergo a personal change . This enables medical providers to have a fully automated system to uniformly prescribe the best medical treatment for their patients as the medical advances and demographic characteristics change over time.

In many areas, the response with the highest reliability does not have to be an appropriate response, as there are several possible explanations for a problem. For example, in the medical field, various diseases can cause a set of symptoms to a patient. In addition to displaying a list of responsibilities and the reliability of those responses, one or more responses can be selected and drilled down to the dimensions of evidence. Figure 4 is a flow diagram illustrating the contribution of each dimension value of evidence for the overall confidence of a response from evidence sources. The output shown in Figure 4 compares each dimension across multiple responses. Figure 4 shows the marginal contribution of the evidence according to the dimensions of the present disease (402), family history (404), findings (406), and demographic characteristics (408) for the four disease responses For example. In this example, "dimensions" are "current disease," "findings," "family history," and "demographic characteristics," each of which has its own value. A comparative analysis of multiple responses according to these levels of evidence allows the decision maker 108 to consider and visualize trade-offs in evidence to arrive at a decision.

The decision maker 108 may also drill down further into the dimensions of each response and evidence to examine the supporting pieces of evidence that justify the answer along that dimension. For example, the source of the phrase, the type of the source (textbooks, handbooks, journal articles, web content) and links to its source are provided so that the decision maker can further examine and validate it.

The method may also identify missing information in item 216. More specifically, this embodiment automatically identifies information related to responses not included in the problem case information as missing information, and automatically identifies the amount that the missing information affects the corresponding confidence values In both cases, a query-response module is used) and the information is output to the user.

If the responses returned by the query-response system and the evidence of those responses are not sufficient to reach a decision, then the application 104 may be used to identify the missing information that may affect reliability in responses . With respect to a given response, the decision maker 108 may want to know what hypothetical information, if provided, can cause the greatest change in reliability. For example, in the medical domain, if the response is a disease, the missing information may be a lab test that confirms or excludes the disease. It may also be other indications or symptoms not explicitly indicated to the patient. In practice, there may be a large amount of missing information associated with the response, and embodiments of the present application may rank the missing information. The characteristics that can be used to rank the potential values of the missing information may be factors such as the cost of obtaining this information, the time spent, and the amount that the missing information affects the reliability of the response.

If the two responses show similar reliability, making it difficult to choose between the two, it may be helpful to identify the missing information that can cause the greatest difference between the two. For example, in the medical domain, the responses may be two related diseases, and the missing information may be a laboratory test designed to distinguish between the two. This evidence can raise or lower the reliability of an answer and can therefore help to establish an accurate diagnosis in the case of a medical diagnostic system.

The identification of the missing information need not be done only by the decision maker 108. When a certain criterion is met and, for example, the reliability of the two top-level responses is very close, the application 104 may take the initiative and ask for the missing information automatically.

Once the missing information is identified, the decision maker 108 must look up the missing information using procedures specific to that area. In the medical field, it may be necessary to direct laboratory tests or to require more information from the patient. When the missing information becomes available, it is sent back to the decision-support phase as described above and a new iteration of the query-response and decision support process shown in Fig. 2 begins.

Figures 5-14 are schematic diagrams of screenshots that may be presented to users. In FIG. 5, two patient profiles 502, 504 are shown on screen shot 500. Additional profiles for additional patients may also be generated. In FIG. 6, the first patient 502 has been selected and described the symptoms listed in the History of Present Illness section 506. This information may be entered into the electronic health record (EHR) by a health care professional or may be made available only in the system by typing it in box 506. The proposed system automatically pulls relevant information from the health record or the text field above, finds related concepts using analytics, classifies them into belongings of symptoms dimension And automatically generate a query listed in the query field 508. [ Alternatively, the user (physician / patient) may directly enter the query into the query field. The user can then proceed by clicking the "Ask Watson" button 510. Also shown in FIG. 6 is the "Evidence" button 512, which is discussed below.

In Figure 7, the decision-support application 104 generates a set of possible responses for the query with the confidence scores associated with each response and they are displayed in the area 514. In Figure 8, after confirming the status, the user may enter the status into the current disease history section, or the status may be automatically extracted from the EHR. The physician can then make another query or let the decision-support application 104 automatically generate another query.

In FIG. 9, the process continues by adding new information, item 506, analyzed and grouped to related dimensions, such as current disease, family history, In FIG. 10, the same process continues as more information is continually added to item 506, thereby refining the potential diagnosis more precisely. In Figure 11, the process begins at point 514 where the decision-support application 104 has a higher confidence in the other potential responses and the correct diagnosis for a particular patient is Lyme disease . In the example of Figure 11, the information contained in these dimensions includes case information ('uveitis,' 'circular rash ...', 'arthritis',' Connecticut '). Each evidence-level figure comes from the presence of information contained in these dimensions within the medical content in the context of a hypothesized answer (eg, Lyme disease).

In Figure 12, the decision-support application 104 allows the user to select a response lime disease to view the evidence profile 516 for the response. The application 104 informs the levels of evidence and their associated contribution to the diagnosis of Lyme disease. The user can then select more specific dimensions and search for pieces of evidence that contribute to this dimension. Lastly, the application 104 may determine that the entire document that brought the sculptures by clicking on one of the links 518 shown in FIG. 12, such as a textbook, a journal, or a website, It allows us to see. In FIG. 13, the decision-support application 104 is again shown, except that in this case the application 104 is intended to examine possible treatments to treat the identified condition. In Figure 14, new information was added or automatically extracted from the patient &apos; s medical records in connection with the appropriate treatment for the identified condition. In this case, the application 104 has identified that the patient is allergic to penicillin and that the patient is pregnant. The application 104 uses this information to find the appropriate treatment and, in this case, displays the confidence score for the particular treatment option.

As one of ordinary skill in the art will appreciate, embodiments of the present application may be implemented as a system, method, or computer program product. Accordingly, embodiments of the present application may take the form of a hardware embodiment entirely in the form of a software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware, Generally referred to herein as a "circuit "," module "or" system ". Embodiments of the present application may also take the form of a computer program product embodied in one or more computer-readable media (s) in which computer readable program code is embodied.

Any combination of one or more computer-readable media (s) may be used. The computer readable medium may be a computer readable storage medium or a computer readable storage medium. The computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media may include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM) Erasable and programmable read-only memory (EPROM or flash memory), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium can be any type of tangible medium that can hold or store a program to be used by or in connection with an instruction execution system, apparatus, or device.

Computer readable signal media can include computer readable program code, for example, a propagated data signal implemented as a baseband or as part of a carrier wave. Such propagation signals may take a variety of forms including, but not limited to, electromagnetic, optical, or any combination thereof. The computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that is capable of communicating, propagating, or transmitting a program to be used by or in connection with an instruction execution system, apparatus, or device.

The program code embodied on the computer readable medium may be transmitted using any suitable medium, including but not limited to, wireless, wired, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The computer program code for carrying out the operations of the embodiments of the present application may be implemented in any one of a number of ways, including object-oriented programming languages such as Java, Smalltalk, C ++, etc., and conventional procedural programming languages such as "C & Or more programming languages. The program code may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer, partly on the remote computer, or entirely on the remote computer or server. In this last case, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made Can be done on an external computer).

Embodiments of the present application are described below with reference to flow and / or block diagrams of methods, devices (systems), and computer program products in accordance with embodiments of the present invention. Each block of flow diagrams and / or block diagrams, and combinations of blocks in flow diagrams and / or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, or other programmable data processing apparatus to create a machine so that these instructions may be executed via the computer or other programmable data processing apparatus And / or to create means for implementing the functions / operations specified in the blocks or blocks of the block diagrams.

These computer program instructions may also be stored in a computer readable medium to cause the computer, other programmable data processing apparatus, or other devices to perform the functions stored in the computer readable medium, such as instructions and / / Instructions to produce an article of manufacture. &Lt; RTI ID = 0.0 &gt; The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operating steps to be performed on a computer, other programmable apparatus, or other devices to produce a computer implemented process , Such that instructions executing on a computer or other programmable device may provide processes for implementing the functions / operations specified in the block diagrams and / or block diagrams of the flowcharts and / or block diagrams.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products in accordance with the various claims herein. In this regard, each block in the flowchart or block diagram may represent a portion of a module, segment, or code that includes one or more executable instructions for implementing the specified logical function (s). It should be noted that the functions mentioned in the blocks in some other implementations may occur differently from the order mentioned in the drawings. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order, depending on the function involved. Each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, may be implemented with specialized hardware-based systems that perform specified functions or operations, or combinations of special purpose hardware and computer instructions As will be appreciated by those skilled in the art.

While this specification contains a detailed description of cloud computing, it is necessary to understand in advance that the implementation of the recited teachings is not limited to a cloud computing environment. Rather, embodiments of the present application may be implemented with any other type of computing environment now known or later developed. Cloud computing is a convenient on-demand network for a shared pool of configurable computing resources (e.g., network, network bandwidth, server, processing, memory, storage, applications, virtual machines, and services) A service delivery model for enabling access and may be provisioned and released with minimal management effort or interaction with a service provider. The cloud model includes at least five characteristics, at least three service models, and at least four deployment models.

The characteristics are as follows:

On-demand self-service: Cloud consumers unilaterally provision computing capabilities such as server time and network storage, as needed, without the need for human interaction with service providers. )can do.

Broad network access: Ability to use capabilities across a network and support for heterogeneous thin or thick client platforms (eg, mobile phones, laptops, and The capabilities are accessed through standard mechanisms that facilitate use by the PDA.

Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources can be pooled, Resources are allocated and reallocated dynamically according to demand. Consumers typically do not have a sense of location (or location) in that they are able to locate at a high level of abstraction (eg, country, region, or data center) independence exists.

Rapid elasticity: Capabilities can be quickly and resiliently, in some cases automatically provisioned to scale out quickly and released to scale in quickly. The customer seems to have no limit on the capabilities available for provisioning and can purchase as much as he wants at any time.

Measured service: The cloud system leverages metering capability at a certain level of abstraction appropriately to the type of service (eg, storage, processing, bandwidth, and active user account) Automatically control and optimize usage. Resource usage can be transparent to both providers and consumers of services that are monitored, controlled and reported and used.

The service model is as follows:

Software as a Service (SaaS): The ability to be offered to consumers is the use of the provider's applications running on the cloud infrastructure. The applications are accessible from a number of client devices through a thin client interface (e.g., web-based email) such as a web browser. Consumers do not manage or control basic cloud infrastructures, including the capabilities of networks, servers, operating systems, storage, or personal applications, but there may be exceptions, such as limited user- have.

Platform as a Service (PaaS): The ability to be provided to consumers is the ability to create consumer-created (consumer-created) software in the cloud infrastructure, ) Applications. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they also have control over consumer deployed applications and may have control over the configuration of the application hosting environment. have.

Infrastructure as a Service (IaaS): The ability to provide to consumers is the provisioning of processing, storage, networking and other basic computing resources, where the consumer is able to use any software, including the operating system and applications, Deploy and run. Consumers do not control or control the underlying cloud infrastructure but have control over the operating system, storage, deployed applications, and have limited control over selected networking components (e.g., host firewalls).

The deployment models are as follows:

Private cloud: This cloud infrastructure operates solely for an organization. The cloud infrastructure can be managed by the organization or a third party and can be on-premises or off-premises. Community cloud: A cloud infrastructure is shared by multiple organizations and supports specific communities that share interests (for example, mission, security requirements, policy, and compliance considerations). The community cloud can be managed by a group or a third party, and can be on campus or off campus.

Public cloud: This cloud infrastructure is made available to the general public or large industrial groups and is owned by organizations that sell cloud services.

Hybrid cloud: This cloud infrastructure is made up of two or more clouds (private, community, or public) and they are maintained as unique entities, but data and application mobility (for example, And cloud-bursting for load balancing).

The cloud computing environment is focused on services that focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

Referring to Fig. 15, a schematic diagram of an example of a cloud computing node is shown. The cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to limit the scope or functionality of the embodiments of the invention described herein.

Regardless, the cloud computing node 10 may implement and / or perform the functions presented above. At the cloud computing node 10, there is a computer system / server 12, which may be operated in many other general purpose or special purpose computing system environment or configurations. Examples of well known computing systems, environments and / or configurations that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, A handheld or laptop device, a multiprocessor system, a microprocessor-based system, a set top box, a programmable consumer electronics product, a network PC, a minicomputer system, a mainframe computer system, But are not limited to, distributed cloud computing environments that include,

The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement concrete abstract data types. The computer system / server 12 may be implemented in a distributed cloud computing environment in which tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may reside on local and remote computer system storage media, including memory storage devices.

As shown in FIG. 15, the computer system / server 12 in the cloud computing node 10 is shown in the form of a general purpose computing device. The components of computer system / server 12 include one or more processors or processing units 16, system memory 28, and bus 18, which may include, but are not limited to, RTI ID = 0.0 &gt; (28) &lt; / RTI &gt; to the processor (16). The bus 18 may be one or more of several types of buses, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using various bus architectures. Structure. By way of example and not limitation, such architectures include an industry standard architecture (ISA) bus, a microchannel architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Committee (VESA) Peripheral component interconnect (PCI) bus. The computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media accessible by computer system / server 12, including volatile and non-volatile media, removable and non-removable media.

The system memory 28 may include a computer-readable medium, such as random access memory (RAM) 30 and / or cache memory 32, in the form of volatile memory. The computer system / server 12 may further include other removable / non-removable volatile / non-volatile computer system storage media. As an example only, the storage system 34 may be provided for reading and writing to / from non-removable, nonvolatile magnetic media (not shown and typically referred to as a "hard drive"). Although not shown, a magnetic disk drive for reading and writing to / from a removable non-volatile magnetic disk (e.g., a "floppy disk") and a removable nonvolatile optical disk such as a CD-ROM, DVD- An optical disk drive for reading or writing to / from the disk may be provided. In this case, each may be connected to the bus 18 by one or more data media interfaces. As further shown and described below, the memory 28 includes at least one program product having a set of (e.g., at least one) program modules configured to perform the functions of the embodiments of the present invention . The program / utility 40 with a set of (at least one) program modules 42 may be stored in the memory 28 (by way of example only, and not by way of limitation) The above application programs, other program modules, and program data may also be stored in memory. The operating system, one or more application programs, other program modules, and / or some combination of program data may include an implementation of a networking environment. Program modules 42 generally implement the functions and / or methods of embodiments of the present invention as described herein. The computer system / server 12 may also include one or more external devices 14, such as a keyboard, pointing device, display 24, and the like; One or more devices that allow a user to interact with the computer system / server 12; (E.g., a network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices and / or the computer system / server 12. Such communication may occur via input / output (I / O) interfaces 22. In addition, the computer system / server 12 may communicate with one or more networks (e. G., The Internet) via a network adapter 20, such as a local area network (LAN), a general wide area network Lt; / RTI &gt; As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. It is to be understood that other hardware and / or software components, although not shown, may be used with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

Referring to FIG. 16, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 and may be, for example, a personal digital assistant (PDA) or cellular telephone 54A, Local computing devices used by cloud consumers, such as computer 54B, laptop computer 54C, and / or automotive computer system 54N, may communicate. The nodes 10 can communicate with each other. The nodes may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, a hybrid cloud, or combinations thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure services, platform services, and / or software services so that cloud consumers do not need to maintain resources on the local computing device. The types of computing devices 54A-N shown in FIG. 16 are merely exemplary in nature and the computing nodes 10 and cloud computing environment 50 may be any type of network and / or network addressable connection It is necessary to understand that it can communicate with any type of computer device (e.g., using a web browser).

Referring now to FIG. 17, there is shown a set of functional abstraction layers provided by the cloud computing environment 50 (of FIG. 16). It should be understood that the components, layers, and functions illustrated in FIG. 17 are for illustrative purposes only and that embodiments of the invention are not limited thereto. As shown, the following layers and corresponding functions are provided: The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes, such as IBM ^® zSeries ^® systems; The RISC (Reduced Instruction Set Computer) architecture based servers, IBM pSeries ^® system one example; IBM xSeries ^® systems; IBM BladeCenter ^® systems; Storage devices; Networks and networking components, and the like. Examples of software components include network application server software, such as IBM WebSphere ^® application server software; And database software, such as IBM DB2 ^® database software. (IBM, zSeries, pSeries, xSeries, BladeCenter, WebSphere, and DB2 are trademarks of International Business Machines Corporation registered in various countries worldwide.) The virtualization layer 62 is an example of the following virtual entities: Lt; RTI ID = 0.0 &gt; virtual servers; &lt; / RTI &gt; Virtual storage; Virtual networks including virtual private networks; Virtual applications and operating systems; And virtual clients. In one example, management layer 64 may provide the functions described below. Resource Provisioning provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing provides cost tracking as resources are used in the cloud computing environment and billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification of cloud consumers and tasks and protection against data and other resources. The user portal provides consumers and system administrators with access to the cloud computing environment. Service Level Management provides cloud computing resource allocation and management to meet the required service levels. Service Level Agreement (SLA) Planning and Fulfillment provides pre-arrangement and procurement of cloud computing resources that anticipate future requirements according to SLAs.

The workload layer 66 provides examples of functions in which the cloud computing environment can be used. Examples of workloads and functions that may be provided from this layer include Mapping and Navigation; Software Development and Lifecycle Management; Virtual Classroom Education Delivery; Data Analytics Processing; Transaction Processing; And Decision-Support for problem solving using a query-response system.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the expressions "a", "an" and "the" or "the" are intended to include the plural forms, unless the context clearly dictates otherwise. Respectively. The word " comprises "and / or" comprising ", when used in this specification, indicate that the presence of stated features, integers, steps, operations, elements, and / And does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

The structures, materials, acts, and equivalents of all means or step-plus functional elements in the claims below may be combined with other claimed elements, Material, or action for performing the function described herein. The description of the embodiments of the present application is presented for purposes of illustration and description only and is not intended to be exhaustive or to limit the invention to the precise form disclosed. It will be apparent to those skilled in the art that many modifications and variations may be made thereto without departing from the scope and spirit of the invention. The embodiments are chosen and described in order to best explain the principles of the invention and its practical application and to the particular use contemplated by the inventors for various embodiments with various modifications to those skilled in the art It was chosen and described in order to make it understandable.

Claims (23)

A method for problem solving, the method comprising:
Receiving a first query and problem case information on a problem within a specific domain by a decision-support system for problem resolution, the decision-support system comprising: A computerized device capable of accessing evidence sources holding domain knowledge content including knowledge within a computer system, the computer device communicating with a repository, Instructions are programmed with instructions for causing the computer device to perform a processing step;
Extracting related information from the problem case information and extending the original query using the related information to generate a specific natural language query;
Generating answers to the natural language query by applying natural language processing techniques to the regional knowledge content in the evidence sources;
Computing numerical values of evidence dimensions for each of the responses;
Computing corresponding confidence values for each of the responses based on the values of each of the evidence dimensions;
Outputting the numerical values of each of the evidence dimensions for the natural language query, the responses, the corresponding confidence values, and the one or more selected responses to a decision-maker and to the repository, The numerical values of each of the query, the responses, the corresponding confidence values and the evidence dimensions are displayed on the display for the decision maker and can be used by the decision maker to make decisions regarding the problem;
Storing, in the repository, the natural language query, the responses, the corresponding confidence values, and the respective values of each of the evidence dimensions;
In response to the new region knowledge being added to the region knowledge content in the evidence sources, the responses to the natural language query, the numerical values of the evidence dimensions for each of the responses, and the corresponding confidence values And determining whether pre-generated responses and corresponding confidence values stored in the repository have changed based on the new domain knowledge; And
Automatically sending an alert to the decision maker based on a change occurring in either of the pre-generated responses and the corresponding confidence values, the alert being used by the decision maker to re- Possible -
Way. 2. The method of claim 1,
Automatically analyzing the problem case information to identify semantic concepts; And
Further comprising automatically generating the natural language query from the semantic concepts.
Way. delete delete delete delete 2. The method of claim 1, wherein the initial query is a form of at least one of a free-form query, a free-form statement, and keyword search.
Way. delete 2. The method of claim 1,
Further comprising, for the selected evidence dimension, outputting each piece of evidence supporting the selected evidence dimension and an associated provenance for the evidence.
Way. 10. The method of claim 9, wherein the method further comprises:
Removing a selected piece of evidence from the evidence sources contributing a specific numerical value of a specific response to a specific numerical value of the specific response;
Recalculating the particular value of the particular evidence dimension from which the selected evidence has been removed to generate a recalculated value; And
Further comprising recalculating a new confidence value of said specific answer of said specific response based on said recalculated value.
Way. 2. The method of claim 1,
Identifying information related to the responses not included in the problem case information as missing information; And
Further comprising the step of outputting a request to add said missing information to said problem case information.
Way. delete A system, comprising:
A repository that maintains problem case information about a problem in a particular area; And
A computer processor operatively coupled to the repository and connected to evidence sources holding domain knowledge content comprising knowledge within the particular domain,
The computer processor receiving an initial query and the problem case information from the repository;
The computer processor automatically extracts relevant information from the problem case information and expands the original query using the related information to generate a natural language query;
The computer processor generating responses to the natural language query by applying natural language processing techniques to the region knowledge content in the evidence sources;
The computer processor calculating values of evidence dimensions for each of the responses;
The computer processor calculating corresponding corresponding confidence values for each of the responses based on the values of each of the evidence dimensions;
The computer processor outputting the natural language query, the responses, the corresponding confidence values, and the values of each of the evidence dimensions to a decision maker and the repository;
The numerical values of each of the natural language query, the responses, the corresponding confidence values and the evidence dimensions are displayed on the display for the decision maker and can be used by the decision maker to make decisions regarding the problem;
The repository storing the natural language query, the responses, the corresponding confidence values, and the respective values of each of the evidence dimensions;
Wherein the computer processor is further programmed to perform the steps of: responsive to adding the new domain knowledge to the region knowledge content in the evidence sources, the responses to the natural language query, the numerical values of the evidence dimensions for each of the responses, Automatically update the corresponding confidence values and determine whether the previously generated responses and the corresponding confidence values stored in the repository have changed based on the new domain knowledge; And
Wherein the computer processor automatically sends an alert to the decision maker based on a change that occurs in either the pre-generated responses and the corresponding confidence values, and the alert is generated by the decision maker to re- Available by,
system. delete delete delete delete delete delete delete delete delete A computer readable storage medium comprising:
The computer readable storage medium storing computer readable program code comprising instructions executable by a computerized device to perform a method comprising:
Receiving an initial query and receiving problem case information on a problem in a particular area;
Extracting related information from the problem case information and extending the original query using the related information to generate a specific natural language query;
Generating answers to the natural language query by applying natural language processing techniques to the regional knowledge content in the evidence sources, wherein the domain knowledge content includes knowledge in the specific region;
Computing numerical values of evidence dimensions for each of the responses;
Computing corresponding confidence values for each of the responses based on the values of each of the evidence dimensions;
Outputting to the decision-maker and to the repository the numerical values of each of the evidence dimensions for the natural language query, the responses, the corresponding confidence values, and the one or more selected responses; The numerical values of each of the query, the responses, the corresponding confidence values and the evidence dimensions are displayed on the display for the decision maker and can be used by the decision maker to make decisions regarding the problem;
Storing, in the repository, the natural language query, the responses, the corresponding confidence values, and the respective values of each of the evidence dimensions;
In response to the new region knowledge being added to the region knowledge content in the evidence sources, the responses to the natural language query, the values of the evidence dimensions for each of the responses and the corresponding confidence values are automatically Updating the pre-generated responses and the corresponding confidence values stored in the repository based on the new domain knowledge; And
Automatically sending an alert to the decision maker based on a change occurring in either of the pre-generated responses and the corresponding confidence values, the alert being used by the decision maker to re- Possible -
Computer readable storage medium.

Images