JP2021520551A - Presentation of augmented reality associated with the patient's medical condition and / or treatment - Google Patents

Abstract

To perform an augmented reality display via a head-mounted display (HMD) system that shows the part of the patient's body that corresponds to the patient's medical condition and / or treatment, overlaid on the patient's actual view. Mechanism is provided. Through the HMD system, real-time images of the patient's body parts viewed by the medical professional are captured. One or more body parts of the patient are identified within the real-time image. One or more identified body parts are correlated with the patient's electronic medical record (EMR) showing the medical condition and / or treatment associated with the patient. An augmented reality display of one or more parts of the patient's body corresponding to the patient's medical condition and / or treatment is then generated within the HMD system to overlay a real-time image of the patient's body part. NS.

Description

The present application generally relates to improved data processing devices and methods, more specifically to provide augmented reality representations associated with the patient's medical condition and / or treatment. Regarding the mechanism for presenting to professional).

An electronic health record (EHR) or electronic medical record (EMR) is a systematic collection of patients and populations that electronically stores health information in a digital manner. These records can be shared across different medical settings. Records are shared through networked enterprise-scale information systems or other information networks and exchanges. EMR includes demographics, medical history, medication and allergies, immunization status, laboratory results, radiographic images, vital signs, personal statistics such as age and weight, and billing information. It can contain various data.

The EMR system is designed to accurately store data and capture past patient status. The EMR system eliminates the need to find out the patient's previous paper medical records and helps ensure that the data is accurate and easy to read. The EMR system reduces the risk of data duplication and reduces the risk of document loss because there is only one changeable file, which means that the file is likely to be up to date. Because the digital information is searchable and in a single file, EMR is more useful when extracting medical data to investigate possible trends and long-term changes in patients. Widespread adoption of EMR can also facilitate population-based studies of medical records.

This overview is provided to introduce, in a simplified form, the selection of concepts further described herein in the detailed description. This summary is not intended to identify the key factors or essential features of the claimed subject matter and is intended to be used to limit the scope of the claimed subject matter. It's not a thing.

In one exemplary embodiment, it is extended through a head-mounted display (HMD) that shows the part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the patient's actual view. A method in a data processing system for implementing a real-life display is provided. An exemplary embodiment captures a real-time image of a part of the patient's body viewed by a healthcare professional via an HMD system by a capture mechanism of a cognitive healthcare system. An exemplary embodiment identifies one or more body parts of a patient in a real-time image. An exemplary embodiment correlates one or more identified body parts with a patient's electronic medical record (EMR) showing the medical condition and / or treatment associated with the patient. An exemplary embodiment provides an augmented reality display of one or more parts of the patient's body that corresponds to the patient's medical condition and / or treatment, overlaying a real-time image of the part of the patient's body within the HMD system. To generate.

In another exemplary embodiment, a computer program product comprising a computer-enabled or readable medium having a computer-readable program is provided. A computer-readable program, when run on a computing device, causes the computing device to perform various of the actions outlined above with respect to exemplary embodiments of the method and combinations thereof.

In yet another exemplary embodiment, a system / device is provided. The system / device can include one or more processors and memory coupled to the one or more processors. Memory contains instructions that, when executed by one or more processors, cause one or more processors to perform various of the operations outlined above with respect to exemplary embodiments of the method and combinations thereof. be able to.

These and other features and advantages of the present invention will be explained or will be apparent to those skilled in the art in light of the following detailed description of the exemplary embodiments of the present invention.

The present invention, and preferred modes of its use, additional purpose and advantages, will be best understood by reference to the following detailed description of exemplary embodiments when read in conjunction with the accompanying drawings. ..

A schematic diagram of one exemplary embodiment of a cognitive system in a computer network is shown. FIG. 6 is a block diagram of an exemplary data processing system in which aspects of an exemplary embodiment are implemented. An exemplary embodiment of a cognitive system processing pipeline for processing inputs to generate overlays within an augmented reality display (HMD) system. show. A head-mounted display, according to an exemplary embodiment, showing the part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as seen by the medical professional. (HMD) Shows an exemplary flowchart of actions performed by a cognitive healthcare system when performing an augmented reality display through a system.

In current medical assessments, medical professionals have to sort out large amounts of medical information about patients in an attempt to find the best information for their treatment. In general, these assessments analyze the large amount of data presented in a patient's EMR as health professionals have access to physical medical files about the patient or the increased use of electronic health records (EMRs). Need that. This requires a lot of time and effort on the part of the medical professional who already has limited time to treat the patient. For this reason, patients often do not know the patient personally or are not treated on an individual basis by a medical professional and have sufficient time for the patient at the time of the scheduled appointment. I feel like I'm not spending, that is, the patient's EMR, rather than having the medical professional actually interact with the patient and look at the patient to get an idea of how to treat the patient. I feel too busy searching for data or physical files.

Thus, an exemplary embodiment is worn to indicate a part of the patient's body that corresponds to the patient's medical condition and / or treatment, which is superimposed on the actual view of the patient as seen by the medical professional. A mechanism for implementing an augmented reality display via a head-mounted display (HMD) system such as a type headset, eyeglasses, etc. is provided. The mechanism of the present invention captures an image of a part of the patient's body as seen by a medical professional. Based on the patient's body parts being viewed, the mechanism identifies the corresponding body parts in the view and indicates those body parts to the patient's medical condition and / or treatment associated with the patient. Correlate with electronic medical record (EMR) data. In some cases, facial recognition can be used to identify a particular patient being seen.

The superpositioned graphical representation on the patient's body can be specific to the medical condition and implementation. That is, the superimposed graphical representation is a graphical image of the medical condition, highlighting of affected or parts of the body that need further examination, textual information of test results, treatment options, medical code. Etc. can be included. PMDA also provides access to a medical corpus of annotated data for multiple media views, the mood of a given patient, the time of day, the schedule of a medical professional. It enables real-time selection of media suitable for availability and the like. Regarding the schedule of medical professionals, depending on the schedule, it is possible to display a more concise one (basic organ model) when availability is limited, while a detailed one (patient) when availability is long-term. Surgery techniques simulated on the organs of the patient) can be displayed.

Before beginning a more detailed discussion of the various aspects of the exemplary embodiment, first, throughout this description, the term "mechanism" is meant to refer to an element of the invention that performs various actions, functions, etc. Please understand that it is used. As the term is used herein, "mechanism" can be an implementation of a function or embodiment of an exemplary embodiment in the form of a device, procedure, or computer program product. In the case of a procedure, the procedure is performed by one or more devices, devices, computers, data processing systems, and the like. In the case of a computer program product, the logic represented by the computer code or instructions embodied in or on the computer program product is executed by one or more hardware devices to perform or perform a function. Performs an action associated with a particular "mechanism". Therefore, the mechanisms described herein can be implemented as dedicated hardware, software executed on general purpose hardware, software instructions stored on a medium, and the instructions are dedicated or general purpose hardware. It can be easily executed by a procedure or method for performing a function, or a combination of any of the above.

The present description and claims cover "one (a)", "at least one of" and "one or more of" with respect to specific features and elements of the exemplary embodiments. (One or more of) "may be used. It is understood that these terms and phrases are intended to state that at least one of the particular features or elements present within a particular exemplary embodiment is present, but more than one can be present. sea bream. That is, these terms / phrases are not intended to limit the scope of the description or claims to a single feature / element that exists, or to require the existence of multiple such features / elements. Conversely, these terms / phrases require only at least a single feature / element that has the potential for multiple such features / elements within the scope of the description and claims.

Moreover, as used herein with respect to the description of embodiments and features of the present invention, the use of the term "engine" may be attributed to, or performed by, an engine, an operation, a step, a step. It should be understood that it is not intended to limit any particular implementation in order to achieve, execute, or both process, etc. The engine includes, but is not limited to, the use of a general purpose processor and / or a dedicated processor in combination with appropriate software loaded or stored in machine-readable memory and executed by the processor. It can be software, hardware, and / or firmware, or any combination thereof that performs a specified function, not limited to this. Furthermore, unless otherwise specified, any name associated with a particular engine is for convenience of reference and is not intended to be restricted to a particular implementation. In addition, any function resulting from an engine may be performed equally by multiple engines, incorporated into and / or combined with the function of another engine of the same or different type, or one of various configurations. Or it can be distributed across multiple engines.

In addition, the following description further illustrates the exemplary implementation of the exemplary embodiment with a number of different examples of the various elements of the exemplary embodiment to aid in understanding the mechanism of the exemplary embodiment. I want to be understood. These examples are not limited and do not cover the various possibilities for implementing the mechanisms of exemplary embodiments. It is possible that there are many other alternative implementations of these various elements that can be used in addition to or in place of the examples provided herein without departing from the spirit and scope of the invention. It will be apparent to those skilled in the art in light of this description.

As mentioned above, the invention refers to a part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as seen by a medical professional. Provided is a mechanism for implementing an augmented reality display via a head-mounted display (HMD) system such as a wearable headset, eyeglasses, or the like. Illustrative embodiments can be used in many different types of data processing environments. FIGS. 1-3 are provided below as exemplary environments in which the embodiments of the exemplary embodiments can be implemented to provide context for the description of specific elements and functions of the exemplary embodiments. It should be understood that FIGS. 1-3 are merely examples and are not intended to assert or imply any limitation on the environment in which aspects or embodiments of the invention can be practiced. Many modifications to the embodiments shown can be made without departing from the scope of the invention.

1 to 3 are wearable, showing the part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as seen by the medical professional. A description of an exemplary cognitive system for implementing an augmented reality display via a head-mounted display (HMD) system such as a headset, eyeglasses, etc. is directed. Therefore, in order to identify the medical condition and / or treatment associated with a patient, the cognitive system implements a requirements processing pipeline, a requirements processing method, and a requirements processing computer program that implements the mechanisms of an exemplary embodiment. Implement the product. These requests can be provided as a structured or unstructured request message, a natural language question, or any other suitable form for requesting an action performed by a cognitive system. As described in more detail below, the particular application performed in the cognitive system of the present invention overlays the patient's medical condition on the actual view of the patient as seen by a medical professional. Alternatively, it is an application for performing an augmented reality display via a head-mounted display (HMD) system that shows a part of the patient's body corresponding to treatment or both.

It should be understood that a cognitive system may actually have multiple request processing pipelines, although it is shown as a single request processing pipeline in the example below. Each request processing pipeline can be trained separately and / or configured to process requests associated with different regions, or input requests (or use QA pipelines), depending on the desired implementation. It can be configured to perform the same or different analysis for the implementation question). For example, in some cases, the first request processing pipeline may be trained to work in response to input requests directed to identify the patient's medical condition, and the patient's physician may be associated with the patient's medical condition. It is possible to be able to see the part of. In other cases, for example, a requirements processing pipeline may provide different types of cognitive functions or different types of applications, such as one request processing pipeline being used to identify a patient's medical treatment. It can be configured to support and allow the nurse performing the patient's treatment to see the patient's site, etc. to which the applied treatment is associated.

In addition, each request processing pipeline can have an associated corpus (s) that they capture and operate, for example, one corpus is for medical condition documents and another corpus is. It is for medical treatment related documents in the above example. In some cases, each request processing pipeline may work for the same area of input questions, but different annotators or trained annotators to generate different analyzes and potential answers. Can have different configurations such as. The cognitive system routes input questions to the appropriate request processing pipeline, such as based on the determined area of the input request, and combines the final results produced by the processing performed by multiple request processing pipelines. Additional logic for evaluation, as well as other control and interaction logics that facilitate the use of multiple request processing pipelines, can be provided.

The present invention is described in the context of a cognitive system that implements one or more requirements pipelines that operate on requirements, but it should be understood that exemplary embodiments are not limited thereto. Rather, the mechanisms of the exemplary embodiments can operate on requests that are not presented as "questions", but with the associated corpus (s) and specific configuration information used to configure the cognitive system. , Formatted as a request to the cognitive system to perform cognitive actions on a particular set of input data.

As described in more detail below, the exemplary embodiment is a patient corresponding to the patient's medical condition and / or treatment, which is overlaid on the actual view of the patient as seen by the medical professional. The capabilities of the request processing pipeline can be integrated, extended and expanded with respect to the implementation of augmented reality displays that show parts of the body. For example, if the patient has a medical condition of ptosis, if the doctor looks at the patient's body through the augmented reality display of the head-mounted display (HMD) system, the patient's cecum is the patient's cul-de-sac through the augmented reality display of the HMD system. The lower abdomen of the body is shown to overlay the lower abdomen of the patient's body when looking at the augmented reality display of the HMD system.

It is intended that the mechanisms described in FIGS. 1-3 are merely examples and are not intended to state or imply any limitation on the type of cognitive system mechanism in which the exemplary embodiments are implemented. I want to be understood. Many modifications to the exemplary cognitive system shown in FIGS. 1 to 3 can be implemented in various embodiments of the invention without departing from the spirit and scope of the invention.

In summary, a cognitive system is a dedicated computer configured to have hardware logic and / or software logic (in combination with the hardware logic performed by the software) to emulate human cognitive functionality. A system, or a set of computer systems. These cognitive systems apply human-like characteristics to convey and process ideas, thereby having high precision and large resilience when combined with the inherent strength of digital computing. Can solve the problem. A cognitive system mimics the human thinking process and allows one or more computer-implemented cognitive operations to allow humans and machines to interact in a more natural way to expand and enhance human expertise and cognition. To execute. A cognitive system is a combination of artificial intelligence logic, such as natural language processing (NLP) -based logic, and dedicated hardware, software that runs on hardware, or dedicated hardware and software that runs on hardware. Includes machine learning logic that can be provided as any combination. The logic of a cognitive system implements cognitive behavior, such as, but not limited to, question answering, identification of related concepts within different parts of content within a corpus, eg, Internet web page search. Includes the generation of other types of recommendations such as intelligent search algorithms such as, medical diagnosis and treatment recommendations, such as items of interest to a particular user, potential new contact recommendations, etc.

IBM Watson ™ is one such cognitive system that can process human-readable languages and identify inferences between text passages much faster than humans, on a larger scale, and with human-like precision. This is an example. In general, such cognitive systems can perform the following functions: :
● Navigate the complexity and understanding of human language ● Capture and process vast amounts of structured and unstructured data ● Generate and evaluate hypotheses ● Weight responses based solely on relevant evidence ● Provide situation-specific advice, insights and advice ● Improve knowledge and learn through dialogue through machine learning processes each time ● Enable decision-making in the event of impact (contextual advice)
● Scale in proportion to the task ● Expand and strengthen human expertise and cognition ● Identify resonating human-like attributes and characteristics from natural language ● Various language-specific from natural language Guess the or agnostic attribute ● Highly relevant memory (memorize and recall) from data points (image, text, voice)
● Predict and perceive with situational awareness that mimics human cognition based on experience ● Answer questions based on natural language and concrete evidence.

In one embodiment, the cognitive systems use the request processing pipeline, and / or processing requests that may or may not be presented as natural language requirements, to the requests presented to these cognitive systems. Provide a mechanism for responding. A request processing pipeline is an artificial intelligence application that runs on data processing hardware that responds to requests for a given subject area presented in natural language. Request processing pipelines include input on the network, corpus of electronic documents or other data, data from content creators, information from one or more content users, and others from other possible input sources. Receives input from various sources, including such input. A data storage device stores a corpus of data. Content creators create in-document content that is used as part of a corpus of data in the request processing pipeline. The document can include any file, text, article, or source of data used in the request processing system. For example, a request processing pipeline accesses a body of knowledge about an area or subject area, such as a financial area, a medical area, a legal area, etc., where the body of knowledge (knowledge base), for example, an area such as an ontology, is accessed. It can be organized in various configurations, such as a structured repository of unique information, or unstructured data related to an area, or a collection of natural language documents about an area.

Content users enter requests into a cognitive system that implements a request processing pipeline. The request processing pipeline then responds to the request using the content in the corpus of data by evaluating the document, the section of the document, the portion of the data in the corpus, and so on. When a process evaluates a given section of a document with respect to semantic content, the process queries these documents from the request processing pipeline, using various rules, for example, querying the request processing pipeline as eligible requests. Send, then the request processing pipeline interprets the request and a response containing one or more responses is given to the request. Semantic content is content based on the relationships between signifiers such as words, phrases, signs and symbols, what they represent, their denotation or denotation. Is. In other words, semantic content is content that interprets expressions using natural language processing or the like.

As described in more detail below, the request processing pipeline receives the request, parses the request, extracts the main features of the request, formulates the query using the extracted features, and data these queries. Applies to the corpus of. Based on applying a query to a corpus of data, the request processing pipeline responds to a request by looking across the corpus of data for a portion of the corpus of data that has some potential, including a useful response to the request. Generate a set of. The request processing pipeline then performs an in-depth analysis of the language of the request and the language used in each of the corpus pieces of data found during the application of queries using various inference algorithms. The inference algorithms applied can be in the hundreds or even thousands, each performing different analyzes such as comparison, natural language analysis, lexical analysis, etc. to generate scores. For example, some inference algorithms can look for matches between terms and synonyms in the requested language and the found portion of the corpus of data. While other inference algorithms examine temporal or spatial features within a language, others can evaluate the source of a corpus portion of the data and evaluate its accuracy.

As mentioned above, the request processing pipeline mechanism accesses information from a data or corpus of information (also called a corpus of content), analyzes it, and then generates response results based on the analysis of this data. It works by doing. Accessing information from a corpus of data generally involves database queries that answer requests for what is in a set of structured records, and unstructured data (text, markup languages, etc.). Includes a search that conveys a set of document links in response to a query on the set of. Traditional request processing systems generate answers based on a corpus of data and input requests, validate the answers to a set of requests for the corpus of data, and use the corpus of data to correct errors in digital text. , Select a response to a request from a pool of potential answers, ie candidate answers.

FIG. 1 illustrates a schematic of one exemplary embodiment of a cognitive system 100 that implements a request processing pipeline 108, which in some embodiments can be a request processing pipeline in a computer network 102. show. For the purposes of this description, the request processing pipeline 108 is assumed to operate for structured and / or unstructured requests in the form of input questions. An example of a question processing operation that can be used in connection with the principles described herein is described in US Patent Application Publication No. 2011/0125734, which is hereby incorporated by reference in its entirety. Incorporated into the book. The cognitive system 100 includes one or more computing devices 104A-D (one or more processors and one or more memories, as well as buses, storage devices, and communications) connected to the computer network 102. Implemented on (including potentially any other computing device element commonly known in the art), including interfaces and the like. For illustration purposes only, FIG. 1 shows a cognitive system 100 implemented only on computing devices 104A, but as described above, the cognitive system 100 includes a plurality of computing devices 104A-D. Can be distributed across multiple computing devices, such as. The network 102 includes a plurality of computing devices 104A-D capable of operating as server computing devices and one or more wired data communication links and / or wireless data communication links to each other and others. Includes computing devices 110-112 that may operate as client computing devices that communicate with devices or components, where each communication link is of a wired, router, switch, transmitter, receiver, etc. Includes one or more. In some exemplary embodiments, the cognitive system 100 and network 102 enable request processing capabilities for users of one or more cognitive systems via their respective computing devices 110-112. .. In other embodiments, the cognitive system 100 and the network 102 are, but are not limited to, depending on the desired implementation, such as, for example, retrieval of cognitive information, user training / instruction, cognitive evaluation of data, and the like. Other types of cognitive behavior can be provided, including request processing and cognitive response generation that can take many different forms. Other embodiments of the cognitive system 100 can be used with components, systems, subsystems, or devices or combinations thereof other than those set forth herein.

The cognitive system 100 is configured to implement a request processing pipeline 108 that receives input from various sources. Requests can be presented in the form of natural language questions, natural language requests for information, natural language requests for performing cognitive actions, and so on. For example, the cognitive system 100 receives input from a network 102, a corpus (s) 106 of electronic documents, a user of the cognitive system, and / or other data and other possible sources of input. In one embodiment, some or all of the inputs to the cognitive system 100 are routed through the network 102. Various computing devices 104A-D on network 102 include access points for content creators and users of cognitive systems. Some of the computing devices 104A-D include devices for databases that store a corpus (s) 106 of data (shown as separate entities in FIG. 1 for illustration purposes only). A portion of a corpus (s) 106 of data is not explicitly shown in one or more databases on one or more other network attached devices, or in FIG. It can also be placed on other computing devices. In various embodiments, the network 102 includes a local network connection and a remote connection, and the cognitive system 100 can operate in an environment of any size, including local and global, such as the Internet.

In one embodiment, the content creator creates content within a document of the data corpus (s) 106 for use as part of the data corpus in the cognitive system 100. The document can include any file, text, article, or source of data used in the cognitive system 100. A user of the cognitive system accesses the cognitive system 100 via a network connection or an internet connection to the network 102, and questions / processed based on the content in the corpus (s) 106 of the data. The request is entered into the cognitive system 100. In one embodiment, the question / request is made using natural language. The cognitive system 100 analyzes and interprets the question / request via the request processing pipeline 108, and responds to the presented question including one or more answers, the response to the request, the result of processing the request, and the like. Is provided to a user of the cognitive system, for example, the user 110 of the cognitive system. In some embodiments, the cognitive system 100 provides the user with a response in the form of a ranked list of candidate responses / responses, while in other exemplary embodiments, the cognitive system 100 provides a response. Provide a single final answer / response, or a combination of a final answer / response and a ranked list of other candidate answers / responses.

The cognitive system 100 implements request processing 108, which includes multiple steps for processing input questions / requests, based on information obtained from a corpus (s) 106 of data. Request processing 108 generates an answer / response to an input question or request based on the processing of the input question / request and the corpus (s) 106 of the data. Request processing 108 is described in more detail below with respect to FIG.

In some exemplary embodiments, the cognitive system 100 is extended by the mechanisms of the exemplary embodiments described below, IBM Watson, available from International Business Machines Corporation in Armonk, NY. Trademark) Can be a cognitive system. As outlined earlier, the IBM Watson ™ cognitive system pipeline receives an input question or request, then parses it to extract the main features of the question / request, and then it It is used to formulate a query that applies to a corpus (s) 106 of data. Includes informative responses to input questions / responses (hereinafter referred to as input questions) across the data corpus (s) 106, based on the application of the query to the data corpus (s) 106. By searching for a portion of the corpus (singular or plural) 106 (hereinafter simply referred to as the corpus 106) of any possible data, a set of hypotheses, i.e. candidate answers / responses to input questions / requests, is generated. The request processing 108 of the IBM Watson ™ cognitive system is then the language used in each of the input question language and the portion of the corpus 106 found during the application of queries using various inference algorithms. Perform a deep analysis of.

Scores from various inference algorithms then summarize the reliability of request processing 108 of IBM Watson ™ cognitive system 100 in this example with respect to evidence that the question infers potential candidate answers. Weights are given to the statistical model to be used. This process is repeated for each of the candidate answers, generating a ranked list of candidate answers and then presenting it to the user who submitted the input question, for example the user of client computing device 110. It can, or the final answer is selected from it and presented to the user. Further information about the request processing 108 of the IBM Watson ™ Cognitive System 100 can be obtained, for example, from the IBM Corporation website, IBM Redbooks, and the like. For example, information about the pipeline of IBM Watson ™ cognitive systems can be found in Yuan et al., "Watson and Healthcare," IBM developerWorks, 2011, and Tob High, "The Era of It can be found in "IBM and Watson and How it Works", IBM Redbooks, 2012.

As mentioned above, the input from the client device to the cognitive system 100 can be presented in the form of a natural language question, but exemplary embodiments are not limited thereto. Rather, in practice, input questions are, but are not limited to, natural language analysis and analysis mechanisms of cognitive systems such as IBM Watson ™ for determining the basis for performing cognitive analysis. It can be analyzed and analyzed using structured and / or unstructured input analysis, including, and formatted or structured as any suitable type of requirement that can provide the results of cognitive analysis. For healthcare-based cognitive systems, this analysis processes patient electronic medical records, medical guidance documents from one or more corpora, etc. to provide the results of a healthcare-focused cognitive system. Can include that.

In the context of the present invention, the cognitive system 100 is a part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as seen by a medical professional. A cognitive function for implementing an augmented reality display can be provided via a head-mounted display (HMD) system indicating. For example, depending on the particular practice, healthcare-based behaviors include patient diagnosis, medical practice management systems, monitoring of individual patient care plans, identification of a patient's medical condition and / or treatment, and medical specialty. For a variety of purposes, such as implementing an augmented reality display that shows the part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient in the eyes of the house. Evaluation of the patient's electronic medical record (EMR) can be included. Thus, the cognitive system 100 operates in the medical or healthcare type domain and has done so via the input of the request processing pipeline 108 such as structured or unstructured requests, natural language input questions, etc. It can be a healthcare cognitive system 100 capable of processing requests for healthcare operations. In one exemplary embodiment, the cognitive system 100 analyzes the patient's EMR and provides an indication of the patient's medical condition and / or the treatment the patient is receiving in the cognitive healthcare system 100. be. Using the identified medical condition and / or treatment, the cognitive healthcare system 100 isolates a particular part of the patient's body associated with the particular medical condition and / or treatment and medical treatment. An augmented reality display is performed that shows the part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as seen by the expert.

As shown in FIG. 1, the cognitive system 100 superimposes on the actual view of the patient as seen by the medical professional, the patient's body corresponding to the patient's medical condition and / or treatment. Implemented with specialized hardware, software running on hardware, or any combination of specialized hardware and software running on hardware to implement a cognitive healthcare system 120 that indicates the location of It is further extended according to the mechanism of the exemplary embodiment to include the logic. As shown in FIG. 1, the cognitive healthcare system 120 includes an image capture and analysis engine 122, a voice capture and analysis engine 124, a correlation engine 126, a medical condition / treatment analysis engine 128, and a display. Includes engine 130.

In the cognitive system 100, more specifically, the cognitive healthcare system 120, the image capture and analysis engine 122 is a patient cared for by a medical professional and / or one or more of the medical professional. Capture real-time images. That is, the image capture and analysis engine 122 uses one or more cameras associated with the HMD system, such as a camera facing the patient, a retinal camera aimed at the eyes of a medical professional, and the like. Alternatively, a plurality of images can be captured. The image capture and analysis engine 122 utilizes one or more real-time images for a number of different aspects of the exemplary embodiment. In one embodiment, the image capture and analysis engine 122 utilizes an image of the medical professional's eyes to identify the medical professional who is caring for the patient. In another embodiment, the image capture and analysis engine 122 captures an image of the patient's face that can be used to identify which patient is being viewed by a medical professional, and / or the patient's mood. .. That is, the image capture and analysis engine 122 can capture an image of the patient's face that can be used to identify the patient being cared for through face recognition. In addition, the image capture and analysis engine 122 can be used to identify a patient's mood by identifying whether the patient is crying or whether facial expressions indicate fear, happiness, anxiety, anxiety, etc. An image of the patient's face can be captured.

Also, in the cognitive healthcare system 120, the audio capture and analysis engine 124 captures one or more audible utterances by the patient and / or the medical professional caring for the patient. The voice capture and analysis engine 124 utilizes one or more audible utterances in a number of different aspects of the exemplary embodiment. In one embodiment, the voice capture and analysis engine 124 captures audible speech by a medical professional that can be used to identify the medical professional who is caring for the patient. In another embodiment, the voice capture and analysis engine 124 can be used by the medical professional in presenting further information to the medical professional via the augmented reality display of the HMD system by the medical professional. Capture the audible instructions given by. In yet another embodiment, the voice capture and analysis engine 124 captures the patient's audible utterances to identify the patient's mood. That is, the voice capture and analysis engine 124 can capture the sound of a patient crying, anxiety in the patient's voice, laughter, etc. in order to identify one or more of concerns, worries, fears, happiness, etc. can.

Using one or more images captured by the image capture and analysis engine 122, and one or more audible utterances captured by the voice capture and analysis engine 124, the correlation engine 126 provides the necessary information to the medical professional. Perform a number of correlations to give to. One exemplary correlation is that the correlation engine 126 uses face recognition to compare one or more images with a medical professional image stored within the medical professional data corpus (s) 140. When doing so, identify the medical professional who is caring for the patient. Another exemplary correlation is that the correlation engine 126 uses speech recognition to compare one or more speech utterances to a medical professional voice pattern stored in a corpus (s) 140 of medical professional data. When doing so, identify the medical professional who is caring for the patient. Similarly, the correlation engine 126 performs correlation to identify patients being cared for by a healthcare professional. To identify a patient, the correlation engine 126 utilizes facial recognition to store one or more images in a corpus (s) 142 of the data, an electronic medical record (EMR) for the patient. Compare with the images in the set. In addition to or as a completely different form of identification, the correlation engine 126 utilizes speech recognition to store one or more audible utterances in a corpus (s) 142 of the data electronically for the patient. Compare with the patient's voice pattern in a set of medical records (EMR).

In addition to identifying the medical professional and the patient, the correlation engine 126 also identifies one or more body parts of the patient that the medical professional is looking at. The identification of the particular body part being viewed is especially important when overlaying the patient's medical condition on the actual view of the patient as seen by the medical professional through an augmented reality display. That is, based on the parts of the patient's body as seen by the medical professional, the correlation engine 126 indicates those bodies in the patient's electronic medical record (EMR) indicating the medical condition and / or treatment associated with the patient. Identify specific body parts to further correlate with parts. Thus, identifying the particular body part seen by the correlation engine 126, the medical condition / treatment analysis engine 128 records the patient's electronic medical record (EMR) stored within a corpus (s) 142 of the data. Analyze to identify the medical condition and / or treatment associated with the patient. Utilizing the patient's identified medical condition and / or treatment, the correlation engine 126 is associated with the specific medical condition and / or treatment when a medical professional looks at that part of the patient's body. Identify the body part of the patient. For example, if the patient has a medical condition of appendicitis, the medical professional looks at the patient's body through the augmented reality display, and when the patient's lower abdomen comes into view, the correlation engine 126 provides a view of the patient's lower abdomen. Correlates with the patient's medical condition and provides an appendicitis overlay shown as an overlay on the lower abdomen of the patient's body. The correlation engine 126 provides this overlay to the display engine 130, which presents the overlay to the medical professional via an augmented reality display in the HMD system.

Of particular note to the exemplary embodiments is that the overlay provided by the correlation engine 126 can vary depending on the view required by the medical professional. That is, if the medical professional is a nurse, the correlation engine 126 can provide a basic organ model that shows common organs. However, if the medical professional is a physician, the correlation engine 126 can provide an actual x-ray overlay of the organ. In addition, if the medical professional is a surgeon, the correlation engine 126 can provide a computed tomography (CAT) scan (CT) overlay or a magnetic resonance imaging (MRI) scan overlay for the entire site. In addition to providing basic organ models, x-rays, CT scans, MRI scans, etc., Correlation Engine 126 includes one or more anatomical models, overlapping organ systems, x-rays from previous medical conditions / treatments. , CT scan, MRI scan, etc., the position of surgery or compression, etc. can also be provided. Anything that should be provided by monitoring eye movements, facial expressions, head movements, audible speech, etc. from medical professionals through the image capture and analysis engine 122 and / or the audio capture and analysis engine 124. The display of additional information can be identified.

Additionally, the overlay provided by the correlation engine 126 can be based on the time the healthcare professional needs to spend on the patient. For example, based on the medical professional's schedule accessible by the correlation engine 126 through the medical professional data corpus 140, the medical professional spends time on the patient as it may occur during the morning round. It may only take a few minutes. Therefore, the correlation engine 126 can provide a basic organ model showing general organs. However, if the schedule indicates that the medical professional is consulting the surgery prior to the surgery, the correlation engine 126 may be an x-ray overlay of the entire site, a computed tomography (CAT) scan (CT) overlay, or a magnetic resonance imaging. Resonance imaging (MRI) scan overlays can be provided. In addition, the correlation engine 126 can provide any additional overlay if the medical professional requires it, regardless of whether the schedule allows more time. This means that the medical professional's schedule shows that the medical professional spends only a few minutes on the patient as it can occur during the morning round, and can provide a basic organ model that first shows the general organs. Even if medical professionals request additional information, the correlation engine 126 may include x-ray overlays, CT scan overlays, MRI scan overlays, one or more anatomical models, overlapping organ systems, previously. X-rays from medical conditions / treatments, CT scans, MRI scans, etc., surgical or compression locations, etc. can be provided. Anything that should be provided by monitoring eye movements, facial expressions, head movements, audible speech, etc. from medical professionals through the image capture and analysis engine 122 and / or the audio capture and analysis engine 124. The display of additional information can be identified.

In addition, the correlation engine 126 can provide overlays based on the particular discipline of the medical professional caring for the patient. That is, if the medical professional's identification is an anesthesiologist or anesthesiologist, the correlation engine 126 can provide an organ overlay that is not even associated with a particular medical condition. That is, the anesthesiologist or anesthesiologist may be more interested in the patient's lungs, airways, nasal passages, etc. Conversely, if the medical professional's identification is a surgeon, the correlation engine 126 can provide an organ overlay that is directly related to a particular medical condition. In addition, if the medical professional's identification is a nurse giving the drug to the patient, the correlation engine 126 may provide an organ overlay of where the drug is administered, such as a particular arm, arm site, etc. can.

In addition to providing overlays that identify graphical images that represent medical conditions and highlight affected or parts of the body that need further investigation, Correlation Engine 126 provides test results, treatment options, medical. Textual information representing codes, the latest medical research studies, organs available for transplantation, etc. can also be provided. By monitoring eye movements, facial expressions, head movements, audible utterances, etc. from medical professionals, the display of any additional information to be provided can be identified. That is, based on the input provided by the medical professional, the correlation engine can identify the requested textual information and then the display engine 130 can display it on the augmented reality display of the HMD system. In addition, based on the mood identified using one or more images of the patient and / or one or more audible utterances, the correlation engine 126 should allow medical professionals to present information to the patient. You can give instructions. That is, if the patient is identified as calm, the correlation engine 126 can instruct the medical professional to speak in a relaxed tone. However, if the patient is identified as nervous, the correlation engine 126 can instruct the medical professional to use a more reassuring tone.

In addition, once the medical professional selects the treatment that the patient should follow, which can be identified by monitoring eye movements, facial expressions, head movements, audible speech, etc. from the medical professional, When directed or otherwise identified, the correlation engine 126, via one or more electronic notification means, determines the schedule of surgery, equipment provided during surgery, requests for consultation, medications to be administered, and the like. Treatments that can be included can be notified to one or more other healthcare professionals in real time, near real time, or non-real time. Treatment instructions are given by monitoring eye movements, facial expressions, head movements, audible speech, etc. from medical professionals via image capture and analysis engine 122 and / or voice capture and analysis engine 124. , Can be identified.

As mentioned above, the mechanisms of the exemplary embodiments are rooted in the field of computer technology and are implemented using the logic that exists within such computing or data processing systems. These computing or data processing systems are specially configured through either hardware, software, or a combination of hardware and software to perform the various operations described above. Therefore, FIG. 2 is given as an example of one type of data processing system capable of implementing aspects of the invention. Similarly, many other types of data processing systems can be configured to specifically implement the mechanisms of exemplary embodiments.

FIG. 2 is a block diagram of an exemplary data processing system in which aspects of the exemplary embodiment are implemented. The data processing system 200 is an example of a computer such as the server 104 or client 110 of FIG. 1, in which a computer-enabled code or instruction that implements the process of an exemplary embodiment of the invention is arranged. In one exemplary embodiment, FIG. 2 shows a server 104 that implements a cognitive system 100 and a request processing pipeline 108 that are extended to include additional mechanisms of the exemplary embodiments described below. Represents a server computing device such as.

In the example shown, the data processing system 200 includes a north bridge and a memory controller hub (NB / MCH) 202 and a south bridge and an input / output (I / O) controller hub (SB / ICH) 204. To use. The processing unit 206, the main memory 208, and the graphics processor 210 are connected to the NB / MCH 202. The graphics processor 210 is connected to the NB / MCH 202 through an accelerated graphics port (AGP).

In the example shown, the local area network (LAN) adapter 212 is connected to the SB / ICH204. Voice adapter 216, keyboard and mouse adapter 220, modem 222, read-only memory (ROM) 224, hard disk drive (HDD) 226, CD-ROM drive 230, universal serial bus (USB) port and other communication ports The 232 and the PCI / PCIe device 234 are connected to the SB / ICH 204 through the bus 238 and the bus 240. PCI / PCIe devices can include, for example, Ethernet adapters, add-in cards, and PC cards for notebook computers. PCI uses a card bus controller, but does not use PCIe. The ROM 224 can be, for example, a flash basic input / output system (BIOS).

The HDD 226 and the CD-ROM drive 230 are connected to the SB / ICH 204 via the bus 240. The HDD 226 and the CD-ROM drive 230 can use, for example, an integrated driven electronics (IDE) or serial advanced technology attributes (SATA) interface. The super I / O (SIO) device 236 is connected to the SB / ICH204.

The operating system runs on processing unit 206. The operating system works together to provide control of the various components within the data processing system 200 of FIG. As a client, the operating system is a commercially available operating system such as Microsoft® Windows 8®. Object-oriented programming systems, such as the Java ™ programming system, can run with the operating system and make calls to the operating system from Java ™ programs or applications running on the data processing system 200. offer.

As a server, the data processing system 200 may, for example, run an Advanced Interactive Executive (AIX®) operating system or an IBM® eServer® System p® operating system. ) Can be a computer system. The data processing system 200 can be a symmetric multiprocessor (SMP) system including a plurality of processors in the processing unit 206. Alternatively, a single processor system can be used.

Instructions to the operating system, object-oriented programming system and application or program are placed on a storage device such as HDD 226, loaded into main memory 208 and executed by processing unit 206. The process for an exemplary embodiment of the invention is performed by processing unit 206 with computer-enabled program code, which is stored in memory such as, for example, main memory 208, ROM 224. Alternatively, it is placed in one or more peripheral devices 226 and 230.

A bus system, such as bus 238 or bus 240 shown in FIG. 2, consists of one or more buses. Of course, bus systems can be implemented using any type of communication fabric or architecture that provides data transfer between different components or devices attached to the fabric or architecture. A communication unit, such as the modem 222 or network adapter 212 of FIG. 2, includes one or more devices used to send and receive data. The memory can be, for example, a cache as found in the main memory 208, ROM 224, or NB / MCH 202 of FIG.

Those skilled in the art will recognize that the hardware shown in FIGS. 1 and 2 can vary from implementation to implementation. Peripheral devices such as other internal hardware, flash memory, equivalent non-volatile memory, or optical disk drives may be used in addition to or in place of the hardware shown in FIGS. 1 and 2. .. It is also possible to apply the process of the exemplary embodiment to a multiprocessor data processing system other than the SMP system described above without departing from the scope of the present invention.

In addition, the data processing system 200 includes a number of different types including client computing devices, server computing devices, tablet computers, laptop computers, telephones or other communication devices, personal digital assistants (PDAs), and the like. It can take any form of data processing system. In some demonstrable examples, the data processing system 200 is configured to include flash memory to provide, for example, non-volatile memory for storing operating system files and / or user-generated data. It can be a portable computing device. In essence, the data processing system 200 can be any well-known or later developed data processing system without architectural limitations.

FIG. 3 is an exemplary diagram showing the interaction of elements of a cognitive system according to one exemplary embodiment. The exemplary diagram of FIG. 3 shows an implementation of the cognitive system 300, which is superimposed on the actual view of the patient as seen by a medical professional, the patient's medical condition. An augmented reality display is configured to be performed via a head-mounted display (HMD) system, such as through a wearable headset, glasses, etc., which indicates the part of the patient's body that corresponds to treatment or both. It can be a cognitive system such as the cognitive system 100 described in FIG. However, it is understood that this is merely an exemplary practice and that other healthcare operations can be performed in other embodiments of the Healthcare Cognitive System 300 without departing from the spirit and scope of the present invention. I want to be.

In addition, FIG. 3 shows the patient 302 and the medical professional 306 as human figures, but the interactions with and between these entities are, but are not limited to, computing devices. It can be performed using a number of devices, including medical devices and / or the like. For example, the interactions 304, 314, 316, and 330 between patient 302 and user 306 can be performed verbally, for example by a doctor asking the patient, and cognitive methods such as patient attribute 318. In order to collect information that can be input to the system 300, it may be necessary to use one or more medical devices, surveillance devices, etc., such as a head-mounted display (HMD) system of an exemplary embodiment. The interaction between the user 306 and the cognitive system 300 will be electronic via a user computing device (not shown) such as the client computing device 110 or 112 of FIG. In an exemplary embodiment, it is an HMD system that communicates with the cognitive system 300 via one or more data communication links and potentially one or more data networks.

As shown in FIG. 3, according to one exemplary embodiment, patient 302 has a medical malady or condition symptom 304, such as a medical professional, health care practitioner, technician, etc. Presented to user 306 such as. User 306 can interact with patient 302 through the exchange of question 314 and response 316, where user 306 gathers further information about patient 302, symptom 304, and patient 302's medical illness or condition. .. The question / response is actually one or more of a wearable health and activity monitoring device associated with patient 302, such as an HMD system, blood pressure monitor, thermometer, FitBit ™, wearable heart monitor, or patient 302. It should also be appreciated that a variety of medical devices, such as any other medical device capable of monitoring the medical features of the patient, may be used to represent the user 306 gathering information from the patient 302. In some cases, such medical devices should be medical devices commonly used in hospitals or medical centers to monitor the vital signs and medical condition of patients in hospital beds for observation or treatment. Can be done. According to an exemplary embodiment, the medical device is an HMD system that collects both images and audible utterances from both patient 302 and user 306.

In response, the user 306 may use an HMD system or the like configured to allow the user to submit a request to the cognitive system 300 in a format that the cognitive system 300 can analyze and process. Request 308 is submitted to the cognitive system 300. Request 308 may include, or may be accompanied by, information identifying attributes 318 of patient 302 and user 306. That is, the HMD system described above is one or more real-time images of one or more real-time images of patient 302 and / or user 306 caring for the patient, and one or more by user 306 caring for patient 302 and / or patient 302. Multiple audible utterances can be captured. Thus, patient attribute 318 can be obtained, for example, from an image of the patient's face in audible speech from patient 302 from which patient EMR322 can be retrieved for patient 302, patient 302, symptoms 304, and response 316 to question 314. May include information obtained from medical devices used to monitor or collect demographic information about the relevant information of the patient, or data about the condition of the patient 302, including the medical condition associated with the patient 302. can. Any information about patient 302 that may be relevant to the cognitive assessment of patient 302 by the cognitive system 300 may be included in request 308 and / or patient attribute 318.

The cognitive system 300 is specially configured to perform the implementation of a particular healthcare-oriented cognitive action. In the example shown, this cognitive precision cohort operation is associated with Symptom 304 reported by User 306 through the process of Question 314 and Response 316 and / or data collection of the medical device. Within the augmented reality display of the HMD system, superimposed on the actual view of the patient as seen by the user 306, to help care for the patient 302 based on other information gathered about the patient 302. To indicate the part of the patient's body that corresponds to the patient's medical condition and / or treatment. The cognitive system 300 operates on request 308 and patient attribute 318 using information gathered from patient EMR322 associated with patient 302 to identify the medical condition of patient 302.

For example, based on request 308 and patient attribute 318, the cognitive system 300 analyzes request 308 and patient attribute 318 and presents not only which patient to treat, but also patient 302 through the display of the HMD system. It may also act on the request to determine a particular medical condition that has an overlay associated with the medical condition available to do so. Therefore, the cognitive system 300 acts on the request to analyze the request 308 and the patient attribute 318 and determine what is required and the criteria by which the request is generated to be identified by the patient attribute 318. And perform various actions to generate a query sent to patient EMR322 to retrieve the data, generate an association display associated with the data, and provide support for evidence found within patient EMR322. be able to. In the example shown, patient EMR322 is a patient information repository that collects patient data from various sources such as hospitals, laboratories, doctor's offices, health insurance companies, pharmacies, and the like. Patient EMR322 is an individual patient, such as patient 302, in a manner that allows information to be retrieved and processed by the cognitive system 300 (structured, unstructured, or a mixture of structured and unstructured). Stores various information about. This patient information includes various demographic information about the patient, personal contact information about the patient, employment information, health insurance information, experiment reports, outpatient doctor reports, medical records, history information about previous diagnoses, symptoms, etc. It can include treatment, medication information, etc. Based on the identifier of patient 302, this patient's repository can be retrieved by the cognitive system 300 and retrieved / processed to provide treatment routes 328 followed by a cohort of similar patients.

According to an exemplary embodiment herein, the cognitive system 300 is extended to include a cognitive healthcare system 340. The cognitive healthcare system 340 operates in a manner similar to that already described above for the corresponding elements 122-130 of FIG. 1, image capture and analysis engine 342, audio capture and analysis engine 344, correlation engine 346, It includes a medical condition / treatment analysis engine 348 and a display engine 350. That is, the image capture and analysis engine 342 captures one or more real-time images of patient 302 and / or user 306 identified from patient attribute 318. The image capture and analysis engine 342 utilizes one or more real-time images in a number of different aspects of the exemplary embodiment. In one embodiment, the image capture and analysis engine 342 utilizes the eye image of user 306 to identify a particular medical professional caring for patient 302. In another embodiment, the image capture and analysis engine 342 captures an image of the patient 302's face that can be used to identify which patient the user 306 is looking at, and / or the mood of the patient 302. That is, the image capture and analysis engine 342 can capture an image of the face of the patient 302 that can be used to identify the patient being cared for through face recognition. In addition, the image capture and analysis engine 342 can be used to identify the patient's mood by identifying whether the patient 302 is crying or the facial expression indicates fear, happiness, anxiety, anxiety, etc. 302 images can be captured.

The voice capture and analysis engine 344 captures one or more audible utterances by patient 302 and / or user 306 caring for patient 302. The voice capture and analysis engine 344 utilizes one or more audible utterances in a number of different aspects of the exemplary embodiment. In one embodiment, the voice capture and analysis engine 344 can capture audible utterances by user 306 and use them to identify a medical professional caring for patient 302. In another embodiment, the voice capture and analysis engine 344 can be used by the cognitive healthcare system 340 in presenting further information to the user 306 via the augmented reality display of the HMD system, by the user 306. Capture the audible instructions provided. In yet another embodiment, the voice capture and analysis engine 344 captures the audible utterances of patient 302 to identify the mood of patient 302. That is, the voice capture and analysis engine 344 captures the patient's crying sound, anxiety in the patient's voice, laughter, etc., which is used to identify one or more of concerns, worries, fears, happiness, etc. be able to.

Using one or more images and audio captured by the image capture and analysis engine 342 and one or more audible utterances captured by the audio capture and analysis engine 344, the correlation engine 346 provides the medical professional with the necessary information. Perform a number of correlations to give. One exemplary correlation is to identify a user 306 caring for a patient, where the correlation engine 346 utilizes facial recognition to produce one or more images into a corpus of medical professionals. And other source data 324 for comparison with images of medical professionals stored. Another exemplary correlation is identifying a user 306 caring for a patient, where the correlation engine 346 utilizes speech recognition to produce one or more audible utterances of a medical professional. Compare with medical professional voice patterns stored in the corpus and other source data 324. Similarly, the correlation engine 346 performs the correlation to identify the patient 302 being cared for by user 306. To identify patient 302, the correlation engine 346 uses facial recognition to compare one or more images with images in a set of electronic medical records (EMR) for the patient stored within patient EMR322. do. In addition to this or as a completely different form of identification, the correlation engine 346 uses speech recognition to make one or more audible utterances, a set of electronic medical records (EMR) for a patient stored within patient EMR322. Compare with the patient's voice pattern in.

In addition to identifying user 306 and patient 302, the correlation engine 346 also identifies one or more body parts of patient 302 that user 306 is viewing via the HMD system. The identification of the particular body part being viewed is of particular importance when overlaying the medical condition of the patient 302 on the actual view of the patient as seen by the user 306 through the augmented reality display of the HMD system. That is, based on the body part of patient 302 seen by user 306, the correlation engine 346 goes to the body part in the data of patient EMR322 indicating the medical condition and / or treatment associated with patient 302. Identify specific body parts for further correlation. Thus, when the correlation engine 346 identifies a particular body part being viewed, the medical condition / treatment analysis engine 348 analyzes the patient 302's electronic medical record (EMR) stored within the patient EMR322 and the patient. Identify the medical condition and / or treatment associated with 302. Utilizing the identified medical condition and / or treatment of patient 302, the correlation engine 346 uses the specific medical condition or treatment when that part of the patient's body is viewed by the user 306 via the HMD system. Identify the part of the patient's body associated with or both. For example, if patient 302 has a medical condition of appendicitis, when user 306 looks at patient 302's body through the augmented reality display of the HMD system and the lower abdomen of patient 302 comes into view, the correlation engine 346 will perform the lower abdomen. The view of is correlated with the medical condition of patient 302 to provide an appendix overlay shown as an overlay on the lower abdomen of patient 302. The correlation engine 346 provides this overlay to the display engine 350, which presents the overlay 328 to the medical professional via the augmented reality display of the HMD system.

Of particular note in the exemplary embodiment is that the overlay provided by the correlation engine 346 can vary depending on the view required by user 306. That is, if the user 306 is a nurse, the correlation engine 346 can provide a basic organ model showing general organs. However, if user 306 is a doctor, the correlation engine 346 can provide an actual X-ray overlay of the organ. In addition, if user 306 is a surgeon, the correlation engine 346 can provide a computed tomography (CT) scan (CT) overlay or a magnetic resonance imaging (MRI) scan overlay for the entire site. In addition to providing basic organ models, x-rays, CT scans, MRI scans, etc., the correlation engine 346 has one or more anatomical models, overlapping organ systems, x-rays from previous medical conditions / treatments. , CT scan, MRI scan, etc., the position of surgery or compression, etc. can also be provided. Anything that should be provided by monitoring eye movements, facial expressions, head movements, audible speech, etc. from medical professionals through the image capture and analysis engine 342 and / or the audio capture and analysis engine 344. The display of additional information can be identified.

Additionally, the overlay provided by the correlation engine 346 can be based on the time the user 306 needs to spend on the patient 302. For example, based on the schedule of user 306 accessible by the correlation engine 346 through the corpus of medical professionals and other source data 324, user 306 spends a number of hours on patient 302 to occur during the morning rounds. Sometimes there is only a minute. Therefore, the correlation engine 346 can provide a basic organ model showing general organs. However, if the schedule indicates that user 306 is consulting for surgery prior to surgery, the correlation engine 346 will have an x-ray overlay of the entire site, a computed tomography (CAT) scan (CT) overlay, or magnetic resonance imaging. Imaging (MRI) scans can be provided. In addition, the correlation engine 346 can provide any additional overlay if the medical professional requires it, regardless of whether the schedule allows more time. This means that the medical professional's schedule shows that the medical professional spends only a few minutes on the patient as it can occur during the morning round, and can provide a basic organ model that first shows the general organs. Even if medical professionals request additional information, the correlation engine 346 will include x-ray overlays, CT scan overlays, MRI scan overlays, one or more anatomical models, overlapping organ systems, previously. X-rays from medical conditions / treatments, CT scans, MRI scans, etc., surgical or compression locations, etc. can be provided. Anything that should be provided by monitoring eye movements, facial expressions, head movements, audible speech, etc. from medical professionals through the image capture and analysis engine 342 and / or the audio capture and analysis engine 344. The display of additional information can be identified.

In addition, the correlation engine 346 can provide overlays based on the particular discipline of user 306 caring for patient 302. That is, if the identification of user 306 is an anesthesiologist or anesthesiologist, the correlation engine 346 can provide an organ overlay that is not even associated with a particular medical condition. That is, the anesthesiologist or anesthesiologist may be more interested in the patient's lungs, airways, nasal passages, etc. Conversely, if the identification of user 306 is a surgeon, the correlation engine 346 can provide an organ overlay that is directly related to the particular medical condition of patient 302. Further, if the identification of user 306 is a nurse giving the drug to patient 302, the correlation engine 346 provides an organ overlay of where the drug is administered, such as a particular arm, arm site, etc. of patient 302. Can be done.

In addition to providing an overlay that identifies a graphical image that represents a medical condition and highlights a part of the body that is affected or needs further examination, the correlation engine 346 provides test results, treatment options, medical. Textual information representing codes, the latest medical research studies, organs available for transplantation, etc. can also be provided. That is, based on the input provided by user 306, the correlation engine identifies the requested textual information and then the display engine 350 displays it on the augmented reality display of the HMD system. Further, based on the mood identified using one or more images of patient 302 and / or one or more audible utterances, the correlation engine 346 presents information to patient 302 by user 306. Can give instructions on what to do. That is, if the correlation engine 346 identifies the mood of the patient 302 as calm, the correlation engine 346 can instruct the user 306 to speak in a relaxed tone, and then the display engine 350 can instruct the user 306 to speak. , Display on the augmented reality display of the HMD system. However, if the correlation engine 346 identifies that the mood of the patient 302 is nervous, the correlation engine 346 instructs the user 306 to use a more reassuring tone and then displays it to the display engine. The 350 can be displayed on the augmented reality display of the HMD system.

In addition, once the medical professional selects the treatment that the patient should follow, which can be identified by monitoring eye movements, facial expressions, head movements, audible speech, etc. from the medical professional, When directed or otherwise identified, the correlation engine 346, via one or more electronic notification means, informs the schedule of surgery, equipment provided during surgery, requests for consultation, medications to be administered, etc. Treatments that can be included can be notified to one or more other healthcare professionals in real time, near real time, or non-real time. Treatment instructions are given by monitoring eye movements, facial expressions, head movements, audible speech, etc. from medical professionals via image capture and analysis engine 342 and / or voice capture and analysis engine 344. , Can be identified.

Thus, an exemplary embodiment is a wearable type that indicates a part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as seen by a medical professional. Provided is a mechanism for implementing an augmented reality display via a head-mounted display (HMD) system, such as through a headset, eyeglasses, or the like. The mechanism of the present invention captures an image of a part of the patient's body as seen by a medical professional. Based on the patient's body parts being viewed, the mechanism identifies the corresponding body parts in the view and indicates those body parts to the patient's medical condition and / or treatment associated with the patient. Correlate with electronic medical record (EMR) data.

The present invention can be a system, method, computer program product, or a combination thereof. The computer program product can include a computer-readable storage medium (s) having computer-readable program instructions on it to cause the processor to perform aspects of the invention.

The computer-readable storage medium can be a tangible device capable of holding and storing instructions used by the instruction executing device. The computer-readable storage medium is, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or an appropriate combination of any of the above. Can be done. A non-exhaustive list of more specific examples of computer-readable storage media includes: Portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, Included are floppy disks, mechanically encoded devices in which instructions are recorded, and the appropriate combination of any of the above. As used herein, a computer-readable storage medium is a radio wave, or other freely propagating electromagnetic wave, or an electromagnetic wave propagating through a waveguide or other transmission medium (eg, an optical pulse through a fiber optic cable). , Or is not interpreted as a temporary signal itself, such as an electrical signal sent through a wire.

The computer-readable program instructions described herein are from computer-readable storage media to their respective computing / processing devices, or networks such as, for example, the Internet, local area networks, wide area networks, or wireless networks or combinations thereof. Can be downloaded to an external computer or external storage device via. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers or combinations thereof. The network adapter card or network interface on each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions to the computer-readable storage medium within each computing / processing device. Store.

Computer-readable program instructions for performing the operations of the present invention include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or Smalltalk, C ++, etc. Source code or object code written in any combination of one or more programming languages, including object-oriented programming languages and traditional procedural programming languages such as the "C" programming language or similar programming languages. can do. Computer-readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, and partly on the user's computer. It may be run, partly on a remote computer, or entirely on a remote computer or server. In the final scenario, the remote computer may be connected to the user's computer through either type of network, including a local area network (LAN) or wide area network (WAN), or a connection to an external computer. It may also be done (eg, through the internet with an internet service provider). In some embodiments, electronic circuits, including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are computers to implement aspects of the invention. By using the state information of the readable program instruction, the computer readable program instruction can be executed to individualize the electronic circuit.

Aspects of the present invention will be described with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present invention. It will be appreciated that each block of the flow chart and / or block diagram, and the combination of blocks in the flow chart and / or block diagram, can be implemented by computer-readable program instructions.

These computer-readable program instructions are given to the processor of a general purpose computer, dedicated computer, or other programmable data processor to build a machine, thereby being executed by the processor of the computer or other programmable data processor. The instructions can be made to create means for performing the specified function / operation within one or more blocks of the flowchart and / or block diagram. These computer program instructions are stored in a computer-readable medium that can instruct a computer, other programmable data processor, or other device or combination thereof to function in a particular manner, thereby. The instructions stored in the computer-readable medium may also include a product containing instructions that perform the specified function / operation mode in one or more blocks of the flowchart and / or block diagram. ..

A computer program instruction can be loaded onto a computer, other programmable data processor, or other device to perform a series of operating steps on the computer, other programmable data processor, or other device. It produces a computer-implemented process in which instructions executed on a computer or other programmable device perform a specified function / operation in one or more blocks of a flow chart and / or block diagram. It is also possible to provide a process for this.

FIG. 4 shows a part of the patient's body that corresponds to the patient's medical condition and / or treatment, superimposed on the actual view of the patient as perceived by the medical professional, according to an exemplary embodiment. FIG. Upon initiation of operation, the cognitive healthcare system receives one or more real-time images and / or one or more real-time audible utterances captured from the HMD system (step 402). Using one or more real-time images and / or one or more real-time audible utterances, the cognitive healthcare system uses the patient's facial image, the user's facial image, the patient's audible utterance, the user's audible utterance, Identify one or more specific images and / or utterances, such as a patient's body part (step 404). The cognitive healthcare system puts each face image in one or more face images into a set of electronic medical records (EMR) for the patient stored in the patient's EMR corpus. Patient identification can be identified by comparison with the set (step 406). Alternatively or additionally, the cognitive healthcare system puts each audible utterance in one or more audible utterances within a set of electronic medical records (EMR) for the patient stored in the patient's EMR corpus. Patient identification can be identified by comparing with a set of audio recordings stored in (step 408). In addition to identifying patients, the cognitive healthcare system places each facial image in one or more facial images within a set of records about the medical professional stored within the medical professional corpus. The identification of the user can be identified by comparing with the stored set of facial images (step 410). Alternatively or additionally, the cognitive healthcare system stores each audible utterance within one or more audible utterances within a set of records about the medical professional stored within the medical professional corpus. The user's identification can be identified by comparing with the set of facial images (step 412).

Once the user and patient have been identified, the cognitive healthcare system uses the user's identification to identify one or more medical conditions and / or treatments for the patient (step 414). Using the identified medical condition, the cognitive healthcare system scans images of one or more body parts and correlates to the part of the body in which the patient's medical condition and / or treatment is present. Identify one or more images to be performed (step 416). Then, when that part of the body is viewed by the user through the augmented reality display of the HMD system, the cognitive healthcare system presents the user with an overlay highlighting where the medical condition is (step 418). ). For example, a cognitive health care system is a basic showing general organs, actual x-ray overlays of organs, computed tomography (CT) overlays or magnetic resonance imaging (MRI) scan overlays of the entire site, etc. A target organ model can be presented. The overlays used by the cognitive healthcare system can be based on the user's level or discipline, and can also be based on the schedule associated with the user, etc.

In addition to providing an overlay associated with the identified medical condition and / or treatment, the cognitive healthcare system will relate overlays showing the patient's lungs, airways, nasal passages, etc. to the next surgery. Obtaining Other overlays that are not prominent in the medical condition, such as providing to an anesthesiologist or anesthesiologist, but may be important in the treatment of the identified medical condition, or overlays where a particular drug is administered. Can be presented to the nurse caring for the patient (step 420). In addition, the cognitive healthcare system can present textual information representing test results, treatment options, medical codes, and the like (step 422). In addition, the cognitive healthcare system can present information associated with the detected patient's mood so that the user can change tone when talking to the patient (step 424). Despite the presentation of the identified overlay and / or textual information, the cognitive healthcare system sends the overlay and / or textual information to the HMD system for display on the augmented reality display of the HMD system (step). 426). The cognitive healthcare system then determined if the HMD system was turned off (step 428). In step 428, if the HMD system is not turned off, the process returns to step 414 as the overlay and / or textual information may need to be changed later when the user interacts with the patient. In step 428, when the HMD system is turned off, the operation ends.

Flowcharts and block diagrams in the drawings show the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram can represent a part of a module, segment, or code that contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown within a block may occur in a different order than shown in the figure. For example, two blocks shown in succession may actually be executed at substantially the same time, depending on the function involved, or these blocks may sometimes be executed in reverse order. Each block of the block diagram and / or flowchart diagram, and the block combination in the block diagram or flowchart diagram or both, performs a specified function or operation, or performs a combination of dedicated hardware and computer instructions. Also note that it can be implemented by a dedicated hardware-based system.

Thus, an exemplary embodiment is worn to indicate a part of the patient's body that corresponds to the patient's medical condition and / or treatment, which is superimposed on the actual view of the patient as seen by the medical professional. Provided is a mechanism for implementing an augmented reality display via a head-mounted display (HMD) system, such as through a type headset, eyeglasses, or the like. The mechanism of the present invention captures an image of a part of the patient's body as seen by a medical professional. Based on the part of the patient's body being viewed, the mechanism identifies the corresponding body parts in the view and refers to those body parts of the patient as indicating the medical condition and / or treatment associated with the patient. Correlate with electronic medical record (EMR) data. In some cases, facial recognition can be used to identify a particular patient being seen.

Graphical representations superimposed on the patient's body can be medical conditions and specific implementations. That is, the superimposed graphical representation includes a graphical image of the medical condition, highlighting of affected or further examined body parts, textual information of test results, treatment options, medical code, etc. be able to. PMDA also provides access to a medical corpus of annotated data for multiple media views, such as the mood of a given patient, the time of day, the appointment status of a medical professional's schedule, etc. Allows the selection of suitable media in real time. Regarding the schedule of medical professionals, depending on the schedule, if the reservation status is limited, it is possible to display a more concise one (basic organ model), while if the reservation status is long-term, it is detailed. (Simulated surgical technique on the patient's organ) can be displayed.

As mentioned above, exemplary embodiments can take the form of fully hardware embodiments, fully software embodiments, or embodiments that include both hardware and software elements. In one exemplary embodiment, the mechanisms of the exemplary embodiments are implemented in the form of software or program code, including, but not limited to, firmware, resident software, microcode, and the like.

A data processing system suitable for storing and / or executing program code includes at least one processor directly or indirectly coupled to a memory element via a communication bus, such as a system bus. Memory elements include local memory used during the actual execution of program code, mass storage, and at least some programs to reduce the number of times code must be retrieved from mass storage during execution. It can include cache memory that provides temporary storage for code. The memory can be of various types including, but not limited to, ROM, PROM, EPROM, EEPROM, DRAM, SRAM, flash memory, solid state memory and the like.

Input / output or I / O devices (such as, but not limited to, keyboards, displays, pointing devices, etc.) can also be directly coupled to or intervened in a wired or wireless I / O interface. Alternatively, it can be coupled to the system via a controller or both. I / O devices are conventional, such as, but not limited to, communication devices coupled through wired or wireless connections, including, but not limited to, smartphones, tablet computers, touch screen devices, voice recognition devices, and the like. It can take many different forms other than the keyboard, display, pointing device, etc. Any well-known or later developed I / O device is intended to be within the scope of exemplary embodiments.

A network adapter can also be coupled to a system to allow a data processing system to be coupled to another data processing system or remote printer or storage device through an intervening private or public network. Modems, cable modems and Ethernet cards are just a few of the currently available types of network adapters for wired communication. The wireless communication-based network adapter is not limited to these, but a wireless communication-based network adapter including, but is not limited to, an 802.11a / b / g / n wireless communication adapter, a Bluetooth wireless adapter, and the like can also be used. can. Any well-known or later developed network adapter is intended to be within the spirit of the present invention.

The description of the invention has been presented for purposes of illustration and illustration, but these are not intended to be exhaustive or limited to the disclosed form of the invention. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and intent of the embodiments described. Embodiments are intended to best illustrate the principles and practical use of the invention, and one of ordinary skill in the art will understand the invention with respect to various embodiments with various modifications to suit the particular application intended. Was selected and described so that The terms used herein are to best describe the principles of the embodiment, the practical use, or the technical improvements that are superior to the techniques found on the market, or by those skilled in the art. Selected to allow understanding of morphology.

Claims (20)

A method in a data processing system that includes at least one processor and at least one memory containing instructions executed by the at least one processor to cause the at least one processor to implement a cognitive healthcare system. The cognitive healthcare system is
The capture mechanism of the cognitive healthcare system captures real-time images of the patient's body parts viewed by a medical professional via a head-mounted display (HMD) system.
Identifying one or more body parts of the patient in the real-time image by the cognitive healthcare system.
The cognitive healthcare system correlates the one or more identified body parts with the patient's electronic medical record (EMR) showing the medical condition and / or treatment associated with the patient. That and
One or more of the patient's body corresponding to the patient's medical condition and / or treatment that overlays the real-time image of the part of the patient's body with the cognitive healthcare system. Creating an augmented reality display of the part in the HMD system
How to work to do. The patient's electronic medical record (EMR) is correlated with the patient by the capture mechanism that captures an image of the patient's face and the cognitive healthcare system that identifies the patient using face recognition. The method according to claim 1. The patient's electronic medical record (EMR) is correlated with the patient by the capture mechanism that captures audible utterances from the patient and the cognitive healthcare system that identifies the patient using voice recognition. , The method according to claim 1. The augmented reality display includes a basic organ model, current x-rays, current computed tomography (CAT) scans (CT), current magnetic resonance imaging (MRI) scans, one or more anatomical models, overlapping organ systems. The method of claim 1, wherein one or more of a previous x-ray, a previous CT scan, a previous MRI scan, or a position of surgery or compression is displayed. The method of claim 1, wherein the augmented reality display further highlights a part of the patient's body that has suffered or needs further investigation with respect to the medical condition. The method of claim 1, wherein the augmented reality display further displays textual information representing test results, treatment options, medical codes, the latest medical research studies, or organs available for transplantation. The method of claim 1, wherein the augmented reality display further displays the mood the patient is currently showing. The medical professional treating the patient is identified by the capturing mechanism that captures an image of the medical professional's face and the cognitive healthcare system that identifies the medical professional using face recognition. The method according to claim 1. The medical professional treating the patient is identified by the capturing mechanism that captures audible utterances from the medical professional and the cognitive healthcare system that identifies the medical professional using voice recognition. , The method according to claim 1. The method of claim 1, wherein the information displayed in the augmented reality display is based on the schedule of the medical professional. A computer program product that includes a computer-readable storage medium in which a computer-readable program is stored, said computer-readable program, when executed on the computing device, to the computing device.
The capture mechanism captures real-time images of parts of the patient's body as seen by a medical professional via a head-mounted display (HMD) system.
Identifying one or more body parts of the patient in the real-time image and
Correlating the one or more identified body parts with the patient's electronic medical record (EMR) indicating the medical condition and / or treatment associated with the patient.
An augmented reality display of one or more parts of the patient's body corresponding to the patient's medical condition and / or treatment that overlays the real-time image of the part of the patient's body. To generate in and
A computer program product that lets you do. The augmented reality display includes a basic organ model, current x-rays, current computed tomography (CAT) scans (CT), current magnetic resonance imaging (MRI) scans, one or more anatomical models, overlapping organ systems. The computer program product of claim 11, displaying one or more of a previous x-ray, a previous CT scan, a previous MRI scan, or a surgical or compression position. The computer program product of claim 11, wherein the augmented reality display further highlights the part of the patient's body that has suffered or needs further investigation with respect to the medical condition. The computer program product according to claim 11, wherein the augmented reality display further displays textual information representing test results, treatment options, medical codes, the latest medical research studies, or organs available for transplantation. The computer program product of claim 11, wherein the augmented reality display further displays the mood that the patient is currently exhibiting. It ’s a device,
With the processor
The memory coupled to the processor and
When executed by the processor, the memory includes the processor.
The capture mechanism captures real-time images of parts of the patient's body as seen by a medical professional via a head-mounted display (HMD) system.
Identifying one or more body parts of the patient in the real-time image and
Correlating the one or more identified body parts with the patient's electronic medical record (EMR) indicating the medical condition and / or treatment associated with the patient.
An augmented reality display of one or more parts of the patient's body corresponding to the patient's medical condition and / or treatment that overlays the real-time image of the part of the patient's body. To generate in and
Including instructions to do
Device. The augmented reality display includes a basic organ model, current x-rays, current computed tomography (CAT) scans (CT), current magnetic resonance imaging (MRI) scans, one or more anatomical models, overlapping organ systems. 16. The device of claim 16, displaying one or more of a previous x-ray, a previous CT scan, a previous MRI scan, or a position of surgery or compression. 16. The device of claim 16, wherein the augmented reality display further highlights a part of the patient's body that is affected or needs further investigation with respect to the medical condition. The device of claim 16, wherein the augmented reality display further displays textual information representing test results, treatment options, medical codes, the latest medical research studies, or organs available for transplantation. The device of claim 16, wherein the augmented reality display further displays the mood that the patient is currently showing.

Images