WO2023220646A2

WO2023220646A2 - System to provide postoperative care and monitoring using human voice

Info

Publication number: WO2023220646A2
Application number: PCT/US2023/066840
Authority: WO
Inventors: Chandra Jonelagadda
Original assignee: Kaliber Labs Inc.
Priority date: 2022-05-10
Filing date: 2023-05-10
Publication date: 2023-11-16
Also published as: WO2023220646A3

Abstract

A system and method are disclosed for computer aided postoperative care and monitoring. Disclosed systems and methods that may include analyzing a surgical video of a procedure, generating one or more video clips based on the analysis, generating a surgical narrative to accompany the video clips and generating a voiced narration for the surgical narrative. The voiced narration may be synthesized using a neural model trained to sound like a patient's surgeon or other clinician. Other systems and methods may include prompting the patient to perform guided exercises, capturing patient motion while performing the exercises, analyzing patient motion, and altering a doctor based on the analysis. In some examples, the prompting of the patient may be with a synthesized voice trained to sound like the patient's surgeon.

Description

SYSTEM TO PROVIDE POSTOPERATIVE CARE AND MONITORING USING HUMAN VOICE

CLAIM OF PRIORITY

[0001] This patent application claims priority to U.S. provisional patent application no. 63/340,430, titled “SYSTEM TO PROVIDE POSTOPERATIVE CARE AND MONITORING USING HUMAN VOICE”, filed on May 10, 2022, herein incorporated by reference in its entirety.

INCORPORATION BY REFERENCE

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

FIELD

[0003] The present embodiments relate generally to surgery and more specifically to providing postoperative care to patients using a human voice.

BACKGROUND

[0004] Knee and shoulder concerns were two of the top fifteen reasons for patients to seek ambulatory care during physician office visits in 2015 (28.9 million estimated visits combined). An estimated 984,607 knee arthroscopic procedures and 529,689 rotator cuff repairs and shoulder arthroscopic procedures (over 1.5 million surgical procedures) were performed on an ambulatory basis in 2006.

[0005] Providing consistent and effective postoperative care for these and other surgeries can be difficult. Patients may receive postoperative instructions in an unengaging manner. The surgeons’ staff may provide a set of printouts which are rather generic in nature. Little or no personalization may be included in the postoperative care instructions that generally pertain to the patient’s surgery.

[0006] Patients’ compliance and adherence with postoperative instructions is typically not monitored and the surgeon may only receive periodic assessment from the patient’s physical therapist or when the patient return for their follow-up visits. In the meantime, the patients are left for themselves. This results in numerous questions and calls to the surgeons’ offices from the patients who seek guidance and reassurance. [0007] Furthermore, there is no objective way to assess a patient’s recovery progress. Conventionally, existing systems rely on patient reported outcomes which may be very subjective and unreliable. Moreover, the method in which the outcomes are collected typically result in poor response rates as the patients’ simply tune the outcome requests out.

[0008] Presently, there exists no objective way to determine how a given patient is recovering compared to a clinical expectation. Moreover, the expectations need to be controlled for the patient’s health, complaints, intraoperative findings.

SUMMARY OF THE DISCLOSURE

[0009] Described herein are systems and methods for providing engaging and personalized postoperative care to patients recovering from surgery. Generally, the postoperative care may be delivered to the patient through an executed application (“app”) downloaded to a patient’s device, such as a smart phone, tablet computer, or the like. The postoperative care may include audio (audible) instructions provided in the voice of the patient’s surgeon. The postoperative care may also include capturing video clips of the patient performing guided exercises. Analysis of these video clips may provide clinicians objective information regarding the patients’ recovery process.

[0010] Any of the methods described herein may provide personalized postoperative care to a patient. Any of the methods may include analyzing a surgical video of a procedure performed on a patient, generating one or more video clips based on the analysis of the surgical video, generating a surgical narrative associated with the one or more the video clips, generating a voiced audio narration based on the surgical narrative, and generating a surgery report based on the voiced audio narration and the one or more video clips.

[0011] In any of the methods described herein, analyzing the surgical video may include executing a first neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof. Furthermore, in any of the methods described herein, analyzing the surgical video may include executing a first neural network trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof. Furthermore, in any of the methods, analyzing the surgical video may include identifying the one or more video clips and identifying timestamps associated with each identified video clip.

[0012] In any of the methods, the surgical narrative associated with the one or more the video clips may be based on predetermined text and descriptions. Furthermore, the predetermined text may be layperson-friendly. [0013] In any of the methods described herein, generating the voice audio narration may include executing a second neural network trained to sound like a patient’s surgeon. In some examples, the second neural network may be a generative adversarial network (GAN).

[0014] In any of the methods described herein, the surgery report may include a description of surgical repairs performed on the patient. Furthermore, lengths of video clips in the surgery report may be adjusted to match lengths of associated voiced audio narrations.

[0015] Any of the methods described herein may include generating a postoperative care report describing surgeon-specific postoperative protocols associated with a patient’s intraoperative findings or surgical procedures. In some examples, the postoperative care report may include a voiced audio greeting including a patient’s name generated by a processor executing a neural network trained to sound like the patient’s operating surgeon. In some examples, the surgery report and the postoperative care report may be delivered to the patient through an application running on a mobile phone, laptop computer, tablet computer, or combination thereof.

[0016] Any of the methods described herein may also include capturing, with a video camera, a patient’s movements while performing a defined exercise and modifying one or more postoperative protocols based on the captured patient’s movements. Any of the methods may further include providing a prompt to the patient to perform the defined exercise, wherein the prompt is an audio prompt based on a neural network trained to sound like a patient’s surgeon. In any of the methods described herein, the modification to the one or more postoperative protocols may be based on a motion analysis of the captured patient’s movements. Furthermore, in any of the methods described herein, the surgical video may be captured by an orthoscopic video camera.

[0017] Any of the non-transitory computer-readable storage mediums described herein may include instructions that, when executed by one or more processors, cause a device to perform operations comprising analyzing a surgical video of a procedure performed on a patient, generating one or more video clips based on the analysis of the surgical video, generating a surgical narrative associated with the one or more the video clips, generating a voiced audio narration based on the surgical narrative, and generating a surgery report based on the voiced audio narration and the one or more video clips.

[0018] Any of the non-transitory computer-readable storage mediums described herein may include instructions for analyzing the surgical video and may cause a device (e.g., a computing device) to perform operations further comprising executing a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof. Furthermore, any of the non-transitory computer-readable storage mediums described herein may include instructions for analyzing the surgical video that are configured to cause a device (e.g., a computing device) to perform operations further comprising executing a neural network trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof. In some examples, any of the non-transitory computer-readable storage mediums described herein may include instructions for analyzing the surgical video that may cause a device (e.g., a computing device) to perform operations further comprising identifying the one or more video clips and identifying timestamps associated with each identified video clip.

[0019] In any of the non-transitory computer-readable storage mediums described herein, the surgical narrative may be based on predetermined text and descriptions. In some examples, the predetermined text is layperson-friendly.

[0020] In any of the non-transitory computer-readable storage mediums described herein, the instructions for generating the voice audio narration may cause the device to perform operations further comprising executing a neural network trained to sound like a patient’s surgeon. In some examples, the neural network may be a generative adversarial network (GAN).

[0021] In any of the non-transitory computer-readable storage mediums described herein, the surgery report may include a description of surgical repairs performed on the patient. In some examples, the lengths of video clips in the surgery report are adjusted to match lengths of associated voiced audio narrations.

[0022] In any of the non-transitory computer-readable storage mediums described herein, execution of the instructions may cause the device to perform operations further comprising generating a postoperative care report describing surgeon-specific postoperative protocols associated with a patient’s intraoperative findings or surgical procedures. In some examples, the postoperative care report includes a voiced audio greeting including a patient’s name generated by a processor executing a neural network trained to sound like the patient’s operating surgeon. [0023] In any of the non-transitory computer-readable storage mediums described herein, the surgery report and the postoperative care report may be delivered to the patient through an application running on a mobile phone, laptop computer, tablet computer, or combination thereof.

[0024] In any of the non-transitory computer-readable storage mediums described herein, execution of the instructions may cause the device to perform operations comprising capturing, with a video camera, a patient’s movements while performing a defined exercise and modifying one or more postoperative protocols based on the captured patient’s movements. In some examples, execution of the instructions may cause the device to perform operations further comprising providing a prompt to the patient to perform the defined exercise, wherein the prompt is an audio prompt based on a neural network trained to sound like a patient’s surgeon. In some examples, the modification to the one or more postoperative protocols is based on a motion analysis of the captured patient’s movements.

[0025] Any of the systems described herein may include one or more processors, and a memory configured to store instructions that, when executed by the one or more processors, cause the system to analyze a surgical video of a procedure performed on a patient, generate one or more video clips based on the analysis of the surgical video, generate a surgical narrative associated with the one or more the video clips, generate a voiced audio narration based on the surgical narrative, and generate a surgery report based on the voiced audio narration and the one or more video clips.

[0026] In any of the systems described herein, execution of the instructions to analyze the surgical video may cause the system to further execute a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof. Furthermore, in any of the systems described herein, execution of the instructions to analyze the surgical video may cause the system to further execute a neural network trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof. In any of the systems described herein, execution of the instructions to analyze the surgical video may cause the system to further identify the one or more video clips and identify timestamps associated with each identified video clip.

[0027] In any of the systems described herein, the surgical narrative associated with the one or more the video clips may be based on predetermined text and descriptions. In some examples, the predetermined text may be layperson-friendly.

[0028] In any of the systems described herein, execution of the instructions to generate the voice audio narration may cause the system to further execute a neural network trained to sound like a patient’s surgeon. In some examples, the neural network may be a generative adversarial network (GAN).

[0029] In any of the systems described herein, the surgery report may include a description of surgical repairs performed on the patient. In some examples, lengths of video clips in the surgery report may be adjusted to match lengths of associated voiced audio narrations.

[0030] In any of the systems described herein, execution of the instructions may cause the system to further generate a postoperative care report describing surgeon-specific postoperative protocols associated with a patient’s intraoperative findings or surgical procedures. In some examples, the postoperative care report may include a voiced audio greeting including a patient’s name generated by a processor executing a neural network trained to sound like the patient’s operating surgeon. In some examples, the surgery report and the postoperative care report may be delivered to the patient through an application running on a mobile phone, laptop computer, tablet computer, or combination thereof.

[0031] In any of the systems described herein, execution of the instructions may cause the system to capture, with a video camera, a patient’s movements while performing a defined exercise and modify one or more postoperative protocols based on the captured patient’s movements. In some examples, execution of the instructions may cause the system to further provide a prompt to the patient to perform the defined exercise, wherein the prompt is an audio prompt based on a neural network trained to sound like a patient’s surgeon. In some examples, wherein the modification to the one or more postoperative protocols may be based on a motion analysis of the captured patient’s movements.

[0032] Any of the methods described herein may provide personalized postoperative care to a patient and include receiving, through a mobile device application, a video clip of a patient performing at least one directed exercise after the patient has undergone a surgery, analyzing the video clip to determine motion of the patient, comparing the motion of the patient to motion of an expected progress model, and sending an alert to a doctor based on the comparison. In some examples, the comparison may indicate a difference greater than a predetermined amount between the motion of the patient and the motion of the expected progress model. In some examples, execution of the instructions may provide, through the mobile device application, personalized prompts to the patient to perform a directed exercise, wherein the video clip is received in response to the personalized prompt. In some examples, the personalized prompts may be based on surgery details of the patient’s surgery. In some examples, the personalized prompts may be generated by a processor executing a neural network trained to sound like a patient’s surgeon.

[0033] In any of the methods described herein, the expected progress model may be provided by a surgeon expert panel’s analysis of a collection of surgery details. In some examples, the surgery details may include images and video segments from an initial diagnostic phase and repair phase of the patient. In some examples, the expected progress model is modified by the surgeon expert panel in response to the video clip of the patient performing at least one directed exercise. In still other examples, the expected progress model may include an estimate of patient recovery for a period of time of up to three months after the surgery of the patient. In further examples, the expected progress model may be based on statistical techniques.

[0034] Any of the non-transitory computer-readable storage mediums described herein may include instructions to cause a device to perform operations including receiving, through a mobile device application, a video clip of a patient performing at least one directed exercise after the patient has undergone a surgery, analyzing the video clip to determine motion of the patient, comparing the motion of the patient to motion of an expected progress model, and sending an alert to a doctor based on the comparison.

[0035] In any of the non-transitory computer-readable storage mediums described herein, operations comparing the motion of the patient may indicate a difference greater than a predetermined amount between the motion of the patient and the motion of the expected progress model.

[0036] In any of the non-transitory computer-readable storage mediums described herein, execution of the instructions may cause the device to perform operations further comprising providing, through the mobile device application, personalized prompts to the patient to perform a directed exercise, wherein the video clip is received in response to the personalized prompt. In some examples, the personalized prompts may be based on surgery details of the patient’s surgery. In some other examples, the personalized prompts may be generated by a processor executing a neural network trained to sound like a patient’s surgeon.

[0037] In any of the non-transitory computer-readable storage mediums described herein, the expected progress model may be provided by a surgeon expert panel’s analysis of a collection of surgery details. In some examples, the surgery details may include images and video segments from an initial diagnostic phase and repair phase of the patient. In some examples, the expected progress model may be modified by the surgeon expert panel in response to the video clip of the patient performing at least one directed exercise. In some examples, the expected progress model may include an estimate of patient recovery for a period of time of up to three months after the surgery of the patient. In some examples, the expected progress model may be based on statistical techniques.

[0038] Any of the systems described herein may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, cause the system to receive, through a mobile device application, a video clip of a patient performing at least one directed exercise after the patient has undergone a surgery, analyze the video clip to determine motion of the patient, compare the motion of the patient to motion of an expected progress model, and send an alert to a doctor based on the comparison.

[0039] In any of the systems described herein, the comparison of the motion of the patient may indicate a difference greater than a predetermined amount between the motion of the patient and the motion of the expected progress model.

[0040] In any of the systems described herein, execution of the instructions may cause the device to further provide, through the mobile device application, personalized prompts to the patient to perform a directed exercise, wherein the video clip is received in response to the personalized prompt. In some examples, the personalized prompts may be based on surgery details of the patient’s surgery. In some examples, the personalized prompts may be generated by a processor executing a neural network trained to sound like a patient’s surgeon.

[0041] In any of the systems described herein, the expected progress model may be provided by a surgeon expert panel’s analysis of a collection of surgery details. In some examples, the surgery details may include images and video segments from an initial diagnostic phase and repair phase of the patient. In some examples, the expected progress model may be modified by the surgeon expert panel in response to the video clip of the patient performing at least one directed exercise. In some examples, the expected progress model may include an estimate of patient recovery for a period of time of up to three months after the surgery of the patient. In some examples, the expected progress model may be based on statistical techniques.

[0042] Any of the methods described herein may provide personalized postoperative care to a patient and include receiving a surgical video of a surgical procedure performed on a patient, generating one or more video clips based on an analysis of the surgical video, wherein the analysis is based on a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof, generating a voiced audio narration based on the surgical narrative, and generating a surgery report based on the voiced audio narration and the one or more video clips.

[0043] Any of the methods described herein may include receiving a surgical video of a surgical procedure performed on a patient, analyzing the surgical video, generating a voiced audio narration of the surgical video, wherein the voiced audio narration is based on a neural network trained to sound like a patient’s surgeon, and generating a surgery report including the voiced audio narration.

[0044] All of the methods and apparatuses described herein, in any combination, are herein contemplated and can be used to achieve the benefits as described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0045] A better understanding of the features and advantages of the methods and apparatuses described herein will be obtained by reference to the following detailed description that sets forth illustrative embodiments, and the accompanying drawings of which:

[0046] FIG. 1 shows an example system for providing customized postoperative care to a patient.

[0047] FIG. 2 is a flowchart showing example procedures or steps for interacting with a patient.

[0048] FIG. 3 is a flowchart showing an example method for surgeon onboarding. [0049] FIG. 4 is a flowchart showing an example method for generating a postoperative care report.

[0050] FIG. 5 shows a block diagram of an example postoperative follow-up system.

[0051] FIG. 6 shows a block diagram of a device that may be one example of the compute node of FIG. 1, or any other feasible device.

DETAILED DESCRIPTION

[0052] Providing personalized and engaging postoperative care may be based, at least in part on using artificial intelligence (sometimes referred to as trained neural networks) to shape or create content for the postoperative patient.

[0053] Described herein are systems and methods for providing engaging and personalized postoperative care to patients recovering from surgery. Generally, the postoperative care may be delivered to the patient through an executed application (“app”) downloaded to a patient’s device, such as a smart phone, tablet computer, or the like. The postoperative care may include audio (audible) instructions provided in the voice of the patient’s surgeon. The postoperative care may also include capturing video clips of the patient performing guided exercises. Analysis of these video clips may provide clinicians objective information regarding the patients’ recovery process.

[0054] FIG. 1 shows an example system 100 for providing customized postoperative care to a patient. The system 100 may include a compute node 110. The compute node 110 may include a processor, computer, or the like. The compute node 110 may be, for example, located in or near a surgeon’s medical office or clinic. In another example, the compute node 110 may be a remote, virtual, or cloud-based processor, computer, or the like remotely located with respect to the surgeon, doctor, or other clinician. Generally, the compute node 110 may include, one or more processors, memory (including dynamic, non-volatile, mechanical, solid-state, or the like), and any number of interfaces (including user interfaces), communication interfaces (serial, parallel, wired, wireless, and the like).

[0055] The system 100 may provide a customized postoperative care report 120 for any patient and/or any operative procedure. The postoperative care report 120 may include a review or summary of an operation recently undergone by the patient. In some examples, the postoperative care report 120 may include one or more video segments that may have been clipped (e.g., copied) from a video of the patient’s actual surgery. The postoperative care report 120 may also include a voice narration describing the operation, including reference to some or all of the video content. In some examples, the voice narration may describe or discuss intraoperative findings, and in some cases describe or discuss visible or discovered pathologies and diagnostics.

[0056] The voice narration may be generated through an artificial intelligence (Al) based model that has been programmed (trained) to closely mimic the voice of the patient’s surgeon. In this manner, the postoperative care report 120 may be received by the patient through a familiar voice. Advantageously, the familiar voice may influence the patient to more carefully follow any postoperative care instructions.

[0057] In some examples, the voice narration may include words from pre-determined templates that explain surgical procedures in a patient-friendly manner. In some cases, the narration may be in a layperson friendly language.

[0058] In some variations, the system 100 may also provide an ongoing patient assessment 130. The ongoing patient assessment 130 may be used to monitor postoperative patient recovery and healing. In some examples, the ongoing patient assessment 130 may collect objective patient data enabling the clinician or surgeon to more accurately determine the patient’s recovery status. [0059] To generate the postoperative care report 120 and/or provide ongoing patient assessment 130, the compute node 110 may include instructions that when executed by the one or more processors, cause the compute node 110 to receive surgical video data 140 and a surgeon’s voice model 150. In some examples, the surgical video data 140 may be the actual video of the patient’s surgery. Execution of the instructions by the compute node 110 may cause the surgical video data 140 to be analyzed by an Al model trained to recognize one or more procedures performed by the surgeon as well as intraoperative findings and visible pathologies.

[0060] In order to provide voice narration, in some examples, compute node 110 may use the surgeon’s voice model 150. The surgeon’s voice model 150 may be determined through an Al model. The Al model may be trained as part of a surgeon onboarding process. The onboarding process is described below with respect to FIG. 3.

[0061] In some variations, the postoperative care report 120 and/or the ongoing patient assessment 130 may be provided to the patient through a device 160. Example devices may include a mobile phone (e.g., cellular phone, smart phone, etc.), a tablet computer, a laptop computer, desktop computer, or any other feasible device. In some examples, one or more functions or operations performed by the compute node 110, may be performed alternately (or in parallel with) the device 160. Thus, the device 160 may include one or more processors, memory, communication interfaces and the like.

[0062] FIG. 2 is a flowchart showing an example method 200 for interacting with a patient. The method 200 describes a high-level overview or summary of processes that may be associated with using artificial intelligence based methods or procedures to provide postoperative or ongoing patient care. Additional detail associated with any processes described in FIG. 2 are described below with respect to FIGS. 3-5. In some examples, the method 200 may be performed with additional operations, fewer operations, operations in a different order, operations in parallel, and some operations differently. The method 200 is described below with respect to the system 100 of FIG. 1, however, the method 200 may be performed by any other suitable system or device.

[0063] The method 200 begins in block 202 where the compute node 110 performs Al training. In block 202, one or more neural networks may be trained to provide or generate the postoperative care report 120 and/or the ongoing patient assessment 130. In one example, the compute node 110 may train a first neural network to implement a selected human voice. The selected human voice may be the patient’s surgeon. In one training example, the surgeon’s voice may be recorded reading words, phrases, medical terminology, sentences, and the like. Using this recording, an Al model may be trained so that any word or phrase may be generated or synthesized (e.g., “spoken” or made audible) such that word or phrase may sound as if they were spoken by the surgeon. The Al model enables a more natural human speech to be synthesized, and advantageously, the human speech may sound very similar to the patient’s surgeon. In some examples, the first neural network may be realized as a generative adversarial network (GAN). During the training of a GAN, two neural networks may contest each other and train the network by generating and testing candidate data after an initial training has been completed based on the recordings of the surgeon. The resulting synthesized human voice may be used to “speak” words that were not part of the training voice recording. Thus, the synthesized speech is not merely a stitching together of previously spoken phrases from the surgeon.

[0064] In another example, the compute node 110 may train a second neural network to recognize and/or identify elements within a surgical video (e.g., the surgical video data 140). Thus, the second neural network may be trained to locate and/or identify intraoperative findings that may be included in the surgical video. In some examples, the second neural network may also be trained to identify diagnostics or pathologies that may be included in the surgical video. Furthermore, in some examples, the second neural network may be trained to identify repairs performed by the surgeon.

[0065] Next, in block 204 a patient’s surgical video is analyzed. In this block, the compute node 110 may use a neural network to analyze a video captured as the patient was undergoing the surgical procedure and generate a summary of the procedure for the patient. Thus, Al procedures (trained in block 202) may be used to identify not only what medical procedures were performed, but also how they were performed. For example, the Al may not only identify that a ligament was repaired, but may also determine how many anchors were used to affix the ligament during the repair.

[0066] Next, in block 206 a surgery analysis is delivered to the patient. For example, the compute node 110 may use the surgery analysis (performed in block 204) to create the postoperative care report 120. The postoperative care report 120 may include a summary of procedures performed, as well as interoperative findings, and/or any pathologies seen. The postoperative care report 120 may also include patient instructions regarding wound care as well as postoperative physical therapy directions. In some variations, the postoperative care report 120 may include video clips taken from the patient’s surgical video that are associated with salient or important procedures. In some other variations, the postoperative care report 120 may include a voice narration delivered in the surgeon’s voice as modeled by a neural network (trained in block 202) to accompany the video clips.

[0067] Next, in block 208 a follow-up with the patient is performed. For example, the compute node 110 may provide the ongoing patient assessment 130 to the patient. The ongoing patient assessment 130 may include estimates of the patient’s expected progress, as well as prompts (through an application running on a smartphone, for example) to the patient to perform exercises while being monitored through a camera. The patient’s movements may be captured through the camera, analyzed, and follow-up instructions may be provided to the patient.

[0068] The method 200 may provide an overview regarding procedures or steps for interacting with a patient. In some variations, procedures associated with the method 200 may divided into two groups. A first group may be associated with surgeon onboarding. Surgeon onboarding may be associated with procedures or processing steps that are typically performed prior to generating or providing a postoperative care report 120 or ongoing patient assessment 130. A second group of procedures or steps may be performed to provide the postoperative care report 120 and/or the ongoing patient assessment 130. The second group of procedures may advantageously use the outcome or results of procedures or operations associated with first group of procedures. The first group of procedures (e.g., surgeon onboarding) is described in more detail below in conjunction with FIG. 3. The second group of procedures (e.g. providing the postoperative care report and/or ongoing patient assessment 130) is described in more detail below in conjunction with FIGS. 4 and 5.

[0069] FIG. 3 is a flowchart showing an example method 300 for surgeon onboarding. Some examples may perform the operations described herein with additional operations, fewer operations, operations in a different order, operations in parallel, and some operations differently. The method 300 is described below with respect to the system 100 of FIG. 1, however, the method 300 may be performed by any other suitable system or device. [0070] The method 300 begins in block 302 as the compute node 110 trains a neural network using the surgeon’s voice. In some examples, training the neural network may include receiving audio data that includes recordings of the surgeon’s voice. In some cases, the surgeon may read aloud a collection of words and phrases that may be used to train a neural network. In some variations, the neural network that may be trained using the surgeon’s voice is a GAN network. The resulting neural network may be used to generate an audio narration that may be used to provide postoperative care, including ongoing patient assessment.

[0071] Next in block 304, the compute node 110 may receive surgeon-specific postoperative protocols. The surgeon-specific postoperative protocols may be text descriptions corresponding to various surgery types and/or sub-types. These protocols descriptions may be provided by the surgeon’s practice. In some examples, the protocols may be in text form and include instructions regarding patient care, including care that the patients should perform themselves before returning to a postoperative consultation. In some examples, the surgeon-specific postoperative care may include descriptions of the surgeon’s techniques to treat one or more specific pathologies. The surgeon-specific postoperative protocols may be used to formulate the postoperative care report 120 and/or the ongoing patient assessment 130.

[0072] Next in block 306, the compute node 110 stores the trained neural network (trained in block 302) and the surgeon-specific postoperative protocols (received in block 304). For example, the trained neural network and the surgeon-specific postoperative protocols may be stored in a memory accessible by, or contained within the compute node 110.

[0073] In some examples, postoperative care may include providing (e.g., creating, generating or synthesizing) the postoperative care report 120 and also monitoring a patient’s recovery progress through the ongoing patient assessment 130. Data from the surgeon onboarding process outlined in FIG. 3 may be incorporated into both providing the postoperative care report 120 and the ongoing patient assessment 130. FIG. 4 describes steps and procedures associated with generating the postoperative care report 120. FIG. 5 describes steps and procedures associated with providing the ongoing patient assessment 130.

[0074] FIG. 4 is a flowchart showing an example method 400 for generating a postoperative care report. The method 400 is described below with respect to the system 100 of FIG. 1, however, the method 400 may be performed by any other suitable system or device. In some examples, a patient engagement application (sometimes referred to as an “app”) may be executed to perform some or all of the operations described in FIG. 4.

[0075] The method 400 begins in block 402 as the compute node 110 receives and/or obtains a surgical video. The surgical video may be a complete or partial video of the surgical procedure performed on the patient. For example, the surgical video may be a recording of an orthopedic repair or reconstruction captured by an orthoscopic camera. The surgical video may include internal or orthoscopic views of joint repair, ligament replacement, anchoring, reattachment, or the like. In some examples, the surgical video may include views provided from multiple cameras of the same surgery.

[0076] Next, in block 404 the compute node 110 analyzes the surgical video using Al. For example, the surgical video may be analyzed through an Al model that has been previously trained to recognize (identify) one or more aspects, features, or characteristics of the surgery. In some variations, the Al model may have been trained to identify patient anatomies, pathologies, note intraoperative findings, and highlight diagnostics. Furthermore, the Al model may identify one or more surgical repairs, including orthopedic repairs. In some variations, the Al model may have been trained to identify particular surgical techniques used by the surgeon, identify different stages of the surgery and, in some cases, identify the number and/or type of anchors used in orthopedic surgery.

[0077] During the analysis, the compute node 110 may assign labels and corresponding timestamps which describe any identified aspects, features, etc., of the surgical video. For example, the compute node 110 may determine labels and timestamps for the surgical video for each of the Al identified aspects, anatomy, pathology, surgery stage, or the like. In this manner, there may be a correspondence between labels (and/or timestamps) and video clips from the surgical video. The compute node 110 may also store these labels and timestamps in a memory. The labels and timestamps may comprise a summary of the interoperative findings and surgical analysis.

[0078] Next, in block 406 the compute node 110 generates a surgical narrative corresponding to the surgical analysis of block 404. For example, based on the interoperative findings and surgical analysis (determined in block 404), the compute node 110 may assemble a corresponding narrative in text form. In some variations, the narrative may be based on predetermined templates and/or descriptions of the various findings and analysis. The templates may provide a layperson-friendly description (e.g., a description understandable by persons not practicing in the field of medicine) of any findings and surgical analysis. The templates may be predetermined, selected, or stored by the surgeon.

[0079] To generate the surgical narrative, the compute node 110 may match one or more portions of the surgical video (identified by labels or timestamps) with an associated narrative. In some cases, pathology recognition (as identified in block 404) may be used to modify the selected narrative template. For example, pathology recognition may identify a large tear or a small tear. The compute node 110 may modify the narrative based on the recognized pathology by, for example, inserting appropriate large tear or small tear terminology or descriptors. [0080] Next, in block 408, the compute node 110 generates an audio (voiced) narration based on the generated surgical narrative. In some examples, the computer node 110 may generate the audio narration based on the neural network trained by the surgeon during the surgeon onboarding described with respect to FIG. 3. Thus, the generated audio narration may be in the surgeon’s voice. In some variations, text-to-voice libraries may be used to generate an initial audio narration that may undergo a follow-up processing to create a narration in the surgeon’s voice, for example, using the neural network described above.

[0081] Next, in block 410, the compute node 110 generates a surgery report. The surgery report may be a video report assembled from video clips from the patient’s surgery and narrated by the surgeon’s voice. For example, the compute node 110 may put together (e.g., assemble or match) a video clip (as determined in block 404) with an audio narration in the (as determined in block 408). Since, the audio narrations may be determined or based on video clips, there may be a one-to-one correspondence between audio narrations and video clips. In some cases, if the duration of the audio narration exceeds the duration of the video clip, then the video clip may be extended by an addition of duplicate video frames. The audio narration may be generated by the neural network trained by the surgeon’s voice during the surgeon onboarding described with respect to FIG. 3.

[0082] Next, in block 412, the computer node 110 generates a postoperative care report. The postoperative care report may include information for the patient regarding different facets of postoperative surgical care customized for the patient. The information may include what the patient can expect during the recovery process, instructions regarding after care, and contact information for members of the patient’s health care team.

[0083] To compile or assemble the postoperative care report, the compute node 110 may retrieve surgeon-specific postoperative protocols that were stored as part of the surgeon onboarding processes of FIG. 3. The retrieved surgeon-specific postoperative protocols may correspond to patient-specific intraoperative findings that were determined during the video analysis of block 404. In some variations, the surgeon-specific postoperative protocols may be associated with surgical procedures performed on the patient. The surgeon-specific postoperative protocols may be in a text or written form and converted to speech in the surgeon’s voice similar to as described with respect to block 408. For example, the compute node 110 may generate an audio narration based on the patient-specific interoperative findings using the neural network trained by the surgeon’s voice during the surgeon onboarding. In some variations, the postoperative care report may be personalized for the patient. For example, the postoperative care report may include a customized greeting including the patient’s name and referencing some surgery details. [0084] Next, in block 414, the surgery report and the postoperative care report are provided to the patient. In some examples, the surgery report and/or the postoperative care report may be provided, presented, or delivered to the patient through a website or program viewed from the patient’s personal computer, laptop computer, or the like. In another example, the surgery report and/or the postoperative care report may be viewed through a patient engagement application running on the patient’s mobile phone, tablet computer, laptop computer or the like. Thus, the compute node 110 may transmit or transfer the surgery report and/or the postoperative care report to a patient’s mobile phone, tablet computer or laptop computer.

[0085] FIG. 5 shows a block diagram of an example postoperative follow-up system 500. Any of the operations of FIG. 5 may be associated with patient care after the patient has undergone surgery. The system 500 may use a patient engagement application (“app”) 510 that may be loaded onto and executed from a patient’s device such as, but not limited to, a smart phone, tablet computer, laptop computer, or the like. As described above, the patient engagement app 510 may be used to deliver or display a patient’s surgery report and/or a patient’s postoperative care report. In some examples, the patient engagement app 510 may also be used to provide a customized postoperative follow-up for a patient. As is discussed below, the patient engagement app 510 may be used to collect real, objective patient assessments associated with patient recovery. The patient engagement app 510 may compare the objective patient assessment to expected recovery and alert the patient’s surgeon based on the comparison.

[0086] To begin, the patient engagement app 510 may receive or obtain the patient’s surgery details 515. For example, the patient’s surgery details 515 may be obtained from, or based on the surgical video analysis of FIG. 4 in block 404. Alternatively, or in addition, the patient’s surgery details 515 may be provided directly or indirectly from the surgeon or surgeon’s office staff. The patient’s surgery details 515 may include information regarding the type of surgery, including any related pathologies, repairs, or findings associated with the patient.

[0087] Using the surgery details 515, the patient engagement app 510 may provide a videobased functional assessment 520. In some examples, after obtaining the patient’s consent the patient engagement app 510 may prompt the patient to perform one or more controlled or defined exercises. The prompt may be delivered to the patient in a voiced instruction based on a neural network trained to sound like the patient’s surgeon. A camera, such as a camera on a smartphone, tablet computer, or laptop computer can capture or record the patient’s movements including capturing the patient’s range of motion while performing the exercises. The videobased functional assessment 520 may analyze the patient’s movements. In some cases, the analysis may be performed with any feasible motion analysis software [0088] At any point before the patient engagement app 510 interacts with the patient to capture objective patient information (block 520), a surgeon expert panel 530 may provide an expected progress model 537. The expected progress model 537 may be an estimate of the patient’s expected recovery progress over time, beginning with the time after the surgery. In some examples, the expected progress model 537 may include recovery estimates for up to three months after the surgery. In some other variations, the expected progress model 537 may include progress estimates for time periods less than or more than three months after the surgery. Members of the surgeon expert panel 530 may include any number expert surgeons and physical therapists. The surgeon expert panel 530 receive a collection of surgery details 535. In some cases, the collection of surgery details 535 may include actual images and video segments from the patient’s initial diagnostic phase and the repair phase. Thus, the surgeon expert panel 530 may analyze the collection of surgery details 535 to determine the expected progress model 537. In some variations, the expected progress model 537 may be based on statistical techniques or analysis of the surgery details 535 performed by the surgeon expert panel 530. In some variations, the expected progress model 537 may be based on a consensus estimate from the surgeon expert panel 530 and determined with statistical techniques.

[0089] Information from the video-based functional assessment 520 is provided to a variance analysis block 540. The variance analysis block 540 can compare the information from the video-based functional assessment 520, the expected progress model 537, and surgeon protocols 539. In some variations, the variance analysis block 646 may detect any differences or deviations between the information from the video-based functional assessment 520 and the expected progress model 537 and/or the surgeon protocols 539. If the detected differences or deviations (e.g., the comparison) are greater than a predetermined amount, then an alert 545 is generated, sent, and/or delivered to a surgeon, doctor, or other clinician.

[0090] The video-based functional assessment 520 also updates a patient progress database 525. Through the patient progress database 525, actual, objective patient recovery may be captured and provided through a feedback path 527 to the surgeon expert panel 530. The surgeon expert panel 530 may assess the patient recovery in the patient progress database 525 and revise, if necessary, the expected progress model 537.

[0091] Although the patient engagement app 510, the video-based functional assessment 520, and the variance analysis block 540 are described as separate blocks or components, in some variations, these blocks may be combined into fewer blocks. For example, the patient engagement app 510 may also provide the functionality associated with video-based functional assessment 520 and the variance analysis block 540. [0092] The system 500 can provide personalized and objective assessment of a patient’s recovery after surgery. Using the patient’s video as captured through the video-based functional assessment 520 and the expected progress model 537 enables a customized analysis of the patient’s recovery. Furthermore, feedback of the patient’s progress to the surgeon expert panel enables updates to a patient’s expected progress model 537. In this manner, a patient’s recovery path may be adjusted and not limited to a stagnant model.

[0093] FIG. 6 shows a block diagram of a device 600 that may be one example of the compute node 110 of FIG. 1, or any other feasible device. The device 600 may include a transceiver 620, a processor 630, and a memory 640.

[0094] The transceiver 620, which is coupled to the processor 630, may be used to transmit and/or receive data with any other feasible device. For example, the transceiver 620 may communicate with any feasible wired (e.g., ethernet, serial, parallel, optical, or the like) or wireless (Wi-Fi, Bluetooth, ZigBee, loT, or the like) protocol. In some examples, the transceiver 620 may communicate with a plurality of devices through a plurality of wired and/or wireless networks (sometimes referred to as “the cloud” or the “internet”).

[0095] The processor 630 may be any one or more suitable processors capable of executing scripts or instructions of one or more software programs stored in the device 600 (such as within memory 640).

[0096] The memory 640 may include a narrative template database 642 that may be used to store predefined descriptions of surgical procedures in an easy to understand language. In some examples, the narrative template database 642 may include text descriptions of patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, surgical repairs, orthopedic repairs, surgical techniques, or the like. In some examples, the narrative template database 642 may also include text descriptions of various patient exercises.

[0097] The memory 640 may include an exercise database 643. The exercise database 643 may include a variety of exercises that may be used to assess a patient regarding a surgery. For example, the exercise database 643 may include exercises that may assess shoulder, elbow, knee, or hip repair. In other examples, the exercise database 643 may include exercises that may enable a clinician to assess any feasible joint.

[0098] The memory 640 may include patient information 644. The patient information 644 may include patient name, age, or other demographic information. In some examples, the patient information 644 may be used to guide or direct a patient in performing the exercises. In some other examples, the patient information 644 may be used to create a customized greeting for the patient. [0099] The memory 640 may also include a non-transitory computer-readable storage medium (e.g., one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, a hard drive, etc.) that may store the following software modules: a communication software (SW) module 645 to communicate through the transceiver 620;

• a video analysis module 646 to analyze video clips;

• a voice synthesis module 647 to synthesize a clinician’s voice; and

• a patient engagement application module 648 to interact with the patient.

Each software module includes program instructions that, when executed by the processor 630, may cause the device 600 to perform the corresponding function(s). Thus, the non-transitory computer-readable storage medium of memory 640 may include instructions for performing all or a portion of the operations described herein.

[0100] The processor 630 may execute the communication SW module 645 to communicate with one or more devices. For example, execution of the communication SW module 645 may enable the device 600 to communicate with one or more other devices through any feasible wireless protocol. Feasible wireless protocols may include, but are not limited to Wi-Fi (e.g., any feasible IEEE 802.11 protocol), Bluetooth, Zigbee, or any other feasible wireless protocol. In other examples, execution of the communication SW module 645 may enable the device 600 to communicate with one or more other devices through any feasible wired protocol including, but not limited to any feasible Ethernet protocol. In some examples, execution of the communication SW module 645 may enable the device 600 to communicate using both wired and wireless protocols.

[0101] The processor 630 may execute the video analysis module 646 to analyze any feasible video. For example, execution of the video analysis module 646 may analyze surgical videos and/or videos of the patient performing guided exercises. In some examples, execution of the video analysis module 646 may cause a neural network to recognize one or more features, characteristics, and/or events in a surgical video. For example, the neural network may be trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof. Alternatively, or in addition, the neural network may be trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof. [0102] In some variations, execution of the video analysis module 646 may cause a neural network to analyze the motion of a patient. For example, a camera (such as a smart phone or laptop camera) may capture video clips of the patient performing one or more directed exercises. Thus, the execution of the video analysis module 646 may analyze these video clips.

[0103] The processor 630 may execute the voice synthesis module 647 to synthesize a surgeon’s or doctor’s voice. In some examples, the synthesized voice may be the voice of the surgeon who performed the operation on the patient. In some variations, execution of the voice synthesis module 647 may cause a neural network to generate or synthesize a voice that has been trained by the patient’s surgeon. In some implementations, the neural network is a generative adversarial network (GAN). In some examples, the synthesized voice is based on a narrative stored in the narrative template database 642.

[0104] The processor 630 may execute the patient engagement application module 648 to provide a surgery report or a postoperative care report to the patient. In some cases, execution of the patient engagement application module 648 may include video clips and a voiced narration in a voice generated by the voice synthesis module 647. In some examples, execution of the patient engagement application module 648 may direct the patient to perform one or more exercises and also cause a camera to capture the patient performing those exercises, thereby enabling motion analysis of the patient.

[0105] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein and may be used to achieve the benefits described herein.

[0106] The process parameters and sequence of steps described and/or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed. The various example methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or include additional steps in addition to those disclosed.

[0107] Any of the methods (including user interfaces) described herein may be implemented as software, hardware or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions capable of being executed by a processor (e.g., computer, tablet, smartphone, etc.), that when executed by the processor causes the processor to control perform any of the steps, including but not limited to: displaying, communicating with the user, analyzing, modifying parameters (including timing, frequency, intensity, etc.), determining, alerting, or the like. For example, any of the methods described herein may be performed, at least in part, by an apparatus including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the processes(s) of the method.

[0108] While various embodiments have been described and/or illustrated herein in the context of fully functional computing systems, one or more of these example embodiments may be distributed as a program product in a variety of forms, regardless of the particular type of computer-readable media used to actually carry out the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include script, batch, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the example embodiments disclosed herein.

[0109] As described herein, the computing devices and systems described and/or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing device(s) may each comprise at least one memory device and at least one physical processor.

[0110] The term “memory” or “memory device,” as used herein, generally represents any type or form of volatile or non-volatile storage device or medium capable of storing data and/or computer-readable instructions. In one example, a memory device may store, load, and/or maintain one or more of the modules described herein. Examples of memory devices comprise, without limitation, Random Access Memory (RAM), Read Only Memory (ROM), flash memory, Hard Disk Drives (HDDs), Solid-State Drives (SSDs), optical disk drives, caches, variations or combinations of one or more of the same, or any other suitable storage memory. [OHl] In addition, the term “processor” or “physical processor,” as used herein, generally refers to any type or form of hardware-implemented processing unit capable of interpreting and/or executing computer-readable instructions. In one example, a physical processor may access and/or modify one or more modules stored in the above-described memory device. Examples of physical processors comprise, without limitation, microprocessors, microcontrollers, Central Processing Units (CPUs), Field-Programmable Gate Arrays (FPGAs) that implement softcore processors, Application-Specific Integrated Circuits (ASICs), portions of one or more of the same, variations or combinations of one or more of the same, or any other suitable physical processor.

[0112] Although illustrated as separate elements, the method steps described and/or illustrated herein may represent portions of a single application. In addition, in some embodiments one or more of these steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks, such as the method step.

[0113] In addition, one or more of the devices described herein may transform data, physical devices, and/or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules recited herein may transform a processor, volatile memory, non-volatile memory, and/or any other portion of a physical computing device from one form of computing device to another form of computing device by executing on the computing device, storing data on the computing device, and/or otherwise interacting with the computing device.

[0114] The term “computer-readable medium,” as used herein, generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media comprise, without limitation, transmission-type media, such as carrier waves, and non-transitory-type media, such as magnetic- storage media (e.g., hard disk drives, tape drives, and floppy disks), optical -storage media (e.g., Compact Disks (CDs), Digital Video Disks (DVDs), and BLU-RAY disks), electronic-storage media (e.g., solid-state drives and flash media), and other distribution systems.

[0115] A person of ordinary skill in the art will recognize that any process or method disclosed herein can be modified in many ways. The process parameters and sequence of the steps described and/or illustrated herein are given by way of example only and can be varied as desired. For example, while the steps illustrated and/or described herein may be shown or discussed in a particular order, these steps do not necessarily need to be performed in the order illustrated or discussed.

[0116] The various exemplary methods described and/or illustrated herein may also omit one or more of the steps described or illustrated herein or comprise additional steps in addition to those disclosed. Further, a step of any method as disclosed herein can be combined with any one or more steps of any other method as disclosed herein.

[0117] The processor as described herein can be configured to perform one or more steps of any method disclosed herein. Alternatively or in combination, the processor can be configured to combine one or more steps of one or more methods as disclosed herein.

[0118] When a feature or element is herein referred to as being "on" another feature or element, it can be directly on the other feature or element or intervening features and/or elements may also be present. In contrast, when a feature or element is referred to as being "directly on" another feature or element, there are no intervening features or elements present. It will also be understood that, when a feature or element is referred to as being "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to the other feature or element or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intervening features or elements present. Although described or shown with respect to one embodiment, the features and elements so described or shown can apply to other embodiments. It will also be appreciated by those of skill in the art that references to a structure or feature that is disposed "adjacent" another feature may have portions that overlap or underlie the adjacent feature.

[0119] Terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. For example, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as "/".

[0120] Although the terms “first” and “second” may be used herein to describe various features/elements (including steps), these features/elements should not be limited by these terms, unless the context indicates otherwise. These terms may be used to distinguish one feature/ element from another feature/element. Thus, a first feature/element discussed below could be termed a second feature/element, and similarly, a second feature/element discussed below could be termed a first feature/element without departing from the teachings of the present invention.

[0121] Throughout this specification and the claims which follow, unless the context requires otherwise, the word “comprise”, and variations such as “comprises” and “comprising” means various components can be co-jointly employed in the methods and articles (e.g., compositions and apparatuses including device and methods). For example, the term “comprising” will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0122] In general, any of the apparatuses and methods described herein should be understood to be inclusive, but all or a sub-set of the components and/or steps may alternatively be exclusive, and may be expressed as “consisting of’ or alternatively “consisting essentially of’ the various components, steps, sub-components or sub-steps. [0123] As used herein in the specification and claims, including as used in the examples and unless otherwise expressly specified, all numbers may be read as if prefaced by the word "about" or “approximately,” even if the term does not expressly appear. The phrase “about” or “approximately” may be used when describing magnitude and/or position to indicate that the value and/or position described is within a reasonable expected range of values and/or positions. For example, a numeric value may have a value that is +/- 0.1% of the stated value (or range of values), +/- 1% of the stated value (or range of values), +/- 2% of the stated value (or range of values), +/- 5% of the stated value (or range of values), +/- 10% of the stated value (or range of values), etc. Any numerical values given herein should also be understood to include about or approximately that value, unless the context indicates otherwise. For example, if the value " 10" is disclosed, then "about 10" is also disclosed. Any numerical range recited herein is intended to include all sub-ranges subsumed therein. It is also understood that when a value is disclosed that "less than or equal to" the value, "greater than or equal to the value" and possible ranges between values are also disclosed, as appropriately understood by the skilled artisan. For example, if the value "X" is disclosed the "less than or equal to X" as well as "greater than or equal to X" (e.g., where X is a numerical value) is also disclosed. It is also understood that the throughout the application, data is provided in a number of different formats, and that this data, represents endpoints and starting points, and ranges for any combination of the data points. For example, if a particular data point “10” and a particular data point “15” are disclosed, it is understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15 are considered disclosed as well as between 10 and 15. It is also understood that each unit between two particular units are also disclosed. For example, if 10 and 15 are disclosed, then 11, 12, 13, and 14 are also disclosed.

[0124] Although various illustrative embodiments are described above, any of a number of changes may be made to various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments one or more method steps may be skipped altogether. Optional features of various device and system embodiments may be included in some embodiments and not in others. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be interpreted to limit the scope of the invention as it is set forth in the claims.

[0125] The examples and illustrations included herein show, by way of illustration and not of limitation, specific embodiments in which the subject matter may be practiced. As mentioned, other embodiments may be utilized and derived there from, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept, if more than one is, in fact, disclosed. Thus, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

Claims

CLAIMS What is claimed is:

1. A method of providing personalized postoperative care to a patient treated by a surgeon, the method comprising: analyzing a surgical video of a procedure performed on a patient; generating one or more video clips based on the analysis of the surgical video; generating a surgical narrative associated with the one or more the video clips; generating a voiced audio narration based on the surgical narrative in the surgeon’s voice; and generating a surgery report including the voiced audio narration and the one or more video clips.

2. The method of claim 1, wherein analyzing the surgical video includes executing a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof.

3. The method of claim 1, wherein analyzing the surgical video includes executing a neural network trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof.

4. The method of claim 1, wherein analyzing the surgical video includes identifying the one or more video clips and identifying timestamps associated with each identified video clip.

5. The method of claim 1, wherein the surgical narrative is based on predetermined text and descriptions.

6. The method of claim 5, wherein the predetermined text is layperson-friendly.

7. The method of claim 1, wherein generating the voice audio narration includes executing a neural network trained to sound like a patient’s surgeon.

8. The method of claim 7, wherein the neural network is a generative adversarial network (GAN).

9. The method of claim 1, wherein the surgery report includes a description of surgical repairs performed on the patient.

10. The method of claim 1, wherein lengths of video clips in the surgery report are adjusted to match lengths of associated voiced audio narrations.

11. The method of claim 1, further comprising generating a postoperative care report describing surgeon-specific postoperative protocols associated with a patient’s intraoperative findings or surgical procedures.

12. The method of claim 11, wherein the postoperative care report includes a voiced audio greeting including a patient’s name generated by a processor executing a neural network trained to sound like the patient’s operating surgeon.

13. The method of claim 11, wherein the surgery report and the postoperative care report are delivered to the patient through an application running on a mobile phone, laptop computer, tablet computer, or combination thereof.

14. The method of claim 11, further comprising: capturing, with a video camera, a patient’s movements while performing a defined exercise; and modifying one or more postoperative protocols based on the captured patient’s movements.

15. The method of claim 14, further comprising providing a prompt to the patient to perform the defined exercise, wherein the prompt is an audio prompt based on a neural network trained to sound like a patient’s surgeon.

16. The method of claim 14, wherein the modification to the one or more postoperative protocols is based on a motion analysis of the captured patient’s movements.

17. The method of claim 1, wherein the surgical video is captured by an orthoscopic video camera.

18. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause a device to perform operations comprising: analyzing a surgical video of a procedure performed on a patient by a surgeon; generating one or more video clips based on the analysis of the surgical video; generating a surgical narrative associated with the one or more the video clips; generating a voiced audio narration based on the surgical narrative in the surgeon’s voice; and generating a surgery report including the voiced audio narration and the one or more video clips.

19. The non-transitory computer-readable storage medium of claim 18, wherein analyzing the surgical video causes the device to perform operations further comprising executing a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof.

20. The non-transitory computer-readable storage medium of claim 18, wherein analyzing the surgical video causes the device to perform operations further comprising executing a neural network trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof.

21. The non-transitory computer-readable storage medium of claim 18, wherein analyzing the surgical video causes the device to perform operations further comprising identifying the one or more video clips and identifying timestamps associated with each identified video clip.

22. The non-transitory computer-readable storage medium of claim 18, wherein the surgical narrative is based on predetermined text and descriptions.

23. The non-transitory computer-readable storage medium of claim 22, wherein the predetermined text is layperson-friendly.

24. The non-transitory computer-readable storage medium of claim 18, wherein generating the voice audio narration causes the device to perform operations further comprising executing a neural network trained to sound like a patient’s surgeon.

25. The non-transitory computer-readable storage medium of claim 24, wherein the neural network is a generative adversarial network (GAN).

26. The non-transitory computer-readable storage medium of claim 18, wherein the surgery report includes a description of surgical repairs performed on the patient.

27. The non-transitory computer-readable storage medium of claim 18, wherein lengths of video clips in the surgery report are adjusted to match lengths of associated voiced audio narrations.

28. The non-transitory computer-readable storage medium of claim 18, wherein execution of the instructions causes the device to perform operations further comprising generating a postoperative care report describing surgeon-specific postoperative protocols associated with a patient’s intraoperative findings or surgical procedures.

29. The non-transitory computer-readable storage medium of claim 28, wherein the postoperative care report includes a voiced audio greeting including a patient’s name generated by a processor executing a neural network trained to sound like the patient’s operating surgeon.

30. The non-transitory computer-readable storage medium of claim 28, wherein the surgery report and the postoperative care report are delivered to the patient through an application running on a mobile phone, laptop computer, tablet computer, or combination thereof.

31. The non-transitory computer-readable storage medium of claim 28, wherein execution of the instructions causes the device to perform operations further comprising: capturing, with a video camera, a patient’s movements while performing a defined exercise; and modifying one or more postoperative protocols based on the captured patient’s movements.

32. The non-transitory computer-readable storage medium of claim 31, wherein execution of the instructions causes the device to perform operations further comprising providing a prompt to the patient to perform the defined exercise, wherein the prompt is an audio prompt based on a neural network trained to sound like a patient’s surgeon.

33. The non-transitory computer-readable storage medium of claim 31, wherein the modification to the one or more postoperative protocols is based on a motion analysis of the captured patient’s movements.

34. A system comprising: one or more processors; and a memory configured to store instructions that, when executed by the one or more processors, cause the system to: analyze a surgical video of a procedure performed on a patient by a surgeon; generate one or more video clips based on the analysis of the surgical video; generate a surgical narrative associated with the one or more the video clips; generate a voiced audio narration based on the surgical narrative in the surgeon’s voice; and generate a surgery report based on the voiced audio narration and the one or more video clips.

35. The system of claim 34, wherein execution of the instructions to analyze the surgical video causes the system to further execute a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, or a combination thereof.

36. The system of claim 34, wherein execution of the instructions to analyze the surgical video causes the system to further execute a neural network trained to recognize surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof.

37. The system of claim 34, wherein execution of the instructions to analyze the surgical video causes the system to further identify the one or more video clips and identify timestamps associated with each identified video clip.

38. The system of claim 34, wherein the surgical narrative is based on predetermined text and descriptions.

39. The system of claim 38, wherein the predetermined text is layperson-friendly.

40. The system of claim 34, wherein execution of the instructions to generate the voice audio narration causes the system to further execute a neural network trained to sound like a patient’s surgeon.

41. The system of claim 40, wherein the neural network is a generative adversarial network (GAN).

42. The system of claim 34, wherein the surgery report includes a description of surgical repairs performed on the patient.

43. The system of claim 34, wherein lengths of video clips in the surgery report are adjusted to match lengths of associated voiced audio narrations.

44. The system of claim 34, wherein execution of the instructions causes the system to further generate a postoperative care report describing surgeon-specific postoperative protocols associated with a patient’s intraoperative findings or surgical procedures.

45. The system of claim 44, wherein the postoperative care report includes a voiced audio greeting including a patient’s name generated by a processor executing a neural network trained to sound like the patient’s operating surgeon.

46. The system of claim 44, wherein the surgery report and the postoperative care report are delivered to the patient through an application running on a mobile phone, laptop computer, tablet computer, or combination thereof.

47. The system of claim 44, wherein execution of the instructions causes the system to further: capture, with a video camera, a patient’s movements while performing a defined exercise; and modify one or more postoperative protocols based on the captured patient’s movements.

48. The system of claim 47, wherein execution of the instructions causes the system to further provide a prompt to the patient to perform the defined exercise, wherein the prompt is an audio prompt based on a neural network trained to sound like a patient’s surgeon.

49. The system of claim 47, wherein the modification to the one or more postoperative protocols is based on a motion analysis of the captured patient’s movements.

50. A method of providing personalized postoperative care to a patient, the method comprising: receiving, through a mobile device application, a video clip of a patient performing at least one directed exercise after the patient has undergone a surgery; analyzing the video clip to determine motion of the patient; comparing the motion of the patient to motion of an expected progress model; and sending an alert to a doctor based on the comparison.

51. The method of claim 50, wherein the comparison indicates a difference greater than a predetermined amount between the motion of the patient and the motion of the expected progress model.

52. The method of claim 50, further comprising: providing, through the mobile device application, personalized prompts to the patient to perform a directed exercise, wherein the video clip is received in response to the personalized prompt.

53. The method of claim 52, wherein the personalized prompts are based on surgery details of the patient’s surgery.

54. The method of claim 52, wherein the personalized prompts are generated by a processor executing a neural network trained to sound like a patient’s surgeon.

55. The method of claim 50, wherein the expected progress model is provided by a surgeon expert panel’s analysis of a collection of surgery details.

56. The method of claim 55, wherein the surgery details include images and video segments from an initial diagnostic phase and repair phase of the patient.

57. The method of claim 55, wherein the expected progress model is modified by the surgeon expert panel in response to the video clip of the patient performing at least one directed exercise.

58. The method of claim 55, wherein the expected progress model includes an estimate of patient recovery for a period of time of up to three months after the surgery of the patient.

59. The method of claim 55, wherein the expected progress model is based on statistical techniques.

60. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors, cause a device to perform operations comprising: receiving, through a mobile device application, a video clip of a patient performing at least one directed exercise after the patient has undergone a surgery; analyzing the video clip to determine motion of the patient; comparing the motion of the patient to motion of an expected progress model; and sending an alert to a doctor based on the comparison.

61. The non-transitory computer-readable storage medium of claim 60, wherein the comparison indicates a difference greater than a predetermined amount between the motion of the patient and the motion of the expected progress model.

62. The non-transitory computer-readable storage medium of claim 60, wherein execution of the instructions causes the device to perform operations further comprising providing, through the mobile device application, personalized prompts to the patient to perform a directed exercise, wherein the video clip is received in response to the personalized prompt.

63. The non-transitory computer-readable storage medium of claim 62, wherein the personalized prompts are based on surgery details of the patient’s surgery.

64. The non-transitory computer-readable storage medium of claim 62, wherein the personalized prompts are generated by a processor executing a neural network trained to sound like a patient’s surgeon.

65. The non-transitory computer-readable storage medium of claim 60, wherein the expected progress model is provided by a surgeon expert panel’s analysis of a collection of surgery details.

66. The non-transitory computer-readable storage medium of claim 65, wherein the surgery details include images and video segments from an initial diagnostic phase and repair phase of the patient.

67. The non-transitory computer-readable storage medium of claim 65, wherein the expected progress model is modified by the surgeon expert panel in response to the video clip of the patient performing at least one directed exercise.

68. The non-transitory computer-readable storage medium of claim 65, wherein the expected progress model includes an estimate of patient recovery for a period of time of up to three months after the surgery of the patient.

69. The non-transitory computer-readable storage medium of claim 65, wherein the expected progress model is based on statistical techniques.

70. A system comprising: one or more processors; and a memory configured to store instructions that, when executed by the one or more processors, cause the system to: receive, through a mobile device application, a video clip of a patient performing at least one directed exercise after the patient has undergone a surgery; analyze the video clip to determine motion of the patient; compare the motion of the patient to motion of an expected progress model; and send an alert to a doctor based on the comparison.

71. The system of claim 70, wherein the comparison indicates a difference greater than a predetermined amount between the motion of the patient and the motion of the expected progress model.

72. The system of claim 70, wherein execution of the instructions causes the device to further provide, through the mobile device application, personalized prompts to the patient to perform a directed exercise, wherein the video clip is received in response to the personalized prompt.

73. The system of claim 72, wherein the personalized prompts are based on surgery details of the patient’s surgery.

74. The system of claim 72, wherein the personalized prompts are generated by a processor executing a neural network trained to sound like a patient’s surgeon.

75. The system of claim 70, wherein the expected progress model is provided by a surgeon expert panel’s analysis of a collection of surgery details.

76. The system of claim 75, wherein the surgery details include images and video segments from an initial diagnostic phase and repair phase of the patient.

77. The system of claim 75, wherein the expected progress model is modified by the surgeon expert panel in response to the video clip of the patient performing at least one directed exercise.

78. The system of claim 75, wherein the expected progress model includes an estimate of patient recovery for a period of time of up to three months after the surgery of the patient.

79. The system of claim 75, wherein the expected progress model is based on statistical techniques.

80. A method of providing personalized postoperative care to a patient, the method comprising: receiving a surgical video of a surgical procedure performed on a patient; generating one or more video clips based on an analysis of the surgical video, wherein the analysis is based on a neural network trained to recognize patient anatomies, patient pathologies, intraoperative findings, patient diagnostics, surgical repairs, orthopedic repairs, surgical techniques, or a combination thereof; generating a voiced audio narration based on the analysis; and generating a surgery report based on the voiced audio narration and the one or more video clips.

81. A method of providing personalized postoperative care to a patient, the method comprising: receiving a surgical video of a surgical procedure performed on a patient; analyzing the surgical video; generating a voiced audio narration of the surgical video, wherein the voiced audio narration is based on a neural network trained to sound like a patient’s surgeon; and generating a surgery report including the voiced audio narration.