JP2018200493A - Treatment process estimation device - Google Patents

Abstract

To provide a treatment process estimation device capable of providing information for facilitating confirmation and editing of images related to treatment actions without requiring any special equipment or standard treatment program.SOLUTION: A process estimation device 15 for estimating a treatment process on the basis of a treatment process chart which contains videos, sounds, equipment signals, and written details of treatment actions, includes: a determinable treatment step determination part 151 configured to determine a treatment process which can be deterministically estimated; a feature extraction auxiliary information acquisition part 152 configured to acquire a piece of supplementary information on the basis of the treatment process; a feature information extraction part 153 configured to extract a piece of feature information using the supplementary information; and a treatment process estimation section 154 configured to estimate a treatment process on the basis of the feature information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a treatment process estimation apparatus that detects a treatment process at each time during a treatment action using acquired video, audio, and device information related to the treatment action.

  In recent years, opportunities for using images have increased in the medical field. For example, video images of the surgical site and operating room during surgery are used for examining surgical content and educating medical personnel. At that time, the video is not used as it is, but the contents of the surgery are examined and edited in accordance with the education of the medical staff. For that purpose, it is necessary to check and edit the shot video, especially when the video is for a long time or when multiple videos are recorded for one treatment. There has been a problem that the editing work takes a lot of time and effort.

  In order to solve this problem, Patent Document 1 proposes a technique for tagging an important part of a therapeutic action video and simplifying the confirmation of the video. For example, a technique is described in which a treatment process is estimated based on position information of an IC tag attached to a surgical instrument or a doctor, and an image is reproduced for each estimated treatment process.

  In Patent Document 2, the treatment process at each time is estimated by following the transition of the treatment process in accordance with a standard treatment program (ACLS, JATEC, etc.) in which specific contents and transition conditions of the treatment process are defined. Based on the estimated information, a technique for reproducing a video for each treatment process is described.

JP 2006-164251 A JP 2006-252286 A

  However, the techniques described in Patent Document 1 and Patent Document 2 can estimate a treatment process only for a treatment action that satisfies a specific condition. As described in Patent Document 1, when an IC tag is used for estimating a treatment process, a special instrument to which an IC tag is attached and an IC tag reader are required. There is a problem that the treatment process cannot be estimated. Further, as described in Patent Document 2, when a treatment action is estimated in accordance with a standard treatment program, there is a problem that a treatment process cannot be estimated for a video of a treatment action that does not correspond to the standard treatment program.

  The present invention has been made in view of the above circumstances, and an object thereof is to realize a medical information processing apparatus capable of estimating a treatment process without using a special instrument such as an IC tag or a standard treatment program. It is in.

In order to solve the above problems, a treatment process estimation apparatus according to the present invention is a treatment process estimation apparatus that determines process information related to one or more treatment processes, and includes at least one media data related to the treatment process. A media data acquisition unit to acquire, a treatment process table acquisition unit to acquire a treatment process table indicating the treatment process, and a determination to determine process information regarding a determinable treatment process based on at least the media data and the treatment process table A possible treatment process determination unit, at least process information relating to the definable treatment process, an auxiliary information acquisition unit for obtaining auxiliary information based on the media data and the treatment process table, and at least process information relating to the definable treatment process , A treatment process for estimating the process information based on the media data, the treatment process table, and the auxiliary information And a tough.

  The auxiliary information is sample information.

  Further, the auxiliary information is processing setting information.

  Further, the sample information is information that can be acquired from an image of an operation site based on at least process information relating to the definable treatment process and the treatment process table.

  Further, the sample information is information that can be acquired from voice information in which an instruction during treatment is recorded based on at least process information relating to the definable treatment process and the treatment process table.

  In addition, the treatment process estimation unit estimates the process information based on at least a straight line detection process result based on the process setting information and an identification result identifying a medical instrument used in the video.

  The determinable treatment process determining unit determines the process information of each work constituting the treatment process, and the treatment process estimating unit estimates the process information of each work constituting the treatment process. It is characterized by.

  In addition, the treatment process estimation unit uses at least the treatment process table and process information on the definable treatment process to determine the types and number of medical devices to be detected at a time when the process information is not estimated. The process information is estimated based on the estimated information including the information.

  In addition, when the process information of the treatment process before and after the unestimated treatment process is estimated or determined, the treatment process estimation unit includes at least the treatment process table and process information on the determinable treatment process; Based on the estimated process information, information on an unestimated treatment process is estimated.

According to the present invention, a treatment process that can be deterministically determined is determined from the input treatment process table, video, audio, and device signal, and from there, auxiliary information (feature extraction auxiliary information consisting of sample information and processing setting information) It is also possible to provide a function of extracting feature information from video, audio, and device signals input using the information, and estimating a treatment process using the feature information.

  By using sample information for feature information extraction, what the extracted feature information means and what is its criterion can be extracted from the information recorded in the actual treatment. Compared with a certain prior art, feature information can be used more effectively.

  By using processing setting information for feature information extraction, more effective feature information extraction is possible because optimal control settings can be used compared to conventional techniques that require preset control settings for feature information extraction. .

It is a figure which shows the structural example of the process information estimation part contained in the treatment process estimation apparatus in embodiment of this invention. It is a block diagram which shows the structural example of the treatment process estimation apparatus. It is a figure which shows an example of a treatment process table | surface for demonstrating operation | movement of the treatment process estimation apparatus. It is a figure which shows an example of the process information output from each component of a process estimation part and a process estimation part for demonstrating operation | movement of the treatment process estimation apparatus. It is a figure which shows an example of the flowchart of feature extraction auxiliary information acquisition for demonstrating operation | movement of the treatment process estimation apparatus. It is a figure which shows the structural example of the feature information extraction part for demonstrating operation | movement of the treatment process estimation apparatus. It is a figure which shows the modification of a process information estimation part for demonstrating operation | movement of the treatment process estimation apparatus.

  FIG. 2 is a block diagram illustrating a configuration example of the treatment process estimation apparatus 1 in the present embodiment. As shown in FIG. 2, the treatment process estimation device 1 is a device that generally derives and records process information based on a treatment process table and media data (video, audio, device signal). Before explaining each component, below, first, the treatment process table in this embodiment and the detail of an image | video, an audio | voice, and an apparatus signal are demonstrated.

[Treatment schedule]
The treatment process table is a table including the order of treatment processes and detailed information of each treatment process for a plurality of treatment processes executed from the start to the end of the treatment process. The detailed information on the treatment process includes the name of the treatment process, information on devices used in the treatment process (for example, use order and type), and information on electronic devices in the treatment process (for example, use order and type).

  Hereinafter, an example of a treatment process table used for explaining the present embodiment will be described with reference to FIG. FIG. 3 is an example of a treatment process table for cardiac adhesion excision surgery. The treatment process table includes columns of treatment process names, instruments used, and electronic devices. The treatment process name represents the name of each treatment process. As shown in FIG. 3, the treatment process table includes “treatment start”, “anaesthesia”, “disinfection”, “skin incision”, “stereotomy”, “pericardiotomy”, “artificial cardiopulmonary” as treatment process names. `` Operation '', `` heart adhesion excision '', `` artificial cardiopulmonary arrest '', `` pericardial suture '', `` sternum suture '', `` skin suture '', `` procedure end '' are listed in this order, Match the order. The column of used instruments represents the names and the order of use of medical instruments used in the treatment process such as a knife and tweezers, and the column of electronic instruments represents the names of the electronic instruments used in the treatment process and the order of use. Use of appliances and electronic devices is performed in the order from the top item in the table. In addition, the order of use of the used instrument or electronic device in one treatment process is the order described from the top of the treatment process table regardless of the used instrument and the electronic device. For example, in the case of a “skin incision” treatment process, the order of use in the process is the order of a knife, tweezers, an electric knife cutting mode, and an electric knife hemostasis mode. As for the treatment process of “anesthesia”, an anesthetic injection operation using an electronic device is shown. For the “disinfection” treatment process, work using tweezers is shown. Regarding the treatment process of “skin incision”, two types of instruments, a scalpel and tweezers, and an operation sequence using an electric scalpel are shown. FIG. 3 is an example of a treatment process table, and items such as scheduled work time, work person in charge, and patient status may be added as detailed information of each treatment process.

Note that the treatment process table that can be used as an input of the treatment process estimation apparatus 1 of the present embodiment does not necessarily have the form of the table as shown in FIG. 3. As long as the detailed information can be derived, another format may be used. For example, data indicating the order of treatment steps and data for reading detailed information of each treatment step may be read and combined for use.

[Video / Audio / Equipment signal]
The video, audio, and device signal are signals transmitted from various devices regarding video, audio, and device control information related to treatment.

  The video is composed of a plurality of frames associated with time. For the association, for example, a time stamp given for each frame is used. As a video at a certain time t, a frame having a time stamp closest to that time t can be used.

  Video supplementary information is attached to the video. The video supplementary information includes classification information indicating the video classification according to the shooting target. The classification information is information for identifying the following video, for example. (1) Video taken of the patient's surgical site, (2) Video taken of the medical device, (3) Video taken of the whole or part of the treatment room, (4) Medical staff assisting the doctor Medical equipment such as a picture of the equipment at hand, (5) a picture of the doctor's facial expression and eyes, a picture taken from the doctor's eyes with a camera worn by the doctor, and (6) a biological monitor for treatment. (7) Video of the operation of a medical device such as a heart-lung machine or video of the operation panel of the device, (8) Video of an endoscopic camera or microscope, (9) Excised from a patient An image of a human body tissue and its location.

  Voice is composed of a plurality of frames associated with time. For the association, for example, a time stamp given for each frame is used. When a sound signal is detected at a time t as a sound at a certain time t, a sound signal from a frame where the sound signal starts to be detected to a frame where it is no longer detected can be used. Audio is accompanied by audio supplementary information. The audio supplementary information includes classification information indicating the audio classification corresponding to the recording target. The voice classification is information for identifying the following voices, for example. (1) Voice recorded by the doctor, voice recorded by the patient, (2) voice recorded by the doctor and the medical staff, (3) voice recorded by the entire treatment room, (4) medical care Audio that records the alarm sound of the device.

  The device signal is an input / output signal of a medical electronic device associated with time. For the association, for example, a time stamp given to the signal is used. As a device signal at a certain time t, a signal having a time stamp closest to that time t can be used. The medical electronic device is, for example, a biological monitor, an artificial heart-lung machine, or an electric knife. The device signal of the heart-lung machine includes the signal indicating the operating status of the blood pump, the signal indicating the operating status of the heater, the signal indicating the temperature setting of the heater, the signal indicating the operating status of the heart-lung machine, the heart rate of the heart-lung machine And a signal indicating the operating status of the cardioprotective fluid injection device. In addition, the device signal of the electric scalpel includes a signal indicating the operating status, a signal indicating the amount of current flowing through the scalpel, a signal indicating the amount of current flowing through the electric scalpel, and a signal indicating the operating mode in use (for example, incision mode, or Signal indicating which mode of hemostasis mode is used). The device signal of the biological monitor includes a signal indicating various biological values measured by the biological monitor (for example, a signal indicating a pulse rate or a signal indicating an electrocardiogram waveform).

  Device signal supplementary information is attached to the device signal. The device signal supplementary information includes information indicating what device signal the device signal is and what the signal means. For example, the signal indicating the operating status of the heart-lung machine includes information indicating that this signal is a signal indicating the operating status of the oxygenator. The device signal supplementary information may be included in the input / output signal of the device from the beginning, or may be added by a device signal acquisition unit described later.

In the following description, a signal that expresses the operation status of each device / function as a binary value as in the above example is referred to as a device operation signal. The device operation signal is used by a determinable treatment process determination unit 151 described later. The device operating signal is, for example, a signal indicating the operating status of the oxygenator, a signal indicating the operating status of the blood pump, a signal indicating the operating status of the heater, a signal indicating the operating status of the cardioprotective fluid infusion device, It is a signal which shows the signal which shows an operation condition. A signal indicating output setting / control information of each device is referred to as a device setting signal. The device setting signal is used by a feature information extraction unit 153 described later. The device setting signal is, for example, a signal indicating the heater temperature setting, a signal indicating the heart rate of the heart-lung machine, a signal indicating the operation mode of the electric knife, a signal indicating the amount of current flowing through the electric knife, and / or measurement with a biological monitor. It is a signal showing various living body numerical values.

  It is assumed that the time stamps of the video, audio, and device signal are synchronized with the actual time during treatment.

(Configuration of treatment process estimation device)
FIG. 2 is a block diagram illustrating a configuration example of the treatment process estimation apparatus 1. As shown in FIG. 2, the treatment process estimation device 1 includes a video acquisition unit 11, an audio acquisition unit 12, a device signal acquisition unit 13, a treatment process table acquisition unit 14, a process information estimation unit 15, and a process information accumulation unit 16. .

  The image acquisition unit 11 acquires one or more images related to the treatment that is the target of the treatment process estimation, and supplies the acquired image to the process information estimation unit 15. It is necessary to acquire at least a video in which the classification indicated by the classification information is taken from the patient's surgical site and a video in which the medical device is placed as the video related to treatment. Furthermore, you may acquire the image | video regarding the treatment other than the above as needed.

  The voice acquisition unit 12 acquires one or more voices related to the treatment that is the target of the treatment process estimation, and supplies it to the process information estimation unit 15. As the voice related to the treatment, it is necessary to acquire at least a voice in which the classification indicated by the classification information records the voice of the doctor. Furthermore, you may acquire audio | voices other than the above as needed.

  The device signal acquisition unit 13 acquires one or more device signals of medical electronic devices related to the treatment that is the target of the treatment process estimation, and supplies the acquired device signal to the process information estimation unit 15. In the device signal acquisition unit 13, when the acquired device signal does not include the above-described device signal supplementary information, this signal is accompanied by device signal supplementary information indicating what device the signal is and what it means. .

  The treatment process table acquisition unit 14 acquires one treatment process table for the treatment to be estimated and supplies the acquired treatment process table to the process information estimation unit 15. The treatment process table includes at least a treatment process name, information on devices used in the treatment process, and information on electronic devices in the treatment process. The treatment process table may be acquired by the treatment process acquisition unit 14 that is directly input to the treatment process estimation apparatus 1 using a keyboard or a touch panel. Further, the treatment process table input to another apparatus may be acquired by the treatment process acquisition unit 14. The acquisition of the treatment process table may be performed before or after the treatment is actually performed.

  The process information estimation unit 15 estimates process information from the input video, audio, device signal, and treatment process table, and supplies the process information to the process information storage unit 16. Here, the process information is information including the start time and end time of the treatment process and the treatment process name for each treatment process described in the treatment process table. Details of the process information estimation unit 15 will be described later.

  The process information storage unit 16 stores the process information estimated by the process information estimation unit 15. The process information storage unit 16 includes a storage medium such as a hard disk, an optical disk, or a flash memory. The process information storage unit 16 may be connected via a network. The process information recorded in the process information storage unit 16 may be handled as index information for cueing or searching for a video when browsing a video related to a treatment action. Here, the video search may be a search by treatment process name or a search by time.

(Configuration of process information estimation unit)
Next, the process information estimation unit 15 included in the treatment process estimation apparatus 1 will be described. The process information estimation unit 15 estimates and outputs process information from the input treatment process table and video, audio and device signals. In the process information estimation, it is not always necessary to have all information of video, audio, and device signals.

  The detailed structure of the process information estimation part 15 is demonstrated using FIG. FIG. 1 is a functional block diagram illustrating a configuration example of the process information estimation unit 15. As illustrated in FIG. 1, the process information estimation unit 15 includes a determinable treatment process determination unit 151, a feature extraction auxiliary information acquisition unit 152, a feature information extraction unit 153, and a treatment process estimation unit 154.

[Determinable treatment process determination unit 151]
The determinable treatment process determining unit 151 obtains the start time and end time of the treatment process (determinable treatment process) that can be deterministically determined from the input treatment process table, video, audio, and device signal, that is, process information. The determined one or more pieces of process information are supplied to the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, and the treatment process estimation unit 154. A determinable treatment process is a treatment process that can be determined only by a device operation signal. Specifically, the first and last work is a treatment process in which each electronic device is operated. A specific example of the definable treatment process will be described with reference to the treatment process table of FIG. Among the treatment steps shown in FIG. 3, the treatment steps of “anesthesia”, “pericardiotomy”, “artificial cardiopulmonary operation”, “heart adhesion excision”, and “artificial cardiopulmonary arrest” are the first steps performed in the treatment step. The last work is the operation of the electronic devices. For this reason, the time at which the above operation is performed can be extracted from the device operation signal. Since the time at which the first and last work of each treatment process is performed can be taken out, the start time and end time of the treatment process can be determined.

  In the treatment steps of “skin incision”, “pericardial incision”, and “cardiac adhesion part excision” in the treatment step table of FIG. 3, electronic devices are used in the order of an electric knife cutting mode and an electric knife hemostasis mode. In this case, even if the device signals in the above order are detected, the order of the three treatment steps is not determined only by the device signals. Therefore, the order in which the treatment steps are performed is determined from the treatment step table and time information included in the device signal. In other words, when device signals are detected in the above order, the order of treatment steps described in the treatment step table of FIG. 3 and the time information of the device signals are referred to. It may be determined that the time of is “pericardiotomy” and the next time is “heart excision resection”. It should be noted that the treatment process of “skin incision” is not a definable treatment process because the first operation is not an operation of the electronic device.

  The operation of the determinable treatment process determination unit 151 described above can also be expressed as follows. That is, the definable treatment process determining unit 151 first determines a definable treatment process from the treatment processes included in the treatment process table based on the treatment process table. Subsequently, information including a start time and an end time and a treatment process name is derived for each determinable treatment process and output as process information.

[Feature extraction auxiliary information acquisition unit 152]
The feature extraction auxiliary information acquisition unit 152 extracts feature extraction auxiliary information from the input video, audio, and device signal based on the input process information of the definable treatment process and the treatment process table, and sends it to the feature information extraction unit 153. Supply. The feature extraction auxiliary information is used as auxiliary information when the feature information extraction unit 153 performs feature extraction from video, audio, and device signals. Details of the feature extraction auxiliary information and details of the derivation process will be described later.

[Feature information extraction unit 153]
The feature information extraction unit 153 is a feature information serving as a clue for estimating a treatment process based on input feature extraction auxiliary information and video, audio, device signal, treatment process table, and process information of a definable treatment process. Is taken out and supplied to the treatment process estimation unit 154. Details of the feature information extraction process will be described later.

[Treatment process estimation unit 154]
The treatment process estimation unit 154 estimates process information of a treatment process other than the definable treatment process based on the input feature information, the treatment process table, and the process information determined by the determinable treatment process determination unit, and the process information Are added to the process information determined by the determinable treatment process determination unit 151 and output from the process information estimation unit 15. Here, the treatment process other than the definable treatment process refers to a treatment process that is not determined by the definable treatment process determination unit 151 among the treatment processes described in the treatment process table. That is, the treatment process estimation unit 154 estimates process information for the remaining treatment steps obtained by removing the treatment steps already determined by the determinable treatment step determination unit 151 from the treatment steps described in the treatment step table.

  The treatment process estimation unit 154 estimates the treatment process for the time when the process information is not determined based on the process information determined by the determinable treatment process determination unit 151. That is, it is not necessary to estimate the process information for the time when the determinable treatment process determining unit 151 determines that the definable treatment process is being performed. Specifically, referring to FIG. 4, the time from 14:00 to 14:30 when the “anesthesia” treatment process, which is a determinable treatment process, was performed, and “heart”, which is also the definable treatment process, For the time from 14:55 to 16:50 when the treatment process from “membrane incision” to “artificial cardiopulmonary arrest” was performed, the treatment process estimation unit 154 does not need to estimate the treatment process.

  A specific order of process estimation will be described for the “skin incision” treatment process in the treatment process of FIG. 3. In the treatment process of “skin incision”, instruments are used in the order of a knife, tweezers, an electric knife cutting mode, and an electric knife hemostasis mode. For this reason, if it can judge from characteristic information that said instrument is used in said order, the treatment process of "skin incision" can be estimated. That is, in order, it is detected that the scalpel is used from the operating voice of the surgeon and the image of the operation part, similarly, it is detected that the tweezers are used, and the electric scalpel is used in the cutting mode from the device signal. If it is detected that the electric knife is used in the hemostatic mode, it is estimated that the treatment process of “skin incision” has been performed. In this case, it is assumed that the treatment process of “skin incision” is performed from the time when the scalpel starts to be used until the time when the use of the electric scalpel in the hemostatic mode is finished. It should be noted that not all instruments within one treatment process need to be detected. In terms of the above, for example, even if the use of tweezers is not detected, it may be determined that the treatment process of “skin incision” has been performed.

  Further, when the feature information that can exist at a certain time is limited from the process information already determined by the determinable treatment process determination unit 151 and the treatment process estimated by the treatment process estimation unit 154, the treatment process table, In the extraction of feature information, results that cannot be extracted may be excluded. For example, in the treatment process table of FIG. 3, when the end time of the treatment process of “artificial cardiopulmonary operation” and the start time of the treatment process of “artificial cardiopulmonary arrest” have already been determined, Only scissors, tweezers, and electric scalpels are used. When using the name of the instrument to be used as the feature information, it is possible to use only the above-mentioned instrument name as the feature information indicating the name of the instrument to be used at the time. good. Similarly, when the doctor's utterance content is used as the feature information, if the voice that the doctor can utter is limited, the other voice may not be treated as the feature information.

In estimating the treatment process, the treatment process may be estimated from the process information already determined by the determinable treatment process determination unit 151, the process information estimated by the treatment process estimation unit 154, and the treatment process table. For example, in the treatment process table of FIG. 3, when the end time of the treatment process of “artificial cardiopulmonary operation” and the start time of the treatment process of “artificial cardiopulmonary arrest” have already been determined, It may be determined that the treatment process of “cardiac adhesion resection” has been performed.

  Moreover, when there are two or more treatment steps using the device in the same order, the order of the treatment steps may be in accordance with the treatment order described in the treatment step table. For example, in the treatment process chart of FIG. 3, in the treatment process of “pericardial suture” and “skin suture”, the instruments are used in the same order. “Pericardial suture” may be judged as “skin suture”.

  Although not described here as an example, the process information determined by the determinable treatment process determination unit 151 is not supplied to the treatment process estimation unit 154, and the process of the determinable treatment process is performed by the treatment process estimation unit 154. Information may be determined and process information for the remaining treatment processes estimated and output. In this case, there is no need to add the process information determined by the determinable treatment process determination unit 151 and the process information estimated by the treatment process estimation unit 154, and the process determined and estimated by the treatment process estimation unit 154. The information may be output from the process information estimation unit 15.

(Operation flow of process information estimation unit)
The process information estimation unit 15 estimates process information according to the following procedure. First, in the determinable treatment process determining unit 151, process information of the definable treatment process is determined from the input device operation signal and the treatment process table, and the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, and the treatment process are determined. It supplies to the estimation part 154.

  Next, the feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input process information, treatment process table, video, audio, and device signal, and supplies the feature extraction auxiliary information to the feature information extraction unit 153.

  Next, in the feature information extraction unit 153, based on the input feature extraction auxiliary information, the treatment process table, and the process information of the definable treatment process, the feature information of the input video, audio, and device signal is extracted, and the treatment process It supplies to the estimation part 154.

  Next, in the treatment process estimation unit 154, based on the input feature information, the treatment process table, and the process information determined by the determinable treatment process determination unit 151, process information that has not been determined by the determinable treatment process determination unit 151 is obtained. Estimate and output.

  Finally, process information obtained by adding the process information determined by the determinable treatment process determination unit 151 and the process information of the treatment process estimation unit 154 is output from the process estimation unit 15, and the process information estimation is terminated.

  With reference to FIG. 4, the flow of the treatment process estimation in the process information estimation part 15 is demonstrated concretely. FIG. 4 is a diagram illustrating process information output from each component when the treatment process table of FIG. 3 is input to the process information estimation unit 15. In FIG. 4, the column of the determinable treatment process determination unit indicates process information output from the definable treatment process determination unit 151. The same applies to the column of treatment process estimation units and the column of process estimation units. Further, the column of the process name and the column of time included in each column indicate the treatment process name included in the process information output from the component, and the start time and end time of the treatment process.

In the process information estimation unit 15, first, the determinable treatment process determination unit 151 determines process information of the definable treatment process that can be determined by the device operation signal. That is, as shown in the column of the determinable treatment process determination unit in FIG. 4, treatment process names of “anesthesia”, “pericardiotomy”, “artificial cardiopulmonary operation”, “cardiac adhesion part excision”, and “artificial cardiopulmonary arrest” Then, the start time and end time of the treatment process are determined and output as process information. Next, feature extraction auxiliary information is detected by the feature extraction auxiliary information acquisition unit 152 for the process information determined above. Next, feature information is extracted by the feature information extraction unit 153 based on the feature extraction auxiliary information, the treatment process table, and the process information of the definable treatment process. Next, based on the feature information, the treatment process table, and the process information determined by the determinable treatment process determination unit 151, process information of the treatment process that has not been determined by the determinable treatment process determination unit 151 is estimated. That is, as shown in the column of the treatment process estimation unit in FIG. 4, the treatment process names of “disinfection”, “skin incision”, “stereotomy”, “pericardial suture”, “sternum suture”, “skin suture” The start time and end time of the treatment process are estimated and output as process information. As described above, in the treatment process estimation in the treatment process estimation unit 154, it is not necessary to estimate the process information for the time when the treatment process determined by the determinable treatment process determination unit 151 is performed. Finally, as shown in the column of the process estimation unit in FIG. 4, all treatments obtained by adding the process information determined by the determinable treatment process determination unit 151 and the process information estimated by the treatment process estimation unit 154 are added. Process information about the process is output from the process estimation unit 15. In FIG. 4, the start time and end time of the treatment process are shown in notation up to a minute unit, but the time is not limited to a minute unit.

(Details of feature extraction auxiliary information acquisition unit 152)
Next, details of the feature extraction auxiliary information acquisition unit 152 will be described. As described above, the feature extraction auxiliary information acquisition unit 152 extracts feature extraction auxiliary information from the input video, audio, and device signal based on the input process information and the treatment process table, and supplies the feature extraction auxiliary information to the feature information extraction unit 153. .

(Flow of feature extraction auxiliary information acquisition)
With reference to FIG. 5, the flow of feature extraction auxiliary information acquisition will be described. FIG. 5 is a diagram illustrating an example of an operation flow of the feature extraction auxiliary information acquisition unit 152. The feature extraction auxiliary information acquisition is sequentially executed in the following steps S151 to S155.

  (S151) From the process information determined by the determinable treatment process determination unit 151, one treatment process whose feature extraction auxiliary information is not detected is selected and set as the target treatment process. Next, S152 is executed.

  (S152) From the video, audio, and device signal information input to the feature extraction auxiliary information unit 152, one piece of information whose feature extraction auxiliary information has not been detected is selected and set as target information. Next, S153 is executed.

  (S153) Feature extraction auxiliary information is acquired for the target treatment process and target information. The feature extraction auxiliary information includes sample information and process setting information. Below, the method of sample information acquisition and processing setting information acquisition will be described in order. Note that acquisition of sample information and acquisition of processing setting information may be performed in parallel or sequentially. Moreover, when performing in order, whichever may be performed first. Moreover, only either one may be performed. Next, S154 is executed.

[Sample information acquisition]
For the target treatment process and target information, sample information is acquired and used as feature extraction auxiliary information. The sample information is information including a part of sample values of media data (video, audio, device signal) corresponding to a specific time or a specific period. The sample information handled by the process information estimation unit 15 refers to the feature extraction auxiliary information acquisition unit 152 at the time when the treatment process is performed or a time near the treatment process determined by the determinable treatment process determination unit 151. This is a collection of input treatment process tables and sample values of video, audio and device signals, and is referred to by the feature information extraction unit 153. The sample information can be used as information representing the surrounding situation at a specific time. The ambient situation is, for example, an instruction voice given by a doctor or the number of unused instruments. Even in different treatment steps, a situation can occur in which a particular ambient situation matches. In such a case, the correspondence between the surrounding situation observed in the treatment process that can be definitely determined and the sample information can be used when estimating another treatment process that occurs in the same surrounding situation. In the case of the process information estimation unit 15, when the feature information extraction unit 153 detects a sample value that is the same as or close to the sample information representing a specific surrounding situation, it is determined that the same surrounding situation has occurred. Thereby, the accuracy of feature extraction can be improved. A specific example of sample information acquisition is shown below.

[Specific example of sample information]
1. Color information of operation part image As an example of sample information, there is color information of operation part image. The treatment process of “artificial cardiopulmonary operation” in the treatment process table of FIG. 3 will be described as an example. Since the heart-lung machine is attached to the incised surgical part, the surgical part is incised when the treatment process of “artificial heart-lung operation” is performed. For this reason, the surgical site incised is recorded in the surgical site image when the treatment process is being performed. From here, when the treatment process of “artificial cardiopulmonary operation” is the target process, the surgical part image is set as the target information, and the color information is taken out as sample information. This sample information represents the surrounding situation where the surgical site is incised. Based on this information, in the feature information extraction unit 153, when color information that matches or is similar to the sample information is detected in the operation part image, the surrounding situation in which the operation part is incised occurs at that time. Judge. Similarly, when the “anesthesia” treatment process is being performed, the surgical site is closed. For this reason, the color information of the surgical part image when the treatment process is being performed is sample information indicating the surrounding situation where the surgical part is closed. Based on this information, in the feature information extraction unit 153, when color information that matches or is similar to the sample information is detected in the operation part image, an ambient situation in which the operation part is closed occurs at that time. Judge.

  At this time, the region of the surgical site can be determined by comparing the color information of the uncut surgical site image immediately after the start of surgery with the sample information. It can be said that the area where there is a large difference in color information between both images is the area where the incision is made and the body is exposed. In the case of processing an operation part video in the feature information extraction unit 153, the processing range is limited to the region by restricting the processing range to the region, and false detection is performed by excluding other regions from the processing target. You can suppress it. The above description can also be expressed as follows. That is, when a specific surrounding situation (incision situation of the surgical site in the above description) changes at two different times, the sample information at both times is compared, and a region where a certain difference or more occurs is compared. A subset of sample information affected by changes in ambient conditions can be identified.

2. Also, for example, an instruction voice requesting doctor's tweezers is used as sample information. That is, audio sample information is used as sample information indicating the surrounding situation where the doctor requests tweezers. The treatment process of “artificial cardiopulmonary arrest” described in the treatment process table of FIG. 3 is a treatment process determined by the determinable treatment process determination unit 151. From the treatment process table, it can be seen that tweezers are used in the treatment action of “pericardial suture” immediately after the treatment process of “artificial cardiopulmonary arrest”. Therefore, it can be determined that the sound that is recorded first after the end time of “artificial cardiopulmonary arrest” is the sound that the doctor requests tweezers. From here, when “artificial cardiopulmonary arrest” is the target process, the voice recorded with the doctor's instruction voice is the target information, and the voice information is detected first after the time when the target process is completed. A voice pattern is used as sample information. The sample information represents the instruction voice of the doctor described above. That is, since the voice pattern that the doctor requests tweezers is represented, when the voice information that matches or is similar to the sample information is detected in the feature information extraction unit 153, an ambient situation in which the doctor requests tweezers occurs. Judge that

[Get processing setting information]
For the target information regarding the target treatment process, process setting information is acquired and used as feature extraction auxiliary information. The process setting information is control information of a feature information extraction process in which the feature information extraction unit 153 extracts feature information from video, audio, and device signal information. The control information for the feature information extraction process includes parameters for image processing. The image processing parameter includes, for example, a threshold value in image binarization processing. The result of the feature information extraction process varies depending on the setting of the control information. In order to increase the accuracy of process estimation, it is necessary to perform effective feature extraction, that is, it is necessary to set control information that can extract effective features, but it is necessary to manually set all control information It is difficult, and even if the control information is set to an appropriate value, effective feature extraction is not always possible.

  Therefore, the optimal control information setting is detected using the treatment process table and the target treatment process. From the treatment process table, it is possible to determine the situation around the moment when the target treatment process is performed or a time near the target treatment process. For the target information at the above time, the same feature information extraction processing as that of the feature information extraction unit 153 is repeated while changing the setting of the control information. With respect to the plurality of feature information obtained in this way, the surrounding information determined from the treatment process table and the control information obtained by extracting the closest feature information are acquired as process setting information. By using the processing setting information obtained in this way, it is possible to extract features that are closer to reality, so that the accuracy of feature extraction can be improved.

  The process setting information acquisition will be described with a specific example. In the target treatment action, if a surgical instrument is used in the treatment action or before and after the treatment action, feature information is detected in the operation part image at the time of the treatment action or in the vicinity thereof Processing is performed, and the setting of control information for determining that the medical instrument has been used is taken out.

  For example, in the treatment process chart of FIG. 3, the instrument used immediately after the treatment process of “artificial cardiopulmonary operation” is scissors. Therefore, it can be determined that the instrument detected at the earliest time after the end of the treatment process of “artificial cardiopulmonary operation” is scissors. From here, for the time after the end time of the treatment process of “artificial cardiopulmonary operation”, it is possible to repeatedly perform image processing and device detection while changing the control settings, and the possibility that the device used at that time is scissors The control setting detected the highest is taken out and used as processing setting information used for scissors detection. An example of processing setting information acquisition is shown below.

[Specific example of processing setting information acquisition]
1. Acquisition of processing setting information for straight line detection The surgical instrument can be identified by various methods. Here, a method of identifying the surgical instrument from the surgical part image input to the feature extraction auxiliary information acquisition unit 152 will be considered. What is shown in the surgical part image is mainly the patient's body, doctor's hand, and surgical instrument. Of these, only the surgical instrument has a linear shape. Therefore, it is possible to detect whether or not a surgical instrument is used by determining whether or not a straight line exists in the surgical part image. The scalpel, scissors, and tweezers often used in surgery are all composed of two straight lines, and the configurations are parallel, crossed, and spread, respectively. Therefore, the type of surgical instrument can be identified by detecting the relative position of the two straight lines. In addition, although an instrument such as a thoracotomy device that is placed in the surgical site for a long time is also detected as a straight line, erroneous detection can be suppressed by excluding a straight line that does not move for a long time from the processing target.

For example, Hough transform is used for the straight line detection. In order to perform the Hough transform on the operation part image, it is necessary to perform binarization processing on the operation part image. The same applies to the feature information extraction unit 153. In this case, processing information settings include a binary threshold value in binarization processing, a threshold value for determining a straight line in Hough transform, and a conversion pixel pitch. Depending on these settings, the surgical instrument may not be correctly identified. Therefore, the Hough transform is repeatedly performed while changing the setting of the processing information to identify the instrument being used. All the results are compared, and the setting value of the processing information that gives the result that matches or is closest to the device used in the treatment process table is extracted as the processing setting information. By using this processing setting information in the feature information extraction unit 153, a detection result closer to the actual can be obtained.

(S154) It is determined whether or not feature extraction auxiliary information acquisition has been completed for all video, audio, and device signal information input to the feature extraction auxiliary information acquisition unit 152. If completed (S154 yes), S155 is executed. If it has not been completed (S154 no), the processing from the target information selection S152 is performed for information on video, audio, and device signals for which feature extraction auxiliary information has not been detected.

(S155) It is determined whether the detection of feature extraction auxiliary information has been completed for all the treatment steps determined by the determinable treatment step decision unit 151. If completed (S155 yes),
The feature extraction auxiliary information acquisition ends. If it is not completed (S155 no), processing is performed from the target process selection S151 for a treatment process for which feature extraction auxiliary information is not detected.

(Details of feature information extraction unit 153)
Next, details of the feature information extraction unit 153 will be described. As described above, the feature information extraction unit extracts feature information based on the input feature extraction auxiliary information and the video, audio, device signal, treatment process table, and process information of the definable treatment process, and estimates the treatment process. To the unit 154. Here, the feature information is information obtained by extracting features from video, audio, and device signals, and is used by the treatment process estimation unit 154 to estimate the treatment process. The feature information is extracted at each preset feature extraction time. However, it is not necessary to extract feature information at all times when samples of video, audio, and device signals exist. For example, the feature extraction time may be set to a time for each frame of the video, or may be set to a predetermined interval such as once every two seconds. In extracting feature information, it is not always necessary to extract all feature information. For example, when extracting feature information for a time when no sound is recorded, it is not necessary to extract feature information extracted from the sound (for example, a doctor's instruction sound described later). The same applies to video and device signals. Hereinafter, when a feature extraction process at a specific feature extraction time is described, the feature extraction time is referred to as a feature extraction target time.

  The feature information extraction unit 153 extracts feature information at each feature extraction target time and supplies it to the treatment process estimation unit 154. The feature information extraction unit 153 uses feature extraction auxiliary information for feature information extraction. However, it is not always necessary to use feature extraction auxiliary information for feature information extraction. Examples of feature information are shown below.

<Characteristic information 1: Incision status of surgical site>
As characteristic information, for example, there is an incision state of an operation part in surgery. That is, it is determined from the color information of the image of the surgical site that the surgical site is non-incised if there are many skin color components in the surgical site, and the surgical site is incised if there are many red components. Information is feature information.

  In the above determination, the sample information of the color information of the operation part image included in the feature extraction auxiliary information is used. This sample information includes sample information indicating the surrounding situation of the surgical site that has been cut and sample information indicating the surrounding situation of the closed surgical site. When the color information of the image obtained by photographing the surgical site at the feature extraction target time matches or is similar to the sample information indicating the surgical site that has been cut, it is determined that the surgical site has been cut. The same applies to the case where the information matches or resembles the sample information indicating the closed surgical site.

<Characteristic information 2: Instruction contents of doctor>
Further, for example, the instruction content detected based on the voice recording the doctor's utterance is used as the feature information. That is, when the surgeon instructs the assistant to deliver an instrument such as a scalpel, the instrument name is used as feature information. From this feature information, it can be estimated that the surgeon will use the instrument after giving instructions.

  In the above estimation, sample information of the doctor's instruction voice included in the feature extraction auxiliary information is used. This sample information indicates the ambient situation in which the doctor requests the instrument. When a voice pattern that matches or is similar to the sample information is detected from the voice recording the utterance of the doctor at the feature extraction target time, it is determined that the surrounding situation indicated by the sample information has occurred. That is, it is determined that an instrument indicated by the surrounding situation is requested.

  The feature information 1 and the feature information 2 can be collectively expressed as follows. That is, when sample information representing a specific surrounding situation is included in the feature extraction auxiliary information, the consistency or similarity is evaluated by comparing the sample information at the feature extraction target time with the sample information representing the surrounding situation. By doing so, it can be determined whether or not a certain surrounding situation has occurred at the feature extraction target time.

<Characteristic Information 3: Status of Medical Device Storage>
In addition, for example, the name of the appliance to be used estimated from the video of the medical device storage location is used as the feature information. That is, the number of remaining devices and the number of reduced devices are discriminated from an image of the medical device storage location, the device used at that time is estimated, and the device name is used as the feature information. The discrimination of the instrument may be performed by shape identification by pattern recognition or another method. The treatment process table describes the types and order of surgical instruments such as scalpels used for each treatment action. For this reason, if it is possible to follow what surgical instruments are used in order from the start of surgery, the current treatment process can be estimated from the types of surgical instruments in use and the number of surgical instruments used so far. . For this reason, it is possible to determine how many instruments have been used so far by checking the number of remaining instruments placed in the instrument storage area. The type of appliance used at that time is estimated from this information, and the appliance name is used as feature information.

<Characteristic information 4: Instruments used in the surgical department>
In addition, for example, the name of the tool used estimated from the video obtained by photographing the surgical site is used as the feature information. That is, an instrument used at the surgical site at each time is estimated from an image of the surgical site, and the instrument name is used as feature information. The instrument used may be identified by shape identification by pattern recognition or by line recognition.

  In the identification of an instrument by straight line recognition, processing setting information for straight line detection included in the feature extraction auxiliary information is used. This processing setting information includes setting information that can suitably identify the appliance by the straight line detection. For this reason, by applying straight line detection based on this processing setting information to a video obtained by photographing the surgical part at the feature extraction target time, it is possible to identify the instrument more accurately than when processing setting information is not used. It becomes possible.

<Feature Information 5: Usage Status of Electronic Equipment>
For example, the usage status of various electronic devices is used as the feature information. That is, the device setting signal is used as feature information. For example, when it is not cardiopulmonary arrest, a device setting signal whose determination criterion is not uniquely determined, such as a waveform shape of a heart rate displayed on a living body monitor, is used as feature information. However, it is only when there is an instruction to take out the waveform in the treatment process table.

  The feature information given as an example above is merely an example of feature information, and feature information other than the above may be used.

The feature information extraction unit 153 may extract the feature information only for the time when the process information is not determined based on the process information determined by the definable treatment process determination unit 151. That is, it is not necessary to extract feature information at the time when the determinable treatment process determining unit 151 determines that the definable treatment process is being performed. Thereby, the processing amount of the feature information extraction process can be reduced. Specifically, referring to FIG. 4, the time from 14:00 to 14:30 when the “anesthesia” treatment process, which is a determinable treatment process, was performed, and “heart”, which is also the definable treatment process, The time from 14:55 to 16:50 during which the treatment process from “membrane incision” to “artificial cardiopulmonary arrest” was performed may be excluded from the feature extraction target time.

  In addition, when the feature information that can be extracted at a certain time is limited from the treatment process determined by the determinable treatment process determination unit 151 and the treatment process table, the result that cannot be extracted in the feature information extraction is excluded. Also good. Feature information can be extracted more accurately by narrowing down the extraction target. For example, in the treatment process table of FIG. 3, when the end time of the treatment process of “artificial cardiopulmonary operation” and the start time of the treatment process of “artificial cardiopulmonary arrest” have already been determined, Only scissors, tweezers, and electric scalpels are used. When using the name of the instrument to be used as the feature information, it is possible to use only the above-mentioned instrument name as the feature information indicating the name of the instrument to be used at the time. good.

  As described above, information indicating feature information that can be extracted at each time is referred to as feature extraction restriction information. The feature extraction restriction information may be detected at any time when video, audio, and device signal samples exist, may be detected at each frame time of the video, or may be detected at a predetermined time such as once every 2 seconds. It may be detected at intervals. Moreover, you may detect according to the feature extraction object time.

  Although not described here as an example, the process information determined by the determinable treatment process determination unit 151 is not supplied to the feature information extraction unit 153, and the device operation signal is determined by the feature information extraction unit 153. The determinable treatment process and the time when the determinable treatment process is determined to be performed may be detected.

(Effect of process information estimation unit)
The process information estimation unit 15 described above discriminates a determinable treatment process from the input treatment process table, video, audio, and device signal, and extracts feature extraction auxiliary information composed of sample information and process setting information therefrom. In addition, it has a function of extracting feature information from video, audio, and device signals input using the information and estimating a treatment process using the feature information.

  By using sample information for feature information extraction, what the extracted feature information means and what is its criterion can be extracted from the information recorded in the actual treatment. Compared with a certain prior art, feature information can be used more effectively.

  By using processing setting information for feature information extraction, it is possible to use more suitable control settings compared to the prior art that requires preset control settings for feature information extraction, enabling more effective feature information extraction. Become.

  For this reason, more effective feature information extraction can be performed by using feature extraction auxiliary information for feature information extraction as compared with a technique of extracting feature information from conventional video, audio, and device signals and estimating a treatment process. Accordingly, it is possible to estimate the treatment process with higher accuracy.

[Appendix 1]
In the description of the treatment process estimation apparatus 1 described above, the apparatus has been described as deriving all treatment processes defined by the treatment action. However, it is not always necessary to derive all treatment processes, and only some treatment processes are derived. It may be a deriving device.

[Appendix 2]
In the description of the treatment process estimation apparatus 1 described above, the apparatus has been described on the assumption that the process information of the treatment action is estimated with respect to the media data related to the treatment action that has been recorded in advance. There is no need to deal with it, and it may be a device that estimates process information of the treatment action in parallel with recording of media data related to the treatment action.

  In this case, when the process information estimation unit 15 estimates process information at a certain time, media data at a time earlier than that time is not referred to. That is, media data at a future time is not used for the process information estimation. Specifically, in the feature extraction auxiliary information unit 152, when acquiring feature extraction auxiliary information, only media data at a time earlier than the current time is referred to. In this case, the feature extraction auxiliary information may be acquired again at an arbitrary timing.

In addition, when the process information is estimated by the treatment process estimation unit 154 at a certain time, media data at a future time than that time is not referred to. In this case, unlike the description of the treatment process estimation apparatus 1 described above, it is not necessary to estimate the end time of the treatment process only in the treatment process estimated to be performed at present, and the treatment information includes the treatment process. It is not necessary to include the end time. The same applies to the determination of the process information of the definable treatment process in the definable treatment process determination unit 151. However, a treatment process whose end time is not fixed is not treated as a definable treatment process because the treatment process may not be determined definitely. A specific description will be given with reference to the process information of FIG. At 14:46, when the treatment process estimation unit 154 estimates the treatment process, if the end time of the treatment process of “disinfection” is estimated, the treatment process to be performed next is “skin” It can be determined that this is a treatment process for incision. If the use of the scalpel is detected from the feature information, it is estimated that the “skin incision” treatment process is being performed.

  In parallel with the recording of the media data related to the treatment action, when estimating the process information of the treatment action, the name and details of the treatment process presumed to be currently performed (name, order of equipment used, etc.) If performed, the name and details of the treatment process described in the treatment process table may be presented to doctors and medical staff using video and audio. In the above description, the current treatment process is estimated. However, it is not always necessary to estimate process information for the current treatment process, and only the treatment process that can estimate the start time and end time of the treatment process. The process information may be estimated.

  By estimating the process information of the treatment action in parallel with the recording of the media data regarding the treatment action, the process information estimation of the treatment action can be completed simultaneously with the completion of the treatment action. That is, the process information can be output at an earlier time than when the treatment process is estimated after the recording of the treatment action is completed. In addition, by presenting the current treatment process to doctors and medical staff, the progress of treatment can be confirmed. Similarly, by presenting information on a treatment process to be performed next, the treatment action to be performed next can be confirmed even if the treatment process table is not at hand.

[Appendix 3]
In the description of the treatment process estimation apparatus 1 described above, the process information of the treatment action is described as being estimated, but the target of the process estimation is not necessarily limited to the treatment action. That is, the apparatus which estimates process information about the action which is not a treatment action may be sufficient. Specifically, for a certain action that is not a treatment action, there is media data related to the action, and a process chart for the process (process name, details of equipment used in the process, electronic information in the process, and information equivalent to the treatment process chart) If there is a device detail), the process information regarding the action may be estimated in the same manner as the process information of the treatment action. An example of an action that is not a therapeutic action is shown below.

An act that is not a therapeutic action is, for example, a stage such as a theater or a live music. In general, a process chart in which the order and details of processes are determined in advance is created for the stage. Further, since the details of the use of the electronic device (speaker, lighting, etc.) are also described in the process table, the process information can be estimated in the same manner as the process information of the treatment action. By using the process information estimated for the stage, it is easy to view the video image of the stage. As a result, for example, videos obtained by shooting the stage can be classified for each process (scene) and seeked to an arbitrary scene.

  Another example is product assembly. However, it is limited to those using electronic equipment. In general, in the assembly of a product, since the order and details of the assembly process are determined, the process information can be estimated in the same manner as the process information of the treatment action. By using the process information estimated for the product assembly, it is easy to view a video image of the product assembly. Thus, for example, by adding the name of the work being performed at each time to the video of the product assembly, it can be used as an educational video. Moreover, the working time of each process can be measured.

[Appendix 4]
In the description of the treatment process estimation apparatus 1 described above, the apparatus has been described on the assumption that the process information is information including the start time and end time of the treatment process, and the treatment process name. An apparatus including information may be used. Here, the importance level information is information indicating the importance level of the media data at each time when the process information is estimated. The importance level information may be set for each time when the process information is estimated, may be set for each work in the treatment process, or may be set for each treatment process.

If the importance level information is described in advance in the treatment process chart, it may be determined with reference to the information, or may be derived from the media data. For example, in the treatment process chart of FIG. 3, from the device operation signal, the time from the completion of the operation of the heart-lung machine to the start of the stop operation (from the end time of the treatment process of “artificial heart-lung operation” to “artificial heart-lung stop” Until the start of the treatment process)
It is determined that an important scene is reflected in the video taken of the surgical part, and the importance of the time is set high. Moreover, you may derive | lead-out from the estimation result of a treatment process. For example, a treatment process with a long working time is set as important.

  In the case where importance information is included in the process information, the process information stored in the process information storage unit 16 may be an index for performing video search based on importance. Note that importance level information does not necessarily need to be included in the process information of all times, operations, or treatment processes, and importance information may be included in only part of the process information. In addition, individual importance information may be set for each media data, or common importance information may be set for all media data.

  By including importance level information in the process information, video search based on importance level is possible, and video confirmation can be simplified.

[Appendix 5]
In the description of the treatment process estimation apparatus 1 described above, the process information is estimated for the treatment process. However, in the apparatus for estimating the process information for the work (use of instruments, use of electronic devices) constituting the treatment process. There may be. That is, the start time and the end time are estimated for each work constituting the treatment process, and the process information may be combined with the work name information.

The work process information is estimated by the process information estimation unit 15 in the same manner as the process information of the treatment process. The work that constitutes the treatment process includes work that can be definitely determined (determinable work), as in the treatment process. Here, the determinable work is a work that uses an electronic device, and is a work in which process information can be determined by a device operation signal. For example, referring to the treatment process chart of FIG. 3, the operations constituting the treatment process of “skin incision” are a scalpel, tweezers (gauze), an electric knife cutting mode, and an electric knife hemostasis mode. Among these, the operations of the electric knife cutting mode and the electric knife hemostasis mode are operations using an electronic device, and since the start time and the end time of the operation can be determined definitely by the device operation signal, they are definable operations. Hereinafter, estimation of work process information will be described.

  When estimating work process information, first, a determinable treatment process determining unit 151 determines determinable work process information. That is, the process information is determined for an operation that can determine the process information from the equipment operation signal. At this time, even if the work constitutes a treatment process that is not a definable treatment process, the process information is determined if the work is a definable work. For example, in the treatment process chart of FIG. 3, the treatment process of “skin incision” is not a definable treatment process, but a process for definable work (electric knife cutting mode, electric knife hemostasis mode) included in this treatment process. Determine information.

  Next, the treatment process estimation unit 154 estimates process information of work for which process information has not been determined based on the feature information extracted by the feature information extraction unit 153. That is, the process information of work that is not definable work is estimated. At this time, similarly to the estimation of the process information of the treatment process, it is not necessary to estimate the process information of the work at the time when the determinable treatment process determination unit 151 determines that the definable work is being performed. Similarly to the estimation of the process information of the treatment process, the process information of the work may be estimated from the process information of the work already determined or estimated and the treatment process table.

  When estimating work process information, the process information stored in the process information storage unit 16 may be an index for performing a video search based on a work name. Note that it is not always necessary to derive process information for all operations, and only process information for some operations may be derived.

  By including process information related to work in the process information, it is possible to search for a video by work name, and to check the video more easily.

[Modification 1]
In the description of the treatment process estimation device 1 described above, it is assumed that the process information of the treatment action is estimated using only information related to the treatment action (estimation target treatment action) that is the target of the process information estimation. It may be a device that estimates process information of a treatment action using information on a treatment action (non-estimated target treatment action) other than the treatment action. That is, in the estimation of the treatment process information of the estimation target treatment action, the feature information extraction auxiliary information regarding one or more non-estimation target treatment actions is referred to when extracting the feature of the estimation target treatment action, and the feature information is extracted. . Then, the treatment process estimation may be performed using the feature information. The non-estimated target treatment action is, for example, a treatment action performed in the past. In this case, the process information estimation unit 15 is additionally provided with a recording unit that records the estimated process information and the feature extraction auxiliary information used at the time of derivation. For example, it is the treatment action performed in parallel with the estimation object treatment action.

  The feature extraction assistance information of the non-estimated treatment action may be information acquired and accumulated when the process information of the non-estimation treatment action is estimated in the past, or the feature extraction assistance of the estimation object treatment action When acquiring information, you may acquire from the treatment process table | surface and media data regarding a non-estimation object treatment action.

  Even if feature extraction assistance information cannot be obtained or is difficult to obtain, such as when the device operation signal is not included in the media data related to the estimation target treatment action, as described above, feature extraction assistance for the non-estimation target treatment action By using the information, more effective feature information extraction can be performed. Accordingly, it is possible to estimate the treatment process with higher accuracy.

[Modification 2]
In the description of the treatment process estimation apparatus 1 described above, it is described that the estimated process information is stored in the process information storage unit 16, but the process information is not necessarily stored in the process information storage unit 16, and the process information is not stored. An apparatus that outputs from the treatment process estimation apparatus 1 may be used.

  In this case, the treatment process estimation apparatus 1 does not necessarily require a storage medium constituting the process information storage unit 16. Further, it is possible to output the estimated process information without passing through the process information storage unit 16.

[Modification 3]
A detailed configuration of the feature information extraction unit 153 will be described with reference to FIG. FIG. 6 is a functional block diagram illustrating a configuration example of the feature information extraction unit 153. As illustrated in FIG. 6, the feature information extraction unit 153 includes a feature extraction target time setting unit 1531, a feature extraction restriction information setting unit 1532, a media data processing unit 1533, a media data comparison unit 1534, and a feature information acquisition unit 1535.

  The feature extraction target time setting unit 1531 sets the above-described feature extraction target time based on the input process information of the definable treatment process and the treatment process table, and uses the information as the media data processing unit 1533 and the media data comparison To the unit 1534.

  The feature extraction restriction information setting unit 1532 sets the above-described feature extraction restriction information based on the input process information of the definable treatment process and the treatment process table, and uses the information as the media data processing unit 1533 and the media data comparison To the unit 1534.

  The media data processing unit 1533 performs processing such as image processing on the input video, audio, and device signal based on the feature extraction auxiliary information and the feature extraction target time information including the input processing setting information, and the processing result Is supplied to the feature information acquisition unit 1535. Here, processing such as image processing is processing performed on media data in order to extract feature information. For example, binarization, Hough transform, pattern recognition, Also, it is binarization and pattern recognition for a frame of an image obtained by photographing the device place. The above is merely an example, and it is possible to perform processing on not only video but also audio and device signals and extract processing results. The settings and parameters of the various processes described above are set in advance before the treatment process estimation is performed. However, when the above-described process setting information is included in the feature extraction auxiliary information input to the media data processing unit 1533, the process setting information is reflected in the settings and parameters of each process. It is not always necessary to reflect all processing setting information. The processing result is accompanied by time information of the media data to be processed.

The media data comparison unit 1534 compares input video, audio, and device signal information with reference sample information based on feature extraction auxiliary information and feature extraction target time information including input sample information, and compares the result. Is supplied to the feature information acquisition unit 1535. The reference sample information is, for example, color information of the surgical site when the surgical site is incised with respect to an image obtained by imaging the surgical site. Compare the color information of this reference sample information with the color information of the video of the operative part contained in the media data, and use the numerical results to indicate how similar they are. Based on this, feature information is determined. In this case, in the feature information acquisition unit 1535, if the comparison results are similar, the feature information is set assuming that the surgical site is incised. If they are not similar, it is not necessary to set the feature information because the operation part is closed and the feature information is set, or it is not similar to the reference sample information. The reference sample information described above is set in advance before the treatment process estimation is performed. However, if the above-described sample information is included in the feature extraction auxiliary information input to the media data processing unit 1533, the sample information is set as the reference sample information. Otherwise, the specified value is set as the reference sample information. To do. Note that it is not always necessary to reflect all reference sample information. The comparison result is accompanied by time information of the media data to be compared.

  The feature information acquisition unit 1535 determines feature information based on the input feature extraction restriction information, the processing result, and the comparison result, and outputs the feature information from the feature information extraction unit 153.

  When the feature extraction auxiliary information includes only the processing setting information, the media data comparison unit 1534 may not receive the feature extraction auxiliary information and may put a specified value in the reference sample information. Further, when only the sample information is included in the feature extraction auxiliary information, the media data processing unit 1533 does not receive the feature extraction auxiliary information, and the processing setting may be set as the specified value.

[Modification 4]
In the description of the treatment process estimation apparatus 1 described above, it is assumed that the process information estimation unit 15 includes the determinable treatment process estimation unit 151, the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, and the treatment process estimation unit 154. However, this configuration is not necessarily required, and the process information estimation unit 15 is configured by replacing the feature information extraction unit 153 and the treatment process estimation unit 154 with a treatment process estimation unit 153a that integrates these functions. It may be a device.

The function of the treatment process estimation unit 153a will be described with reference to FIG. FIG. 7 is a diagram showing a modification of the process information estimation unit 15. As illustrated in FIG. 7, the process information estimation unit 15 may include a determinable treatment process estimation unit 151, a feature extraction auxiliary information acquisition unit 152, and a treatment process estimation unit 153a. The functions of the definable treatment process estimation unit 151 and the feature extraction auxiliary information acquisition unit 152 are as described above. The treatment process estimation unit 153a extracts feature information based on the input feature extraction auxiliary information and the video, audio, device signal, treatment process table, and process information of the definable treatment process. Furthermore, the process information of the treatment process other than the definable treatment process is estimated based on the feature information, the treatment process table, and the process information of the definable treatment process, and the process information is determined by the determinable treatment process determining unit 151. The process information is added and output from the process information estimation unit 15. The feature information extraction in the treatment process estimation unit 153a is performed in the same manner as the feature information extraction in the feature information extraction unit 153 described above. In addition, the treatment process estimation in the treatment process estimation unit 153a is performed in the same manner as the treatment process estimation unit 154 described above.

  In the description of the treatment process estimation device 1 described above, it is described that the process information is estimated after extracting the feature information for all the feature extraction target times. There is no need to comply, and feature information extraction and process information estimation may be performed in parallel.

[Example of software implementation]
The treatment process estimation apparatus 1 (particularly the process information estimation unit 15) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or software using a CPU (Central Processing Unit). It may be realized by.

In the latter case, the treatment process estimation apparatus 1 includes a CPU that executes instructions of a program that is software that realizes each function, and a ROM (Read Only Memory) in which the program and various data are recorded so as to be readable by a computer (or CPU). ) Or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via any transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

DESCRIPTION OF SYMBOLS 1 Treatment process estimation apparatus 11 Image | video acquisition part 12 Audio | voice acquisition part 13 Apparatus signal acquisition part 14 Treatment process table | surface acquisition part 15 Process information estimation part 151 Determinable treatment process determination part 152 Feature extraction auxiliary information acquisition part 153 Feature information extraction part 153a Treatment Process estimation unit 1531 Feature extraction target time setting unit 1532 Feature extraction restriction information setting unit 1533 Media data processing unit 1534 Media data comparison unit 1535 Feature information acquisition unit 154 Treatment process estimation unit 16 Process information storage unit

Claims (9)

A treatment process estimation apparatus for determining process information related to one or more treatment processes,
A media data acquisition unit for acquiring one or more media data related to the treatment process;
A treatment process table acquisition unit for acquiring a treatment process table indicating the treatment process;
A determinable treatment process determination unit that determines process information regarding a determinable treatment process based on at least the media data and the treatment process table;
An auxiliary information acquisition unit that acquires auxiliary information based on at least process information relating to the definable treatment process, the media data, and the treatment process table;
A treatment process estimation apparatus comprising: a treatment process estimation unit that estimates the process information based on at least process information on the determinable treatment process, the media data, the treatment process table, and the auxiliary information.   The treatment process estimation apparatus according to claim 1, wherein the auxiliary information is sample information.   The treatment process estimation apparatus according to claim 1, wherein the auxiliary information is processing setting information.   3. The treatment process estimation apparatus according to claim 2, wherein the sample information is information that can be acquired from an image of an operation site based on at least process information relating to the definable treatment process and the treatment process table.   3. The sample information according to claim 2, wherein the sample information is information that can be acquired from voice information in which an instruction during treatment is recorded based on at least step information on the definable treatment step and the treatment step table. Treatment process estimation device.   The said treatment process estimation part estimates the said process information based on the identification result which identified the medical device used in the result of the straight line detection process based on the said process setting information, and an image | video. The treatment process estimation apparatus according to 1. The determinable treatment process determination unit determines the process information of each operation constituting the treatment process,
The treatment process estimation apparatus according to claim 1, wherein the treatment process estimation unit estimates the process information of each work constituting the treatment process.   The treatment process estimation unit uses at least the treatment process table and process information relating to the definable treatment process, and includes information on the type and number of medical devices detected for the time when the process information is not estimated. The treatment process estimation apparatus according to claim 1, wherein the process information is estimated based on the information. When the process information of the treatment process before and after the unestimated treatment process is estimated or determined, the treatment process estimation unit,
The treatment according to claim 1, wherein information on an unestimated treatment process is estimated based on at least the treatment process table, process information on the definable treatment process, and the estimated process information. Process estimation device.

Images