WO2017068853A1 - Medical treatment process estimation device - Google Patents

Medical treatment process estimation device Download PDF

Info

Publication number
WO2017068853A1
WO2017068853A1 PCT/JP2016/074579 JP2016074579W WO2017068853A1 WO 2017068853 A1 WO2017068853 A1 WO 2017068853A1 JP 2016074579 W JP2016074579 W JP 2016074579W WO 2017068853 A1 WO2017068853 A1 WO 2017068853A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
treatment process
treatment
determinable
estimation
Prior art date
Application number
PCT/JP2016/074579
Other languages
French (fr)
Japanese (ja)
Inventor
恭平 池田
山本　智幸
久雄 熊井
Original Assignee
シャープ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by シャープ株式会社 filed Critical シャープ株式会社
Publication of WO2017068853A1 publication Critical patent/WO2017068853A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H30/00—ICT specially adapted for the handling or processing of medical images
    • G16H30/40—ICT specially adapted for the handling or processing of medical images for processing medical images, e.g. editing
    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H20/00—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
    • G16H20/40—ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture

Definitions

  • One embodiment of the present invention relates to a treatment process estimation apparatus that detects a treatment process at each time during treatment using the acquired video, audio, and device information related to the treatment.
  • Patent Document 1 proposes a technique for tagging important parts of the video of the treatment action and simplifying the confirmation of the video. For example, a technique is described that estimates a treatment process based on position information of a surgical instrument or an IC tag attached to a doctor, and reproduces an image for each estimated treatment process.
  • the treatment process at each time is estimated by following the transition of the treatment process according to a standard treatment program (such as ACLS or JATEC) in which specific contents of the treatment process and transition conditions are defined.
  • a standard treatment program such as ACLS or JATEC
  • the technique which reproduces an image for every treatment process based on the presumed information is described.
  • Patent Literature 1 and Patent Literature 2 can estimate the treatment process only for the therapeutic action that satisfies the specific condition.
  • Patent Document 1 when an IC tag is used to estimate a treatment process, a special device with an IC tag and an IC tag reader are required, so a video of a treatment action in which the device is not used There is a problem that the treatment process can not be estimated.
  • Patent Document 2 when estimating the treatment action according to the standard treatment program, there is a problem that the treatment process can not be estimated for the image of the treatment action which does not correspond to the standard treatment program.
  • One embodiment of the present invention has been made in view of the above circumstances, and its object is to provide a medical information processing apparatus capable of estimating a treatment process without using a special device such as an IC tag or a standard treatment program. To realize.
  • a treatment process estimating device concerning one embodiment of the present invention is a treatment process estimating device which determines process information concerning one or more treatment processes, and one or more about the treatment process concerned Media data acquisition unit for acquiring media data of a medical treatment process, a treatment process schedule acquisition unit for acquiring a treatment process schedule indicating the treatment process, and process information on a treatment process that can be determined based on at least the media data and the treatment process schedule Determining an determinable treatment process, an auxiliary information acquiring unit acquiring an auxiliary information based on at least process information on the determinable treatment process, the media data, and the treatment process chart, and at least the determinable treatment The process information is estimated based on process information on the process, the media data, the treatment process chart and the auxiliary information. And a treatment step estimator for.
  • a treatment process that can be determined definitively is determined from the inputted treatment process chart and video, audio, and device signal, and auxiliary information (sample information and treatment setting information) is determined therefrom. It is possible to provide a function of extracting feature information (also referred to as feature extraction auxiliary information), extracting feature information from the input video, audio, and device signal using the information, and performing treatment process estimation using the feature information.
  • extracting feature information also referred to as feature extraction auxiliary information
  • sample information for feature information extraction, it is necessary to pre-set what the extracted feature information means, and also for the judgment criteria, because it can be taken out from the information that records the actual treatment.
  • the feature information can be used more effectively than certain prior art.
  • process setting information for feature information extraction enables more effective feature information extraction because it is possible to use optimal control settings as compared to the prior art in which control settings for feature information extraction need to be set in advance. .
  • FIG. 2 is a block diagram showing a configuration example of the treatment process estimation apparatus 1 in the present embodiment.
  • the treatment process estimation apparatus 1 is an apparatus for deriving and recording process information on the basis of a treatment process chart and media data (video, audio, device signal).
  • video, audio, device signal video, audio, device signal
  • the treatment schedule is a table including the order of treatment steps and detailed information on each treatment step for a plurality of treatment steps performed from the start to the end of treatment treatment.
  • the detailed information on the treatment process includes the treatment process name, information on the equipment used in the treatment process (for example, use order and type), and information on the electronic device in the treatment process (for example, use order and type).
  • FIG. 3 is an example of a treatment process chart of cardiac adhesion resection operation.
  • the treatment schedule includes a row of treatment step names, instruments used, and electronic devices.
  • the treatment step name represents the name of each treatment step. As shown in FIG.
  • treatment process names "treatment start”, “anesthesia”, “disinfection”, “skin incision”, “sternal incision”, “pericardio incision”, “artificial cardiopulmonary bypass” “Operation”, “heart adhesion ablation”, “articardiopulmonary arrest”, “pericardial suture”, “sternal suture”, “skin suture”, “end of treatment” are described in order, and this description order is the treatment process Match the order.
  • the row of instruments used indicates the name and order of use of medical devices used in the treatment process such as a scalpel and tweezers
  • the row of electronic devices indicates the name and order of use of electronic devices used in the treatment process.
  • the use of the electronic device is performed in the order from the top item in the table.
  • the use order of the use apparatus or electronic device in one treatment process becomes the order of description from the top of the treatment process chart regardless of the use apparatus and the electronic device.
  • the use order in the process is in the order of a scalpel, tweezers, an electric scalpel cutting mode, and an electric scalpel hemostasis mode.
  • anesthesia an anesthetic injection operation by an electronic device is shown.
  • work using tweezers is shown.
  • FIG. 3 is an example of a treatment process chart, and items such as scheduled work times, workers, and patient status may be added as detailed information on each treatment process.
  • the treatment process schedule which can be used as an input of the treatment process estimation apparatus 1 of this embodiment does not necessarily need to be a format of a table as shown in FIG. 3, and the order of the above-mentioned treatment process and a treatment process If detailed information of can be derived, it may be in another format. For example, data indicating the order of treatment steps and data for reading out detailed information of each treatment step may be read and used in combination.
  • the video, audio, and device signals are signals sent from various devices regarding video, audio, and device control information related to treatment.
  • a video is composed of a plurality of frames associated with time. For association, for example, a time stamp given to each frame is used. As a video at a certain time t, a frame having a timestamp closest to that time t can be used.
  • Video incidental information is attached to the video.
  • the video incidental information includes classification information indicating a video classification according to the imaging target.
  • the classification information is, for example, information for identifying the following video. (1) Video taken of the patient's surgical site, (2) Video taken of the storage area of the medical device, (3) Video taken of the whole or part of the treatment room, (4) Operation of the medical staff assisting the doctor or An image of the device at hand, (5) An image of the doctor's expression and eyes, an image of the doctor's eye with a camera worn by the doctor, and (6) Medical equipment such as a biological monitor related to treatment (7) Images of the operation of medical equipment such as artificial heart and heart and the operation panel of the equipment, (8) Images of endoscopic camera and microscope, (9) Excised from patient Of human body tissue and its storage area.
  • Voice is composed of a plurality of frames associated with time. For association, for example, a time stamp given to each frame is used.
  • a voice at a certain time t when a voice signal is detected at the time t, a voice signal from a frame in which the voice signal starts to be detected to a frame in which the voice signal is not detected can be used.
  • Audio incidental information is attached to the audio.
  • the audio incidental information includes classification information indicating audio classification according to the recording target.
  • the voice classification is, for example, information for identifying the following voice. (1) voice recording doctor's voice, voice recording patient's voice, (2) voice recording doctor's and medical staff's conversation, (3) voice recording voice throughout the treatment room, (4) medical care Voice recording the alarm sound of the device.
  • the device signal is an input / output signal of the medical electronic device associated with the time. For the association, for example, a time stamp given to the signal is used. A signal having a time stamp closest to time t can be used as a device signal at time t.
  • the medical electronic devices are, for example, a living body monitor, a heart-lung machine, and an electrosurgical knife.
  • the device signals of the heart-lung machine include a signal indicating the operating state of the blood pump, a signal indicating the operating state of the heater, a signal indicating the temperature setting of the heater, a signal indicating the operating state of the heart-lung machine, and the heart rate of the heart-lung machine. And a signal indicating the operation status of the cardioprotective fluid injection device.
  • the instrument signal of the electric scalpel includes a signal indicating an operation status, a signal indicating an amount of current flowing to the scalpel, a signal indicating an amount of current flowing to the electric scalpel, and a signal indicating an operation mode in use (for example, a cutting mode or There is a signal that indicates which mode of hemostasis mode is being used.
  • the device signal of the living body monitor includes a signal (for example, a signal showing a pulse rate or a signal showing an electrocardiogram waveform) indicating various living body values measured by the living body monitor.
  • Equipment signal incidental information is attached to the equipment signal.
  • the device signal additional information includes information indicating what device the device signal is and which signal means the device signal.
  • the signal indicating the operation status of the heart-lung machine includes information indicating that the signal is a signal indicating the operation status of the heart-lung machine.
  • the device signal incidental information may be included in the input / output signal of the device from the beginning or may be added by the device signal acquisition unit described later.
  • a signal representing the operating status of each device / function in binary as in the above example will be referred to as a device operating signal.
  • the device operation signal is used by the below-described determinable treatment process determination unit 151.
  • the device operation signal is, for example, a signal indicating the operating condition of the artificial heart-lung machine, a signal indicating the operating condition of the blood pump, a signal indicating the operating condition of the heater, a signal indicating the operating condition of the cardioprotective solution injection device, an electric scalpel It is a signal which shows the signal which shows an operating condition.
  • a signal indicating output setting / control information of each device is called a device setting signal.
  • the device setting signal is used by a feature information extraction unit 153 described later.
  • the instrument setting signal is, for example, a signal indicating the temperature setting of the heater, a signal indicating the heart rate of the artificial heart and lung, a signal indicating the operation mode of the electric scalpel, a signal indicating the amount of current flowing to the electric scalpel and / or measurement with a living body monitor Is a signal representing various biometric values.
  • time stamps of the video, audio and device signals are synchronized with the actual time of treatment.
  • FIG. 2 is a block diagram showing a configuration example of the treatment process estimation apparatus 1.
  • the treatment process estimation apparatus 1 includes a video acquisition unit 11, an audio acquisition unit 12, a device signal acquisition unit 13, a treatment process schedule acquisition unit 14, a process information estimation unit 15, and a process information storage unit 16. .
  • the image acquisition unit 11 acquires one or more images related to the treatment to be a target of treatment process estimation, and supplies the image to the process information estimation unit 15.
  • a video relating to treatment it is necessary to obtain at least a video of a classification indicated by the classification information obtained by photographing the operation part of the patient and a video obtained by photographing the place of the medical device. Furthermore, you may acquire the image regarding treatment other than the above as needed.
  • the voice acquisition unit 12 acquires one or more voices related to the treatment to be a target of the treatment process estimation, and supplies the one or more sounds to the process information estimation unit 15.
  • the voice related to the treatment it is necessary to obtain at least the voice in which the classification indicated by the classification information records the doctor's utterance. Furthermore, voices other than the above may be acquired as needed.
  • the device signal acquisition unit 13 acquires one or more device signals of the medical electronic device related to the treatment to be subjected to the treatment process estimation, and supplies the one or more device signals to the process information estimation unit 15.
  • the device signal acquiring portion 13 attaches the device signal incidental information indicating what device the signal is and what it means. .
  • the treatment process chart acquisition unit 14 acquires one treatment process chart of the treatment to be estimated, and supplies the process process estimation unit 15 with it.
  • the treatment schedule includes at least a treatment step name, information on an apparatus used in the treatment step, and information on an electronic device in the treatment step.
  • the treatment process schedule may be acquired by the treatment process acquisition unit 14 that has been directly input to the treatment process estimation apparatus 1 by a keyboard or a touch panel. Further, the information on the treatment process chart input to another device may be acquired by the treatment process acquisition unit 14. Acquisition of the treatment schedule may be performed before or after the treatment operation is actually performed.
  • the process information estimation unit 15 estimates the process information from the input video, audio, device signal, and treatment process chart, and supplies the process information to the process information storage unit 16.
  • the process information is information including the start time and end time of the treatment process and the treatment process name for each treatment process described in the treatment process chart. Details of the process information estimation unit 15 will be described later.
  • the process information storage unit 16 stores the process information estimated by the process information estimation unit 15.
  • the process information storage unit 16 is configured of a storage medium such as a hard disk, an optical disk, or a flash memory.
  • the process information storage unit 16 may be connected by a network.
  • the process information stored in the process information storage unit 16 may be treated as index information for performing cueing or video search when viewing a video related to a treatment action.
  • the image search may be a search based on a treatment process name or a search based on time.
  • the process information estimation unit 15 estimates and outputs process information from the input treatment process chart, video, audio, and device signal. In the estimation of the process information, it is not always necessary to have all the information of video, audio and device signals.
  • FIG. 1 is a functional block diagram showing a configuration example of the process information estimation unit 15.
  • the process information estimation unit 15 includes a determinable treatment process determination unit 151, a feature extraction auxiliary information acquisition unit 152, a feature information extraction unit 153, and a treatment process estimation unit 154.
  • the determinable treatment process determination unit 151 determines the start time and end time of the treatment process (determinable treatment process) that can be determined deterministically from the input treatment process chart and the video, audio, and device signal, that is, the process information
  • the determined and determined one or more pieces of process information are supplied to the feature extraction auxiliary information acquiring unit 152, the feature information extracting unit 153, and the treatment process estimating unit 154.
  • the determinable treatment process is a treatment process that can be determined only by the device operation signal. Specifically, the first and last operations are the treatment steps in which the electronic device is operated. A specific example of the determinable treatment process will be described with reference to the treatment process chart of FIG. Among the treatment steps shown in FIG.
  • the treatment steps “anesthesia”, “pericardiotomy”, “articardiopulmonary operation”, “heart adhesion ablation” and “articardiopulmonary arrest” are the first to be performed in the treatment step.
  • the last work is both the operation of the electronic device. Therefore, the time when the above work is performed can be taken out from the device operation signal. Since the time when the first and last work of each treatment process is performed can be taken out, the start time and the end time of the treatment process can be determined.
  • the earliest time is "skin incision", next The time may be determined to be a treatment process of “pericardiotomy” and the next time “cardiac adhesion removal”.
  • the treatment process of "skin incision” is not operation of an electronic device for the first work, it is not a determinable treatment process.
  • the operation of the above-described determinable treatment process determination unit 151 can also be expressed as follows. That is, the determinable treatment process determination unit 151 first determines the determinable treatment process from among the treatment processes included in the treatment process flowchart based on the treatment process chart. Subsequently, information including the start time and end time and the treatment step name is derived for each determinable treatment step, and is output as step information.
  • the feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input video, audio and device signal based on the process information of the determinable treatment process input and the treatment process chart, and the feature information extraction unit 153 Supply.
  • the feature extraction auxiliary information is used as auxiliary information when the feature information extraction unit 153 extracts features from video, audio, and device signals. The contents of the feature extraction auxiliary information and the details of the derivation process will be described later.
  • the feature information extraction unit 153 is a feature information to be a clue for estimating the treatment process based on the input feature extraction auxiliary information, the video, the audio, the device signal, the treatment process chart, and the process information of the determinable treatment process. Are supplied to the treatment process estimation unit 154. Details of the feature information extraction process will be described later.
  • the treatment process estimation unit 154 estimates the process information of the treatment process other than the determinable treatment process based on the input feature information and the treatment process table and the process information determined by the determinable treatment process determination unit, and the process information Are added together with the process information determined by the determinable treatment process determination unit 151, and output from the process information estimation unit 15.
  • the treatment process other than the determinable treatment process refers to a treatment process that is not determined by the determinable treatment process determination unit 151 among the treatment processes described in the treatment schedule. That is, in the treatment process estimation unit 154, process information is estimated for the remaining treatment processes excluding the treatment process already determined by the determinable treatment process determination unit 151 from the treatment processes described in the treatment schedule.
  • the treatment process estimation unit 154 estimates a treatment process at a time when process information is not determined based on the process information determined by the determinable treatment process determination unit 151. That is, it is not necessary to estimate the process information at the time when the determinable treatment process determination unit 151 determines that the determinable treatment process is being performed. Specifically speaking with reference to FIG. 4, the time from 14:00 to 14:30 at which the treatment process of “anaesthesia” which is a determinable treatment process was performed, and “heart” which is also a determinable treatment process The treatment process estimation unit 154 does not have to estimate the treatment process for the time from 14:55 to 16:50 at which the treatment process of “cardiotomy” to “cardiopulmonary arrest” was performed.
  • the treatment step of “skin incision” treatment process the instruments are used in the order of a scalpel, forceps, an electrocautery cutting mode, and an electrocautery hemostasis mode. Therefore, if it can be judged from the feature information that the above-mentioned instruments are used in the above-mentioned order, the treatment process of "skin incision" can be estimated.
  • results that can not be extracted may be excluded. For example, in the treatment process chart of FIG. 3, if the end time of the treatment process of "artificial cardiopulmonary operation" and the start time of the treatment process of "artificial cardiopulmonary arrest” have already been determined, the time is between them.
  • the only instruments used are scissors, tweezers and electric scalpels.
  • the instrument name to be used is the feature information
  • only the above instrument name can be the feature information indicating the instrument name to be used at the above time, even if the instrument name other than the above is not treated as the feature information good.
  • speech content of a doctor is used as the feature information, if speech that the doctor can utter is limited, other speech may not be treated as the feature information.
  • the treatment process may be estimated from the process information already determined by the determinable treatment process determination unit 151, the process information estimated by the treatment process estimation unit 154, and the treatment process table. For example, in the treatment process chart of FIG. 3, if the end time of the treatment process of "artificial cardiopulmonary operation” and the start time of the treatment process of "artificial cardiopulmonary arrest” have already been determined, the time is between them. It may be determined that the treatment process of “heart adhesion ablation” has been performed.
  • the order of the treatment steps may be in accordance with the treatment order described in the treatment schedule.
  • the treatment steps of “pericardial suture” and “skin suture” instruments are used in the same order, but when both are detected, the one with the earlier time is used.
  • Periodal suture it may be determined that the slower one is “skin suture”.
  • the process information determined by the determinable treatment process determination unit 151 is not supplied to the treatment process estimation unit 154, and the process of the determinable treatment process is performed by the treatment process estimation unit 154.
  • Information may be determined, and process information may be estimated and output for the remaining treatment processes. In this case, there is no need to add the process information determined by the determinable treatment process determination unit 151 and the process information estimated by the treatment process estimation unit 154, and the process determined and estimated by the treatment process estimation unit 154.
  • the information may be the output of the process information estimation unit 15.
  • the process information estimation unit 15 estimates the process information according to the following procedure. First, in the determinable treatment process determination unit 151, the process information of the determinable treatment process is determined from the input device operation signal and the treatment process table, and the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, the treatment process The information is supplied to the estimation unit 154.
  • the feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input process information, the treatment process chart, the video, the audio, and the device signal, and supplies it to the feature information extraction unit 153.
  • the feature information extraction unit 153 extracts the feature information of the input video, voice, and device signal based on the feature extraction auxiliary information, the treatment process chart, and the process information of the determinable treatment process, and the treatment process The information is supplied to the estimation unit 154.
  • the treatment process estimation unit 154 the process information not determined by the determinable treatment process determination unit 151 based on the input characteristic information and the treatment process table, and the process information determined by the determinable treatment process determination unit 151. Estimate and output.
  • the process information obtained by adding the process information determined by the determinable treatment process determination unit 151 and the process information of the treatment process estimation unit 154 is output from the process estimation unit 15, and the process information estimation ends.
  • FIG. 4 is a diagram showing process information output from each component when the treatment process chart of FIG. 3 is input to the process information estimation unit 15.
  • the row of the determinable treatment process determination unit indicates process information output from the determinable treatment process determination unit 151.
  • the row of process names and the row of time included in each row indicate the treatment process name included in the process information output from the component, and the start time and end time of the treatment process.
  • the determinable treatment process determination unit 151 determines process information of a determinable treatment process that can be determined by the device operation signal. That is, as shown in the row of the determinable treatment process determination unit in FIG. 4, the treatment process names of "anesthesia”, “pericardiotomy”, “artificial cardiopulmonary operation”, “heart adhesion ablation”, and “artificial cardiopulmonary arrest” The start time and the end time of the treatment process are determined and output as process information.
  • the feature extraction auxiliary information acquisition unit 152 detects feature extraction auxiliary information for the process information determined above.
  • the feature information extraction unit 153 extracts feature information based on the feature extraction auxiliary information, the treatment process chart, and the process information of the determinable treatment process.
  • the process information of the treatment process not determined by the determinable treatment process determination unit 151 is estimated. That is, as shown in the column of the treatment process estimation unit in FIG. 4, the treatment process names of “disinfection”, “skin incision”, “sternum incision”, “pericardial suture”, “sternum suture”, and “skin suture”
  • the start time and the end time of the treatment process are estimated and output as process information.
  • the feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input video, audio, and device signal based on the input process information and the treatment process chart, and supplies it to the characteristic information extraction unit 153. .
  • FIG. 5 is a diagram showing an example of an operation flow of the feature extraction auxiliary information acquisition unit 152. As shown in FIG. The feature extraction auxiliary information acquisition is sequentially performed in the following steps S151 to S155.
  • the feature extraction auxiliary information is acquired for the target treatment process and the target information.
  • the feature extraction auxiliary information includes sample information and process setting information. Below, the method of sample information acquisition and process setting information acquisition is demonstrated in order. The acquisition of the sample information and the acquisition of the processing setting information may be performed in parallel or sequentially. Moreover, when performed in order, you may perform whichever first. Also, only one of them may be performed. Next, S154 is executed.
  • Sample information is acquired for the target treatment process and the target information, and is used as feature extraction auxiliary information.
  • the sample information is information that is configured by a part of sample values of media data (video, audio, device signal) corresponding to a specific time or a specific period.
  • the sample information handled by the process information estimation unit 15 refers to the feature extraction auxiliary information acquisition unit 152 at the time when the treatment process was performed or the time near the time when the treatment process was determined by the determinable treatment process determination unit 151. It is a collection of input treatment process charts, and video, audio, and sample values of device signals, and is referred to by the feature information extraction unit 153.
  • the sample information can be used as information representing the surrounding situation at a specific time.
  • the surrounding situation is, for example, a voice of an instruction issued by a doctor, the number of unused devices, and the like. Even different treatment steps may result in situations where certain circumstances are consistent. In such a case, the correspondence between the surrounding situation and the sample information observed in the treatment process that can be determined definitively can be utilized in estimating another treatment process occurring in the same surrounding situation.
  • the feature information extraction unit 153 determines that the same surrounding situation is occurring when a sample value identical or close to the sample information representing the specific surrounding situation is detected. Thereby, the accuracy of feature extraction can be improved.
  • the specific example of sample information acquisition is shown below.
  • sample information there is color information of surgical site video.
  • the treatment process of "artificial cardiopulmonary operation" in the treatment process chart of FIG. 3 will be described as an example. Since the heart-lung machine is attached to the incised operative part, the operative part is incised when the treatment process of "artificial cardio-pulmonary operation" is performed. For this reason, the operation part incised is recorded on the operation part image
  • the surrounding situation in which the operation site is incised occurs at the time. I will judge.
  • the color information of the operation site image when the above-mentioned treatment process is performed is set as sample information indicating the surrounding situation in which the operation section is closed.
  • the area of the operative site can be determined by comparing the color information of the unincised surgical site image immediately after the start of the operation with the sample information. It can be said that the area where there is a large difference in color information between the two images is the area in which the body is incised and exposed.
  • the processing target is limited to the surgical part only by narrowing the processing range to the relevant area, and the false detection is eliminated by excluding the other areas from the processing target. You may suppress it.
  • the above description can also be expressed as follows.
  • sample information of both times is compared, and a region where a difference of a certain level or more occurs is compared.
  • a subset of sample information affected by changes in ambient conditions can be identified.
  • the instruction voice requesting the doctor's tweezers is used as sample information. That is, voice sample information is used as sample information indicating the surrounding situation where the doctor is requesting tweezers.
  • the treatment process of “cardiopulmonary arrest” described in the treatment schedule of FIG. 3 is a treatment process determined by the definable treatment process determination unit 151. From the treatment schedule, it can be seen that tweezers is used in the treatment operation of "pericardial suture" immediately after the treatment step of "cardiopulmonary arrest". Therefore, it can be determined that the voice recorded first at the end of the "cardiopulmonary arrest" end time is a voice for which the doctor requests tweezers.
  • the voice recording the instruction voice of the doctor is set as the target information, and the voice information is first detected after the time when the target process is finished.
  • Voice pattern as sample information.
  • the sample information represents the doctor's instruction voice described above. That is, since the voice pattern for which the doctor requests tweezers is represented, when the voice information pattern that matches or is similar to the sample information is detected in the feature information extraction unit 153, the surrounding situation for which the doctor requests tweezers occurs. It is determined that
  • Process setting information is acquired about the said object information regarding an object treatment process, and it is set as feature extraction auxiliary information.
  • the process setting information is control information of a feature information extraction process for extracting feature information from information of video, audio, and device signal in the feature information extraction unit 153.
  • Control information of the feature information extraction process includes parameters of image processing.
  • the parameters of the image processing include, for example, a threshold value in the binarization processing of the image.
  • the result of the feature information extraction processing differs depending on the setting of the control information. In order to improve the accuracy of process estimation, it is necessary to perform effective feature extraction, that is, setting of control information that can extract effective features is required, but manually setting of all control information is necessary. It is difficult, and setting control information to an appropriate value does not necessarily enable effective feature extraction.
  • the setting of the optimal control information is detected using the treatment schedule and the target treatment step. From the treatment process chart, it is possible to judge the situation around the moment when the target treatment process was performed or in the vicinity thereof. For the target information at the time, feature information extraction processing similar to that of the feature information extraction unit 153 is repeatedly performed while changing the setting of control information. About the several characteristic information obtained by this, the control information which extracted the surrounding condition judged from the treatment process chart, and the nearest characteristic information is acquired as process setting information. By using the process setting information obtained in this way, it is possible to extract features that are closer to actuality, so that the accuracy of feature extraction can be improved.
  • the device used immediately after the “cardiopulmonary operation” treatment process is scissors. Therefore, it can be determined that the instrument detected at the earliest time is the scissors after the end time of the treatment process of "artificial cardiopulmonary operation". From here, for the time after the end time of the treatment process of "artificial cardio-pulmonary operation", the image processing and the device detection are repeated while changing the control setting, and it is possible that the device used at that time is scissors.
  • the control setting detected the highest is taken out, and is taken as processing setting information used for scissors detection.
  • An example of processing setting information acquisition is shown below.
  • the type of surgical instrument can be identified by detecting the relative position of the two straight lines.
  • the instrument installed in the operation part such as a chest opener
  • an erroneous detection can be suppressed by excluding the straight line which does not move for a long time from a process target.
  • Hough transform is used for straight line detection.
  • it is necessary to perform binarization processing on the operation site image.
  • setting of the processing information there are a binary threshold value in the binarization processing, a threshold value determined to be a straight line in the Hough transform, and a conversion pixel pitch.
  • the surgical instrument may not be identified correctly. Therefore, the Hough transform is repeatedly performed while changing the setting of the processing information to identify the device being used. All the results are compared, and the setting value of the processing information which gives the result that is the same as or closest to the used instrument described in the treatment schedule is taken out as the processing setting information.
  • S154 It is determined whether acquisition of feature extraction auxiliary information has been completed for all video, audio, and device signal information input to the feature extraction auxiliary information acquisition unit 152. If it is completed (S154 yes), S155 is executed. If not completed (S154 no), processing is performed from target information selection S152 on the information of the video, audio, and device signal of which feature extraction auxiliary information is not detected.
  • the feature information extraction unit extracts feature information based on input feature extraction auxiliary information and video, audio, device signal, treatment process chart, and process information of determinable treatment process, and estimates treatment process It supplies to the part 154.
  • the feature information is information obtained by extracting features from video, audio, and device signals, and is used in the treatment process estimation unit 154 to estimate the treatment process.
  • Feature information is extracted at each preset feature extraction time. However, it is not necessary to extract feature information at all times when samples of video, audio and device signals exist.
  • the feature extraction time may be set to the time of each frame of the video, or may be set to a predetermined interval such as once every two seconds.
  • feature information for example, an instruction voice of a doctor described later
  • the feature extraction time is referred to as a feature extraction target time.
  • the feature information extraction unit 153 extracts feature information at each feature extraction target time, and supplies the feature information to the treatment process estimation unit 154.
  • the feature information extraction unit 153 uses feature extraction auxiliary information to extract feature information. However, it is not necessary to use feature extraction auxiliary information for feature information extraction. An example of feature information is shown below.
  • ⁇ Characteristic information 1 Incision situation of operation part>
  • feature information there is, for example, an incision status of a surgical site in surgery. That is, from the color information of the image obtained by photographing the operative site, if the operative site has more skin color components, the operative site is non-incising, and if the red is more, it is determined that the operative site is incised. Information is used as feature information.
  • sample information of the color information of the surgical site image included in the feature extraction auxiliary information is used.
  • the sample information includes sample information indicating the surrounding situation of the incision site and sample information indicating the surrounding situation of the closed surgery part.
  • ⁇ Feature information 2 Doctor's instruction content> Further, for example, the instruction content detected based on the voice recording the doctor's utterance is used as the feature information. That is, when the surgeon instructs the assistant to deliver an instrument such as a scalpel, the instrument name is used as the feature information. From this feature information, it can be inferred that the surgeon uses the device after the instruction.
  • the sample information of the instruction voice of the doctor included in the feature extraction auxiliary information is used.
  • This sample information indicates the circumstances at which the physician requests the device.
  • a voice pattern matching or similar to the sample information is detected from the voice recording the doctor's utterance at the feature extraction target time, it is determined that the surrounding situation indicated by the sample information is generated. That is, it is determined that an appliance indicated by the surrounding situation has been requested.
  • the feature information 1 and the feature information 2 can be collectively expressed as follows. That is, when the feature extraction auxiliary information includes sample information representing a specific surrounding situation, the sample information at the feature extraction target time is compared with the sample information representing the surrounding situation to evaluate the match or similarity. By doing this, it can be determined whether or not a certain surrounding situation occurs at the feature extraction target time.
  • ⁇ Feature Information 3 Status of Medical Device Storage>
  • a used instrument name estimated from an image obtained by photographing a place of a medical instrument is used as the feature information. That is, the number of remaining devices and the number of reduced devices are determined from the image of the storage place of the medical device, the device used at that time is estimated, and the device name is used as the feature information.
  • the determination of the device may be performed by shape identification by pattern recognition, or another method may be used.
  • the treatment schedule describes the type and sequence of surgical instruments such as a scalpel used for each treatment action.
  • ⁇ Feature information 4 Instruments used in the operation department>
  • a used instrument name estimated from a video obtained by photographing an operation part is set as feature information. That is, an instrument used at the operation site is estimated at each time from the image obtained by photographing the operation site, and the instrument name is used as feature information.
  • the identification of the used instrument may be performed by shape identification by pattern recognition or by linear recognition.
  • processing setting information of straight line detection included in the feature extraction auxiliary information is used.
  • the process setting information includes setting information that can suitably identify the device by linear detection. For this reason, by applying straight line detection based on the processing setting information to an image obtained by photographing the operation part at the feature extraction target time, more accurate identification of the device is possible compared to the case where the processing setting information is not used. It becomes possible.
  • ⁇ Feature information 5 Usage of electronic devices> Also, for example, the usage status of various electronic devices is used as feature information. That is, the device setting signal is used as feature information. For example, when cardiopulmonary arrest is not performed, a device setting signal such as a waveform shape of a heart rate displayed on a living body monitor and for which a determination criterion is not uniquely determined is used as feature information. However, this is limited to the case where there is an instruction to take out the waveform in the treatment schedule.
  • the feature information exemplified above is merely an example of the feature information, and feature information other than the above may be used.
  • the feature information extraction unit 153 may extract feature information only at times when the process information is not determined based on the process information determined by the determinable treatment process determination unit 151. That is, it is not necessary to extract feature information at the time when it is determined in the determinable treatment process determination unit 151 that the determinable treatment process is being performed. Thereby, the processing amount of the feature information extraction processing can be reduced. Specifically speaking with reference to FIG.
  • the time from 14:55 to 16:50 at which the treatment process from “membrane incision” to “cardiopulmonary arrest” is performed may be excluded from the feature extraction target time.
  • the feature information that can be extracted at a certain time is limited from the treatment process and the treatment schedule determined by the determinable treatment process determination unit 151, excluding the result that can not be extracted in the extraction of feature information Also good.
  • feature information can be extracted more accurately. For example, in the treatment process chart of FIG. 3, if the end time of the treatment process of "artificial cardiopulmonary operation" and the start time of the treatment process of "artificial cardiopulmonary arrest” have already been determined, the time is between them.
  • the only instruments used are scissors, tweezers and electric scalpels.
  • the instrument name to be used is the feature information
  • only the above instrument name can be the feature information indicating the instrument name to be used at the above time, even if the instrument name other than the above is not treated as the feature information good.
  • the feature extraction restriction information may be detected at all times when video, audio, or a sample of an equipment signal is present, may be detected at each time of a frame of the video, or may be predetermined at, for example, once every two seconds. It may be detected at intervals. Also, it may be detected according to the feature extraction target time.
  • the process information determined by the decidable treatment process determination unit 151 is not supplied to the feature information extraction unit 153, and the feature information extraction unit 153 determines the device operation signal.
  • the determinable treatment process and the time when it is determined that the determinable treatment process is performed may be detected.
  • the process information estimation unit 15 described above discriminates a determinable treatment process from the input treatment process chart, video, audio, and device signal, and extracts feature extraction auxiliary information including sample information and process setting information therefrom. It has a function of extracting feature information from video, audio and device signals input using the information, and performing treatment process estimation using the feature information.
  • sample information for feature information extraction, it is necessary to pre-set what the extracted feature information means, and also for the judgment criteria, because it can be taken out from the information that records the actual treatment.
  • the feature information can be used more effectively than certain prior art.
  • control setting information for feature information extraction By using processing setting information for feature information extraction, more preferable control setting can be used as compared with the prior art in which control setting for feature information extraction needs to be set in advance, and thus more effective feature information extraction is possible. Become.
  • the process information estimation unit 15 estimates the process information at a certain time, it is assumed that the media data of the time earlier than that time is not referred to. That is, the media data of the future time is not handled for estimation of process information.
  • the feature extraction auxiliary information when acquiring the feature extraction auxiliary information in the feature extraction auxiliary information unit 152, only media data of a time past the current time is referred to. In this case, the feature extraction auxiliary information may be acquired again at an arbitrary timing.
  • the treatment process estimation unit 154 estimates the process information at a certain time, media data of a time later than that time is not referred to.
  • the determination of the process information of the determinable treatment process in the determinable treatment process determination unit 151 is not limited to the process information of FIG.
  • the treatment process estimation unit 154 estimates the treatment process, if the end time of the “disinfection” treatment process is estimated, the treatment process to be performed next is “skin” according to the treatment process chart. It can be judged that it is a treatment process of “incision” And if use of a scalpel is detected from feature information, it will presume that the treatment process of "skin incision" is performed.
  • the name and details of the treatment step (name, order of instruments used, etc.) presumed to be currently performed, and the next If performed, the names and details of the treatment steps described in the treatment step chart may be presented to a doctor or medical staff using video and audio.
  • the process information need not necessarily be estimated for the current treatment process, and only the treatment process which can estimate the start time and the end time of the treatment process.
  • the process information may be estimated.
  • the process information estimation of the therapeutic action can be output at an earlier time than when the treatment process estimation is performed after the recording of the treatment action is completed. Also, by presenting the current treatment process to the doctor or medical staff, the progress of treatment can be confirmed. Similarly, by presenting information on the treatment process to be performed next, it is possible to confirm the next treatment action even if the treatment schedule is not at hand.
  • the target of the process estimation is not necessarily limited to the treatment action. That is, it may be an apparatus that estimates process information for an action that is not a treatment action. Specifically, a process schedule (process name, details of equipment used in process, process in process, electronic in process) having media data on the action and information equivalent to the treatment process chart, for an action that is not a treatment act.
  • estimation of the process information related to the action may be performed as in the case of estimation of the process information of the treatment action.
  • the following is an example of an act that is not a therapeutic act.
  • An action that is not a therapeutic action is, for example, a stage such as a play or live music.
  • a process chart in which the order and details of the processes are predetermined is created.
  • the details of the use of the electronic device are also described in the process chart, it is possible to estimate the process information as in the case of the process information of the therapeutic action.
  • the process information estimated for the stage it becomes easy to browse the video of the stage.
  • videos obtained by shooting the stage can be classified for each process (scene), and seek can be made to an arbitrary scene.
  • assembly of a product For example, assembly of a product. However, it limits to what uses an electronic device. In general, since the assembly of the product is defined in the order and details of the assembly process, estimation of process information is possible as well as estimation of process information of a therapeutic action. By utilizing the process information estimated for the product assembly, it becomes easy to view the image of the product assembly. Thus, for example, by adding the name of the work being performed at each time to the image obtained by photographing the product assembly, it can be used as an educational image. Moreover, the working time of each process can be measured.
  • the apparatus has been described on the assumption that the process information is information including the start time and the end time of the treatment process and the treatment process name in the above description of the treatment process estimation apparatus 1, It may be an apparatus containing information.
  • the information on the importance is information indicating the importance of media data at each time at which the process information is estimated.
  • the information on the degree of importance may be set for each time at which the process information is estimated, may be set for each operation in the treatment process, or may be set for each treatment process.
  • the information on the degree of importance may be determined with reference to the information if it is described in advance in the treatment schedule, or may be derived from media data.
  • the time from the completion of the operation of the artificial heart to the start of the stop work (the “cardiopulmonary stop from the end time of the treatment process of It is judged that the important scene is shown in the image which photographed the operation part, and the importance degree of the time is set high until the time when the treatment process is started.
  • it may be derived from the estimation result of the treatment process. For example, a treatment process with a long working time is set as important.
  • the process information accumulated in the process information accumulation unit 16 may be an index for performing a video search based on the importance.
  • the information on the importance does not necessarily have to be included in the process information of all the times or work or treatment processes, and the information on the importance may be included in only a part of the process information. Further, information of individual importance may be set for each media data, or information of common importance may be set for all media data.
  • the treatment process is described as estimating the process information.
  • the apparatus for estimating the process information with respect to the work (use of the instrument and use of the electronic device) constituting the treatment process It may be. That is, for each work constituting the treatment process, the start time and the finish time may be estimated, and may be combined with the information of the work name to be the process information.
  • the estimation of the process information of the work is performed in the process information estimation unit 15 in the same manner as the estimation of the process information of the treatment process.
  • the work that constitutes the treatment process includes, like the treatment process, a work that can be determined deterministically (determinable work).
  • the determinable work is a work using an electronic device, and is a work that can determine process information based on a device operation signal.
  • the work constituting the treatment process of “skin incision” is a scalpel, tweezers (gauze), electric scalpel cutting mode, and electric scalpel hemostasis mode.
  • the operation of the electric scalpel cutting mode and the electric scalpel hemostasis mode is an operation using the electronic device, and since the start time and the end time of the operation can be determined definitely by the device operation signal, the operation can be determined.
  • the estimation of work process information will be described below.
  • the determinable treatment process determination unit 151 determines the process information of the determinable work. That is, process information is determined for an operation that can determine the process information from the device operation signal. At this time, even if the operation constitutes a treatment process that is not a determinable treatment process, the process information is determined if it is an determinable operation. For example, in the treatment process chart of FIG. 3, although the treatment process of "skin incision" is not a determinable treatment process, it is a process about the determinable work (electric knife cutting mode, electric scalpel hemostasis mode) included in this treatment process. Determine the information.
  • the treatment process estimation unit 154 estimates the process information of the work for which the process information is not determined. That is, the process information of the work which is not a determinable work is estimated. At this time, similarly to the estimation of the process information of the treatment process, it is not necessary to estimate the process information of the operation at the time when the determinable treatment process determination unit 151 determines that the determinable operation is performed. Further, similarly to the estimation of the process information of the treatment process, the process information of the operation may be estimated from the process information of the operation already determined or estimated and the treatment process chart.
  • the process information stored in the process information storage unit 16 may be an index for performing a video search by the work name.
  • the video search by the work name becomes possible, and the confirmation of the video can be simplified.
  • the process information of the treatment action is estimated using only the information on the treatment action (the estimation target treatment action) which is the target of the process information estimation.
  • the apparatus may be an apparatus that estimates process information of a therapeutic action using information of a therapeutic action (non-estimated target therapeutic action) other than the therapeutic action. That is, in the estimation of the treatment process information of the estimation target treatment practice, the feature information extraction auxiliary information on one or more non-estimation target treatment practices is referred to when extracting the feature of the estimation target treatment practice, and the feature information is extracted. . Then, the treatment process may be estimated using the feature information.
  • the non-estimated target treatment act is, for example, a treatment act performed in the past.
  • the process information estimation unit 15 additionally includes a recording unit that records the estimated process information and the feature extraction auxiliary information used at the time of derivation thereof. Also, for example, it is a therapeutic action performed in parallel with the presumed target therapeutic action.
  • the feature extraction auxiliary information of the non-estimated target therapeutic action may be information acquired and accumulated when the process information of the non-estimated target therapeutic action is estimated in the past, or the feature extraction aid of the estimated target therapeutic action When acquiring information, it may be acquired from the treatment schedule and media data regarding the non-estimated target treatment practice.
  • the feature extraction assistance of the non-estimation target treatment More effective feature information extraction can be performed by using the information. Therefore, more accurate treatment process estimation is possible.
  • the treatment process estimation apparatus 1 does not necessarily need a storage medium that configures the process information storage unit 16. Further, the estimated process information can be output without passing through the process information storage unit 16.
  • FIG. 6 is a functional block diagram showing a configuration example of the feature information extraction unit 153.
  • the feature information extraction unit 153 includes a feature extraction target time setting unit 1531, a feature extraction restriction information setting unit 1532, a media data processing unit 1533, a media data comparison unit 1534, and a feature information acquisition unit 1535.
  • the feature extraction target time setting unit 1531 sets the feature extraction target time described above based on the input process information of the determinable treatment process and the treatment process chart, and the information is compared with the media data processing unit 1533 and media data comparison. It supplies to the part 1534.
  • the feature extraction restriction information setting unit 1532 sets the above-mentioned feature extraction restriction information based on the process information of the determinable treatment process and the treatment process chart inputted, and the information is compared with the media data processing unit 1533 and the media data comparison. It supplies to the part 1534.
  • the media data processing unit 1533 adds processing such as image processing to the input video, audio, and device signal based on the feature extraction auxiliary information including the process setting information to be input and the feature extraction target time information, and the processing result Are supplied to the feature information acquisition unit 1535.
  • processing such as image processing is processing performed on media data to extract feature information, and, for example, binarization, Hough transform, pattern recognition on a frame of an image obtained by photographing a surgical site, In addition, it is binarization and pattern recognition for a frame of an image obtained by photographing an instrument storage area.
  • processing may be performed not only on video but also on audio and device signals, and processing results may be extracted.
  • the settings and parameters of the various processes described above are preset before the treatment process estimation is performed. However, when the above-mentioned processing setting information is included in the feature extraction auxiliary information input to the media data processing unit 1533, the processing setting information is reflected on the setting of each processing and the parameter. Note that it is not necessary to reflect all the process setting information.
  • the processing result is accompanied by time information of the media data that is the target of the processing.
  • the media data comparison unit 1534 compares the information of the input video, audio, and device signal with the reference sample information based on the feature extraction auxiliary information including the input sample information and the feature extraction target time information, and the comparison result Are supplied to the feature information acquisition unit 1535.
  • the reference sample information is, for example, color information of the surgical site when the surgical site is incised with respect to an image obtained by photographing the surgical site.
  • the color information of this reference sample information is compared with the color information of the image obtained by photographing the surgical site included in the media data, and a numerical result indicating how similar it is is regarded as a comparison result, and the comparison result is Based on the feature information is determined.
  • the feature information acquisition unit 1535 sets feature information on the assumption that the surgical site is incised. If they are not similar, it is not necessary to set feature information as the operation part is closed, or not to set the feature information as not being similar to the reference sample information.
  • the reference sample information described above is preset before the treatment process estimation is performed. However, when the above-described sample information is included in the feature extraction auxiliary information input to the media data processing unit 1533, the sample information is set as the reference sample information, and otherwise, the specified value is set as the reference sample information. Do. Note that it is not necessary to reflect all reference sample information.
  • the comparison result is accompanied by time information of the media data which is the target of the comparison.
  • the feature information acquisition unit 1535 determines feature information based on the input feature extraction restriction information, the processing result, and the comparison result, and outputs the feature information extraction unit 153.
  • the media data comparison unit 1534 may not receive the feature extraction auxiliary information, and may add a specified value to the reference sample information.
  • the media data processing unit 1533 may not receive the feature extraction auxiliary information, and the setting of the process may be set as the specified value.
  • the process information estimation unit 15 is configured by the determinable treatment process estimation unit 151, the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, and the treatment process estimation unit 154. Although it is not necessary to have this configuration, the process information estimation unit 15 is configured by replacing the feature information extraction unit 153 and the treatment process estimation unit 154 with a treatment process estimation unit 153 a that integrates their functions. It may be an apparatus.
  • FIG. 7 is a view showing a modification of the process information estimation unit 15.
  • the process information estimation unit 15 may be configured of a determinable treatment process estimation unit 151, a feature extraction auxiliary information acquisition unit 152, and a treatment process estimation unit 153a.
  • the functions of the determinable treatment process estimation unit 151 and the feature extraction auxiliary information acquisition unit 152 are as described above.
  • the treatment process estimation unit 153a extracts the feature information based on the input feature extraction auxiliary information and the video, audio, device signal, treatment process chart, and process information of the determinable treatment process.
  • the process information of the treatment process other than the determinable treatment process is estimated based on the feature information, the treatment process table, and the process information of the determinable treatment process, and the process information is determined by the determinable treatment process determination unit 151
  • the process information is added up with the process information and output from the process information estimation unit 15. Extraction of feature information in the treatment process estimation unit 153a is performed in the same manner as extraction of feature information in the feature information extraction unit 153 described above. Further, estimation of the treatment process in the treatment process estimation unit 153a is performed in the same manner as the treatment process estimation unit 154 described above.
  • the process information is estimated after extracting the feature information for all feature extraction target times, but in the case of this configuration, it is not always in the order described above.
  • the extraction of feature information and the estimation of process information may be performed in parallel.
  • the treatment process estimation apparatus 1 (in particular, the process information estimation unit 15) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or software using a CPU (Central Processing Unit) It may be realized by
  • the treatment process estimation apparatus 1 is a CPU that executes instructions of a program that is software that realizes each function, a ROM (Read Only Memory) in which the program and various data are readably recorded by a computer (or CPU). Or a storage device (these are referred to as “recording media"), a RAM (Random Access Memory) for developing the program, and the like.
  • the object of the embodiment of the present invention is achieved by the computer (or CPU) reading and executing the program from the recording medium.
  • a “non-transitory tangible medium”, for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit or the like can be used.
  • the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program.
  • any transmission medium communication network, broadcast wave, etc.
  • one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.
  • a treatment process estimation apparatus is a treatment process estimation apparatus that determines process information related to one or more treatment processes, and media data for acquiring one or more media data related to the treatment process.
  • a treatment process estimation unit configured to estimate the process information based on the treatment process chart and the auxiliary information.
  • the embodiment of the present invention may also take the following means.
  • auxiliary information is sample information.
  • auxiliary information is processing setting information.
  • sample information is characterized in that the sample information is information that can be acquired from an image of a surgical site based on at least process information on the determinable treatment process and the treatment process chart.
  • sample information is characterized in that the sample information is information that can be acquired from voice information recording an instruction during treatment based on at least process information on the determinable treatment process and the treatment schedule.
  • the treatment process estimation unit estimates the process information based on at least a result of a straight line detection process based on the process setting information and an identification result of identifying a medical device used in a video.
  • the determinable treatment process determination unit determines the process information of each operation constituting the treatment process, and the treatment process estimation unit estimates the process information of each operation constituting the treatment process. It is characterized by
  • the treatment process estimation unit uses at least the treatment process chart and process information on the determinable treatment process to determine the type and number of medical instruments detected for the time when the process information is not estimated. It is characterized in that information including the information is estimated, and the process information is estimated based on the information.
  • the treatment process estimation unit determines at least the treatment process list and process information related to the determinable treatment process. And estimating the information on the unestimated treatment process based on the estimated process information.

Landscapes

  • Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Engineering & Computer Science (AREA)
  • Epidemiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Primary Health Care (AREA)
  • Public Health (AREA)
  • Radiology & Medical Imaging (AREA)
  • Surgery (AREA)
  • Urology & Nephrology (AREA)
  • Medical Treatment And Welfare Office Work (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

The present invention addresses the problem of providing information that facilitates confirmation or editing of a video pertaining to the act of treatment without relying on a special apparatus or a standard treatment program. A process estimation unit (15) for estimating a treatment process from a video, a sound, an apparatus signal, and a table of treatment process in which details of an act of treatment are written is configured from a determinable treatment process determination unit (151) for determining a treatment process that can be determinately estimated, a feature extraction auxiliary information acquisition unit (152) for acquiring auxiliary information on the basis of the treatment process, a feature information extraction unit (153) for extracting feature information utilizing the auxiliary information; and a treatment process estimation unit (154) for estimating a treatment process on the basis of the feature information.

Description

治療工程推定装置Treatment process estimation device
　本発明の一実施態様は、取得した治療行為に関わる映像、音声および機器情報を用いて、治療行為中の各時刻における治療工程を検出する治療工程推定装置に関する。 One embodiment of the present invention relates to a treatment process estimation apparatus that detects a treatment process at each time during treatment using the acquired video, audio, and device information related to the treatment.
　近年、医療分野において、映像が利用される機会が増加している。例えば、手術中の術部や手術室の様子を撮影した映像が、手術内容の検討や医療従事者の教育に利用されている。その際、映像をそのまま利用するのではなく、手術内容の検討や医療従事者の教育に合わせた編集が施される。そのためには、撮影した映像を確認して編集する必要があり、特に、映像が長時間に渡る場合や、一つの治療行為について複数の映像が記録されている場合などには、映像の確認と編集作業に多大な時間と労力が掛かるという課題があった。 In recent years, in the medical field, opportunities for using video have increased. For example, images obtained by photographing the operation part and the operating room during surgery are used for examination of the contents of surgery and education of medical workers. At that time, instead of using the video as it is, editing is performed in accordance with the examination of the contents of the surgery and the education of the medical staff. For that purpose, it is necessary to confirm and edit the taken image, and in particular, when the image is extended for a long time or when a plurality of images are recorded for one treatment action, the image confirmation and the like There was a problem that it took a lot of time and effort for editing work.
　これを解決するため、特許文献１では、治療行為の映像の重要部分にタグ付けし、同映像の確認を簡易化する技術が提案されている。例えば、手術器具や医師に取り付けられたＩＣタグの位置情報に基づき治療工程を推定し、推定した治療工程毎に、映像を再生する技術が記載されている。 In order to solve this, Patent Document 1 proposes a technique for tagging important parts of the video of the treatment action and simplifying the confirmation of the video. For example, a technique is described that estimates a treatment process based on position information of a surgical instrument or an IC tag attached to a doctor, and reproduces an image for each estimated treatment process.
　また、特許文献２では、治療工程の具体的な内容や遷移条件が規定された標準的治療プログラム（ＡＣＬＳやＪＡＴＥＣ等）に則り、治療工程の遷移を追うことで各時刻の治療工程を推定し、その推定情報を元に、治療工程ごとに映像を再生する技術が記載されている。 Further, in Patent Document 2, the treatment process at each time is estimated by following the transition of the treatment process according to a standard treatment program (such as ACLS or JATEC) in which specific contents of the treatment process and transition conditions are defined. The technique which reproduces an image for every treatment process based on the presumed information is described.
日本国公開特許公報「特開２００６－１６４２５１号公報」（公開日：２００６年６月２２日）Japanese Patent Publication "2006-164251" (publication date: June 22, 2006) 日本国公開特許公報「特開２００６－２５２２８６号公報」（公開日：２００６年９月２１日）Japanese Patent Publication "2006-252286" (release date: September 21, 2006)
　しかしながら、特許文献１および特許文献２に記載の技術は、特定の条件を満たした治療行為についてのみ、治療工程の推定が可能となる。特許文献１に記載のように、治療工程の推定にＩＣタグを利用する場合、ＩＣタグの取りつけられた特別な器具およびＩＣタグリーダが必要となるため、当該器具が使用されていない治療行為の映像について、治療工程の推定ができないという課題がある。また、特許文献２に記載のように、標準的治療プログラムに則り治療行為を推定する場合、標準的治療プログラムに対応しない治療行為の映像について、治療工程の推定ができないという課題がある。 However, the techniques described in Patent Literature 1 and Patent Literature 2 can estimate the treatment process only for the therapeutic action that satisfies the specific condition. As described in Patent Document 1, when an IC tag is used to estimate a treatment process, a special device with an IC tag and an IC tag reader are required, so a video of a treatment action in which the device is not used There is a problem that the treatment process can not be estimated. Further, as described in Patent Document 2, when estimating the treatment action according to the standard treatment program, there is a problem that the treatment process can not be estimated for the image of the treatment action which does not correspond to the standard treatment program.
　本発明の一実施態様は、上記の事情を鑑みてなされたものであり、その目的は、ＩＣタグなどの特殊な器具や標準的治療プログラムを使うことなく治療工程推定が可能な医療情報処理装置を実現することにある。 One embodiment of the present invention has been made in view of the above circumstances, and its object is to provide a medical information processing apparatus capable of estimating a treatment process without using a special device such as an IC tag or a standard treatment program. To realize.
　上記の課題を解決するため、本発明の一実施態様にかかる治療工程推定装置は、一つ以上の治療工程に関わる工程情報を決定する治療工程推定装置であって、前記治療工程に関する一つ以上のメディアデータを取得するメディアデータ取得部と、前記治療工程を示す治療工程表を取得する治療工程表取得部と、少なくとも前記メディアデータと前記治療工程表に基づき、確定可能な治療工程に関する工程情報を決定する確定可能治療工程決定部と、少なくとも前記確定可能な治療工程に関する工程情報、前記メディアデータおよび前記治療工程表に基づき、補助情報を取得する補助情報取得部と、少なくとも前記確定可能な治療工程に関する工程情報、前記メディアデータ、前記治療工程表および前記補助情報に基づき、前記工程情報を推定する治療工程推定部とを備える。 In order to solve the above-mentioned subject, a treatment process estimating device concerning one embodiment of the present invention is a treatment process estimating device which determines process information concerning one or more treatment processes, and one or more about the treatment process concerned Media data acquisition unit for acquiring media data of a medical treatment process, a treatment process schedule acquisition unit for acquiring a treatment process schedule indicating the treatment process, and process information on a treatment process that can be determined based on at least the media data and the treatment process schedule Determining an determinable treatment process, an auxiliary information acquiring unit acquiring an auxiliary information based on at least process information on the determinable treatment process, the media data, and the treatment process chart, and at least the determinable treatment The process information is estimated based on process information on the process, the media data, the treatment process chart and the auxiliary information. And a treatment step estimator for.
　本発明の一実施態様によれば、入力された治療工程表と映像、音声、機器信号より、確定的に決定できる治療工程を判別し、そこから、サンプル情報と処理設定情報からなる補助情報(特徴抽出補助情報とも言う)を取り出し、その情報を利用して入力された映像、音声、機器信号より特徴情報抽出を行い、当該特徴情報を用いて治療工程推定を行う機能を提供できる。 According to one embodiment of the present invention, a treatment process that can be determined definitively is determined from the inputted treatment process chart and video, audio, and device signal, and auxiliary information (sample information and treatment setting information) is determined therefrom. It is possible to provide a function of extracting feature information (also referred to as feature extraction auxiliary information), extracting feature information from the input video, audio, and device signal using the information, and performing treatment process estimation using the feature information.
　特徴情報抽出にサンプル情報を利用することで、抽出した特徴情報が何を意味するのか、またその判断基準は何かを、実際の治療を記録した情報から取り出せるため、これらを予め設定する必要のある従来技術に比べ、特徴情報をより有効に利用できる。 By using sample information for feature information extraction, it is necessary to pre-set what the extracted feature information means, and also for the judgment criteria, because it can be taken out from the information that records the actual treatment. The feature information can be used more effectively than certain prior art.
　特徴情報抽出に処理設定情報を利用することで、特徴情報抽出の制御設定を予め設定する必要がある従来技術に比べ、最適な制御設定を使用できるため、より有効な特徴情報抽出が可能となる。 Using process setting information for feature information extraction enables more effective feature information extraction because it is possible to use optimal control settings as compared to the prior art in which control settings for feature information extraction need to be set in advance. .
本発明の実施形態における治療工程推定装置に含まれる、工程情報推定部の構成例を示す図である。It is a figure which shows the structural example of the process information estimation part contained in the treatment process estimation apparatus in embodiment of this invention. 同治療工程推定装置の構成例を示すブロック図である。It is a block diagram which shows the structural example of the treatment process estimation apparatus. 同治療工程推定装置の動作を説明するための、治療工程表の一例を示す図である。It is a figure which shows an example of a treatment process chart for demonstrating the operation | movement of the treatment process estimation apparatus. 同治療工程推定装置の動作を説明するための、工程推定部と、工程推定部の各構成要素から出力される工程情報の一例を示す図である。It is a figure which shows an example of the process information output from each process element of the process estimation part for demonstrating the operation | movement of the treatment process estimation apparatus of the same, and a process estimation part. 同治療工程推定装置の動作を説明するための、特徴抽出補助情報取得のフローチャートの一例を示す図である。It is a figure which shows an example of the flowchart of feature extraction assistance information acquisition for demonstrating the operation | movement of the treatment process estimation apparatus. 同治療工程推定装置の動作を説明するための、特徴情報抽出部の構成例を示す図である。It is a figure which shows the structural example of the characteristic information extraction part for demonstrating the operation | movement of the treatment process estimation apparatus. 同治療工程推定装置の動作を説明するための、工程情報推定部の変形例を示す図である。It is a figure which shows the modification of a process information estimation part for demonstrating the operation | movement of the treatment process estimation apparatus.
　図２は本実施形態における治療工程推定装置１の構成例を示すブロック図である。図２に示すように、治療工程推定装置１は、概略的には、治療工程表、および、メディアデータ（映像、音声、機器信号）に基づいて工程情報を導出して記録する装置である。各構成要素の説明を行う前に、以下では、まず、本実施形態における治療工程表、および、映像、音声、機器信号の詳細を説明する。 FIG. 2 is a block diagram showing a configuration example of the treatment process estimation apparatus 1 in the present embodiment. As shown in FIG. 2, the treatment process estimation apparatus 1 is an apparatus for deriving and recording process information on the basis of a treatment process chart and media data (video, audio, device signal). Before describing each component, in the following, the treatment process chart in the present embodiment and the details of the video, audio, and device signals will be described.
　　［治療工程表]
　治療工程表とは、治療処置の開始から終了までに実行される複数の治療工程について、治療工程の順序、および、各治療工程の詳細情報を含む表である。治療工程の詳細情報には、治療工程名、治療工程における使用器具の情報（例えば、使用順序や種類）、治療工程における電子機器の情報（例えば、使用順序や種類）を含んでいる。 [Processing schedule]
The treatment schedule is a table including the order of treatment steps and detailed information on each treatment step for a plurality of treatment steps performed from the start to the end of treatment treatment. The detailed information on the treatment process includes the treatment process name, information on the equipment used in the treatment process (for example, use order and type), and information on the electronic device in the treatment process (for example, use order and type).
　以下、本実施形態の説明に用いる治療工程表の例について図３を参照して説明する。図３は心臓癒着切除手術の治療工程表の一例である。治療工程表には、治療工程名、使用器具、電子機器の列が含まれている。治療工程名は各治療工程の名称を表す。図３に示すように、治療工程表には、治療工程名として、「処置開始」、「麻酔」、「消毒」、「肌切開」、「胸骨切開」、「心膜切開」、「人工心肺稼働」、「心臓癒着部切除」、「人工心肺停止」、「心膜縫合」、「胸骨縫合」、「肌縫合」、「処置終了」が順に記載されており、この記載順序が治療工程の順序と一致する。使用器具の列にはメスやピンセットなど治療工程で用いられる医療器具の名称と使用順序を表し、電子機器の列には治療工程で用いられる電子機器の名称と使用順序を表している。使用器具、電子機器の使用は、表中で上にある項目から順に行われる。なお、一つの治療工程内での使用器具または電子機器の使用順序は、使用器具と電子機器を問わず治療工程表の上から記載の順となる。例えば、「肌切開」の治療工程であれば、その工程内の使用順序は、メス、ピンセット、電気メス切断モード、電気メス止血モードの順となる。「麻酔」の治療工程については、電子機器による麻酔注入操作が示されている。「消毒」の治療工程については、ピンセットを使用した作業が示されている。「肌切開」の治療工程については、メスとピンセットの二種類の器具と、電気メスを使用した作業の順序が示されている。図３は治療工程表の一例であり、各治療工程の詳細情報として、作業予定時刻、作業担当者、患者の状態などの項目が追加されていてもよい。 Hereinafter, an example of a treatment process chart used for describing the present embodiment will be described with reference to FIG. FIG. 3 is an example of a treatment process chart of cardiac adhesion resection operation. The treatment schedule includes a row of treatment step names, instruments used, and electronic devices. The treatment step name represents the name of each treatment step. As shown in FIG. 3, in the treatment process chart, as treatment process names, "treatment start", "anesthesia", "disinfection", "skin incision", "sternal incision", "pericardio incision", "artificial cardiopulmonary bypass" "Operation", "heart adhesion ablation", "articardiopulmonary arrest", "pericardial suture", "sternal suture", "skin suture", "end of treatment" are described in order, and this description order is the treatment process Match the order. The row of instruments used indicates the name and order of use of medical devices used in the treatment process such as a scalpel and tweezers, and the row of electronic devices indicates the name and order of use of electronic devices used in the treatment process. Equipment used, the use of the electronic device is performed in the order from the top item in the table. In addition, the use order of the use apparatus or electronic device in one treatment process becomes the order of description from the top of the treatment process chart regardless of the use apparatus and the electronic device. For example, in the case of a "skin incision" treatment process, the use order in the process is in the order of a scalpel, tweezers, an electric scalpel cutting mode, and an electric scalpel hemostasis mode. Regarding the treatment process of "anesthesia", an anesthetic injection operation by an electronic device is shown. For the "disinfection" treatment process, work using tweezers is shown. For the treatment process of "skin incision", two kinds of instruments, a scalpel and tweezers, and an operation sequence using an electric scalpel are shown. FIG. 3 is an example of a treatment process chart, and items such as scheduled work times, workers, and patient status may be added as detailed information on each treatment process.
　なお、本実施形態の治療工程推定装置１の入力として利用可能な治療工程表は、必ずしも図３に示したような表の形式である必要はなく、上述の治療工程の順序、および、治療工程の詳細情報を導出できれば別の形式であってもよい。例えば、治療工程の順序を示すデータと、各治療工程の詳細情報を読み出すデータをそれぞれ読み込んで組み合わせて利用してもよい。 In addition, the treatment process schedule which can be used as an input of the treatment process estimation apparatus 1 of this embodiment does not necessarily need to be a format of a table as shown in FIG. 3, and the order of the above-mentioned treatment process and a treatment process If detailed information of can be derived, it may be in another format. For example, data indicating the order of treatment steps and data for reading out detailed information of each treatment step may be read and used in combination.
　　［映像・音声・機器信号]
　映像、音声、機器信号とは、治療に関わる映像、音声、機器制御情報について、各種機器から送出される信号である。 [Video, audio, device signal]
The video, audio, and device signals are signals sent from various devices regarding video, audio, and device control information related to treatment.
　映像は、時刻に関連付けられた複数のフレームで構成される。関連付けには、例えば、フレーム毎に付与されるタイムスタンプが用いられる。ある時刻ｔの映像として、その時刻ｔに最も近いタイムスタンプを持つフレームを利用できる。 A video is composed of a plurality of frames associated with time. For association, for example, a time stamp given to each frame is used. As a video at a certain time t, a frame having a timestamp closest to that time t can be used.
　映像には映像付帯情報が付帯している。映像付帯情報には撮影対象に応じた映像分類を示す分類情報が含まれる。分類情報は、例えば、以下の映像を識別する情報である。（１）患者の術部を撮影した映像、（２）医療器具の置き場を撮影した映像、（３）治療室内全体もしくは一部を撮影した映像、（４）医師を補助する医療スタッフの動作や手元の器具の様子を撮影した映像、（５）医師の表情や目線を撮影した映像、医師が身につけるカメラにより医師の目線で撮影された映像、（６）治療に関わる生体モニタなど医療機器の出力画面を撮影した映像、（７）人工心肺等の医療機器の動作の様子や機器の操作パネルを撮影した映像、（８）内視鏡カメラや顕微鏡の映像、（９）患者から切除された人体組織やその置き場を撮影した映像。 Video incidental information is attached to the video. The video incidental information includes classification information indicating a video classification according to the imaging target. The classification information is, for example, information for identifying the following video. (1) Video taken of the patient's surgical site, (2) Video taken of the storage area of the medical device, (3) Video taken of the whole or part of the treatment room, (4) Operation of the medical staff assisting the doctor or An image of the device at hand, (5) An image of the doctor's expression and eyes, an image of the doctor's eye with a camera worn by the doctor, and (6) Medical equipment such as a biological monitor related to treatment (7) Images of the operation of medical equipment such as artificial heart and heart and the operation panel of the equipment, (8) Images of endoscopic camera and microscope, (9) Excised from patient Of human body tissue and its storage area.
　音声は、時刻に関連付けられた複数のフレームで構成される。関連付けには、例えば、フレーム毎に付与されるタイムスタンプが用いられる。ある時刻ｔの音声として、その時刻ｔに音声信号が検出されている場合、当該音声信号が検出され始めたフレームから、検出されなくなるフレームまでの、音声信号を利用できる。音声には音声付帯情報が付帯している。音声付帯情報には記録対象に応じた音声分類を示す分類情報が含まれる。音声分類は、例えば、以下の音声を識別する情報である。（１）医師の発声を記録した音声、患者の発声を記録した音声、（２）医師と医療スタッフの会話を記録した音声、（３）治療室内全体の音声を記録した音声、（４）医療機器のアラーム音を記録した音声。 Voice is composed of a plurality of frames associated with time. For association, for example, a time stamp given to each frame is used. As a voice at a certain time t, when a voice signal is detected at the time t, a voice signal from a frame in which the voice signal starts to be detected to a frame in which the voice signal is not detected can be used. Audio incidental information is attached to the audio. The audio incidental information includes classification information indicating audio classification according to the recording target. The voice classification is, for example, information for identifying the following voice. (1) voice recording doctor's voice, voice recording patient's voice, (2) voice recording doctor's and medical staff's conversation, (3) voice recording voice throughout the treatment room, (4) medical care Voice recording the alarm sound of the device.
　機器信号とは、時刻に関連付けられた医療電子機器の入出力信号である。関連付けには、例えば、信号に付与されるタイムスタンプが用いられる。ある時刻ｔの機器信号として、その時刻ｔに最も近いタイムスタンプを持つ信号を利用できる。医療電子機器とは、例えば生体モニタ、人工心肺装置、電気メスである。人工心肺装置の機器信号には、血液ポンプの稼働状況を示す信号、ヒータの稼働状況を示す信号、ヒータの温度設定を示す信号、人工心肺装置の稼働状況を示す信号、人工心肺装置の心拍数を示す信号、心臓保護液注入装置の稼働状況を示す信号が含まれる。また、電気メスの機器信号には、稼働状況を示す信号、メスに流れる電流量を示す信号、電気メスに流れる電流量を示す信号、使用中の動作モードを示す信号（例えば切開モード、または、止血モードのいずれのモードを使用しているか示す信号）がある。また生体モニタの機器信号には、生体モニタで計測される各種生体数値を示す信号（例えば、脈拍数を示す信号や心電図波形を示す信号）がある。 The device signal is an input / output signal of the medical electronic device associated with the time. For the association, for example, a time stamp given to the signal is used. A signal having a time stamp closest to time t can be used as a device signal at time t. The medical electronic devices are, for example, a living body monitor, a heart-lung machine, and an electrosurgical knife. The device signals of the heart-lung machine include a signal indicating the operating state of the blood pump, a signal indicating the operating state of the heater, a signal indicating the temperature setting of the heater, a signal indicating the operating state of the heart-lung machine, and the heart rate of the heart-lung machine. And a signal indicating the operation status of the cardioprotective fluid injection device. In addition, the instrument signal of the electric scalpel includes a signal indicating an operation status, a signal indicating an amount of current flowing to the scalpel, a signal indicating an amount of current flowing to the electric scalpel, and a signal indicating an operation mode in use (for example, a cutting mode or There is a signal that indicates which mode of hemostasis mode is being used. The device signal of the living body monitor includes a signal (for example, a signal showing a pulse rate or a signal showing an electrocardiogram waveform) indicating various living body values measured by the living body monitor.
　機器信号には、機器信号付帯情報が付帯している。機器信号付帯情報にはその機器信号が何の機器の信号なのか、何を意味する信号なのかを示す情報が含まれる。例えば、人工心肺装置の稼働状況を示す信号には、この信号は人工心肺装置の稼働状況を示す信号だと示す情報が含まれる。機器信号付帯情報は、機器の入出力信号に初めから含まれていても良いし、後述の機器信号取得部で加えられても良い。 Equipment signal incidental information is attached to the equipment signal. The device signal additional information includes information indicating what device the device signal is and which signal means the device signal. For example, the signal indicating the operation status of the heart-lung machine includes information indicating that the signal is a signal indicating the operation status of the heart-lung machine. The device signal incidental information may be included in the input / output signal of the device from the beginning or may be added by the device signal acquisition unit described later.
　以下の説明では、上記の例にあるような、各機器/機能の稼働状況を二値で表現する信号を機器稼働信号と呼ぶ。機器稼働信号は後述の確定可能治療工程決定部１５１で利用される。機器稼働信号とは、例えば、人工心肺装置の稼働状況を示す信号、血液ポンプの稼働状況を表す信号、ヒータの稼働状況を表す信号、心臓保護液注入装置の稼働状況を示す信号、電気メスの稼働状況を示す信号を示す信号である。また、各機器の出力設定/制御情報を示す信号を機器設定信号と呼ぶ。機器設定信号は後述の特徴情報抽出部１５３で利用される。機器設定信号とは、例えば、ヒータの温度設定を示す信号、人工心肺の心拍数を示す信号、電気メスの動作モードを示す信号、電気メスに流れる電流量を示す信号または／および生体モニタで計測される各種生体数値を表す信号である。 In the following description, a signal representing the operating status of each device / function in binary as in the above example will be referred to as a device operating signal. The device operation signal is used by the below-described determinable treatment process determination unit 151. The device operation signal is, for example, a signal indicating the operating condition of the artificial heart-lung machine, a signal indicating the operating condition of the blood pump, a signal indicating the operating condition of the heater, a signal indicating the operating condition of the cardioprotective solution injection device, an electric scalpel It is a signal which shows the signal which shows an operating condition. Further, a signal indicating output setting / control information of each device is called a device setting signal. The device setting signal is used by a feature information extraction unit 153 described later. The instrument setting signal is, for example, a signal indicating the temperature setting of the heater, a signal indicating the heart rate of the artificial heart and lung, a signal indicating the operation mode of the electric scalpel, a signal indicating the amount of current flowing to the electric scalpel and / or measurement with a living body monitor Is a signal representing various biometric values.
　映像、音声、機器信号それぞれのタイムスタンプは、治療中の実際の時刻と同期されていることとする。 It is assumed that the time stamps of the video, audio and device signals are synchronized with the actual time of treatment.
　（治療工程推定装置の構成）
　図２は治療工程推定装置１の構成例を示すブロック図である。図２に示すように、治療工程推定装置１は、映像取得部１１、音声取得部１２、機器信号取得部１３、治療工程表取得部１４、工程情報推定部１５、工程情報蓄積部１６を備える。 (Configuration of treatment process estimation device)
FIG. 2 is a block diagram showing a configuration example of the treatment process estimation apparatus 1. As shown in FIG. 2, the treatment process estimation apparatus 1 includes a video acquisition unit 11, an audio acquisition unit 12, a device signal acquisition unit 13, a treatment process schedule acquisition unit 14, a process information estimation unit 15, and a process information storage unit 16. .
　映像取得部１１は、治療工程推定の対象となる治療に関わる映像を一つ以上取得し、工程情報推定部１５に供給する。治療に関わる映像として、少なくとも、分類情報が示す分類が患者の術部を撮影した映像と、医療器具の置き場を撮影した映像を取得する必要がある。さらに、必要に応じ、上記以外の治療に関わる映像を取得しても良い。 The image acquisition unit 11 acquires one or more images related to the treatment to be a target of treatment process estimation, and supplies the image to the process information estimation unit 15. As a video relating to treatment, it is necessary to obtain at least a video of a classification indicated by the classification information obtained by photographing the operation part of the patient and a video obtained by photographing the place of the medical device. Furthermore, you may acquire the image regarding treatment other than the above as needed.
　音声取得部１２は、治療工程推定の対象となる治療に関わる音声を一つ以上取得し、工程情報推定部１５に供給する。治療に関わる音声として、少なくとも、分類情報が示す分類が医師の発声を記録した音声を取得する必要がある。さらに、必要に応じ、上記以外の音声を取得しても良い。 The voice acquisition unit 12 acquires one or more voices related to the treatment to be a target of the treatment process estimation, and supplies the one or more sounds to the process information estimation unit 15. As the voice related to the treatment, it is necessary to obtain at least the voice in which the classification indicated by the classification information records the doctor's utterance. Furthermore, voices other than the above may be acquired as needed.
　機器信号取得部１３は、治療工程推定の対象となる治療に関わる医療電子機器の機器信号を一つ以上取得し、工程情報推定部１５に供給する。機器信号取得部１３では、取得した機器信号に前述の機器信号付帯情報が含まれていない場合、この信号は何の機器の信号であり、何を意味するのかを示す機器信号付帯情報を付随する。 The device signal acquisition unit 13 acquires one or more device signals of the medical electronic device related to the treatment to be subjected to the treatment process estimation, and supplies the one or more device signals to the process information estimation unit 15. When the acquired device signal does not include the aforementioned device signal incidental information, the device signal acquiring portion 13 attaches the device signal incidental information indicating what device the signal is and what it means. .
　治療工程表取得部１４は、推定対象の治療の治療工程表を一つ取得し、工程情報推定部１５に供給する。治療工程表には、少なくとも、治療工程名、治療工程における使用器具の情報、治療工程における電子機器の情報を含むものとする。治療工程表は、キーボードやタッチパネルにより治療工程推定装置１に直接入力されたものを、治療工程取得部１４で取得しても良い。また、別の装置に入力された治療工程表の情報を、治療工程取得部１４で取得しても良い。治療工程表の取得は、治療行為が実際行われるよりも先に行われても良いし、後で行われても良い。 The treatment process chart acquisition unit 14 acquires one treatment process chart of the treatment to be estimated, and supplies the process process estimation unit 15 with it. The treatment schedule includes at least a treatment step name, information on an apparatus used in the treatment step, and information on an electronic device in the treatment step. The treatment process schedule may be acquired by the treatment process acquisition unit 14 that has been directly input to the treatment process estimation apparatus 1 by a keyboard or a touch panel. Further, the information on the treatment process chart input to another device may be acquired by the treatment process acquisition unit 14. Acquisition of the treatment schedule may be performed before or after the treatment operation is actually performed.
　工程情報推定部１５は、入力される映像、音声、機器信号、治療工程表より、工程情報を推定し、工程情報蓄積部１６に供給する。ここで、工程情報とは、治療工程表に記された各治療工程について、その治療工程の開始時刻と終了時刻、および、治療工程名を含む情報である。工程情報推定部１５の詳細は後述する。 The process information estimation unit 15 estimates the process information from the input video, audio, device signal, and treatment process chart, and supplies the process information to the process information storage unit 16. Here, the process information is information including the start time and end time of the treatment process and the treatment process name for each treatment process described in the treatment process chart. Details of the process information estimation unit 15 will be described later.
　工程情報蓄積部１６は、工程情報推定部１５で推定された工程情報を蓄積する。工程情報蓄積部１６は、ハードディスク、光ディスク、フラッシュメモリなどの記憶媒体で構成される。工程情報蓄積部１６は、ネットワークで接続されていても良い。工程情報蓄積部１６に記録される工程情報は、治療行為に関わる映像を閲覧する際に、頭出しや、映像検索を行うためのインデックス情報として扱われても良い。ここで、映像検索とは治療工程名による検索であっても良いし、時刻による検索であっても良い。 The process information storage unit 16 stores the process information estimated by the process information estimation unit 15. The process information storage unit 16 is configured of a storage medium such as a hard disk, an optical disk, or a flash memory. The process information storage unit 16 may be connected by a network. The process information stored in the process information storage unit 16 may be treated as index information for performing cueing or video search when viewing a video related to a treatment action. Here, the image search may be a search based on a treatment process name or a search based on time.
　（工程情報推定部の構成）
　次に、治療工程推定装置１に含まれる工程情報推定部１５について説明する。工程情報推定部１５は入力される治療工程表と映像、音声、機器信号から工程情報を推定し、出力する。工程情報の推定においては、必ずしも映像、音声、機器信号全ての情報が揃っている必要はない。 (Configuration of process information estimation unit)
Next, the process information estimation part 15 contained in the treatment process estimation apparatus 1 is demonstrated. The process information estimation unit 15 estimates and outputs process information from the input treatment process chart, video, audio, and device signal. In the estimation of the process information, it is not always necessary to have all the information of video, audio and device signals.
　図１を用いて工程情報推定部１５の詳細構成を説明する。図１は工程情報推定部１５の構成例を示した機能ブロック図である。図１に示すように、工程情報推定部１５は、確定可能治療工程決定部１５１、特徴抽出補助情報取得部１５２、特徴情報抽出部１５３、治療工程推定部１５４を備える。 The detailed configuration of the process information estimation unit 15 will be described with reference to FIG. FIG. 1 is a functional block diagram showing a configuration example of the process information estimation unit 15. As shown in FIG. 1, the process information estimation unit 15 includes a determinable treatment process determination unit 151, a feature extraction auxiliary information acquisition unit 152, a feature information extraction unit 153, and a treatment process estimation unit 154.
　　［確定可能治療工程決定部１５１］
　確定可能治療工程決定部１５１は、入力される治療工程表、および、映像、音声、機器信号から、確定的に決定できる治療工程（確定可能治療工程）の開始時刻と終了時刻、すなわち工程情報を決定し、決定した１以上の工程情報を、特徴抽出補助情報取得部１５２、特徴情報抽出部１５３、および、治療工程推定部１５４に供給する。確定可能治療工程とは、機器稼働信号のみで決定できる治療工程である。具体的には、最初と最後の作業がそれぞれ電子機器の操作となっている治療工程である。図３の治療工程表を参照し、確定可能治療工程の具体例を説明する。図３にある治療工程のうち、「麻酔」、「心膜切開」、「人工心肺稼働」、「心臓癒着部切除」および「人工心肺停止」の治療工程は、その治療工程で行われる最初と最後の作業が、共に電子機器の操作となっている。このため、上記作業が行われた時刻は、機器稼働信号より取り出せる。各治療工程の最初と最後の作業が行われる時刻を取り出せるため、当該治療工程の開始時刻と終了時刻は確定できる。 [Determinable treatment process determination unit 151]
The determinable treatment process determination unit 151 determines the start time and end time of the treatment process (determinable treatment process) that can be determined deterministically from the input treatment process chart and the video, audio, and device signal, that is, the process information The determined and determined one or more pieces of process information are supplied to the feature extraction auxiliary information acquiring unit 152, the feature information extracting unit 153, and the treatment process estimating unit 154. The determinable treatment process is a treatment process that can be determined only by the device operation signal. Specifically, the first and last operations are the treatment steps in which the electronic device is operated. A specific example of the determinable treatment process will be described with reference to the treatment process chart of FIG. Among the treatment steps shown in FIG. 3, the treatment steps "anesthesia", "pericardiotomy", "articardiopulmonary operation", "heart adhesion ablation" and "articardiopulmonary arrest" are the first to be performed in the treatment step. The last work is both the operation of the electronic device. Therefore, the time when the above work is performed can be taken out from the device operation signal. Since the time when the first and last work of each treatment process is performed can be taken out, the start time and the end time of the treatment process can be determined.
　図３の治療工程表のうち、「肌切開」、「心膜切開」および「心臓癒着部切除」の治療工程では、電気メス切断モード、電気メス止血モードの順で電子機器が使用される。この場合、上記順序の機器信号が検出されたとしても、機器信号だけでは、三つの治療工程の順序が確定しない。そこで、治療工程表と、機器信号に含まれる時刻情報より、上記の治療工程が行われる順序を判断する。すなわち、上記の順で機器信号が検出された場合、図３の治療工程表に記載された治療工程の順序と、機器信号の時刻情報を参照し、一番早い時刻は「肌切開」、次の時刻は「心膜切開」、その次の時刻は「心臓癒着部切除」の治療工程だと判断してよい。なお、「肌切開」の治療工程は、最初の作業が電子機器の操作ではないため、確定可能治療工程ではない。 In the treatment steps of “skin incision”, “pericardio incision” and “heart adhesion excision” in the treatment process chart of FIG. 3, electronic devices are used in the order of the electrosurgical cut mode and the electrosurgical hemostasis mode. In this case, even if the device signals in the above order are detected, the order of the three treatment steps can not be determined only by the device signals. Therefore, the order in which the above-described treatment steps are performed is determined from the treatment process chart and the time information included in the device signal. That is, when the device signal is detected in the above order, referring to the sequence of treatment steps described in the treatment procedure chart of FIG. 3 and the time information of the device signal, the earliest time is "skin incision", next The time may be determined to be a treatment process of “pericardiotomy” and the next time “cardiac adhesion removal”. In addition, since the treatment process of "skin incision" is not operation of an electronic device for the first work, it is not a determinable treatment process.
　上述の確定可能治療工程決定部１５１の動作は、次のように表現することもできる。すなわち、確定可能治療工程決定部１５１は、まず、治療工程表に基づいて、治療工程表に含まれる治療工程の中から、確定可能治療工程を決定する。続いて、それぞれに確定可能治療工程について開始時刻と終了時刻、および、治療工程名を含む情報を導出して工程情報として出力する。 The operation of the above-described determinable treatment process determination unit 151 can also be expressed as follows. That is, the determinable treatment process determination unit 151 first determines the determinable treatment process from among the treatment processes included in the treatment process flowchart based on the treatment process chart. Subsequently, information including the start time and end time and the treatment step name is derived for each determinable treatment step, and is output as step information.
　　［特徴抽出補助情報取得部１５２］
　特徴抽出補助情報取得部１５２は、入力される確定可能治療工程の工程情報と、治療工程表に基づき、入力される映像、音声、機器信号から特徴抽出補助情報を取り出し、特徴情報抽出部１５３に供給する。特徴抽出補助情報は、特徴情報抽出部１５３において映像、音声、機器信号から特徴抽出を行う際、補助情報として用いられる。特徴抽出補助情報の内容および導出処理の詳細は後述する。 [Feature extraction auxiliary information acquisition unit 152]
The feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input video, audio and device signal based on the process information of the determinable treatment process input and the treatment process chart, and the feature information extraction unit 153 Supply. The feature extraction auxiliary information is used as auxiliary information when the feature information extraction unit 153 extracts features from video, audio, and device signals. The contents of the feature extraction auxiliary information and the details of the derivation process will be described later.
　　［特徴情報抽出部１５３］
　特徴情報抽出部１５３は、入力される特徴抽出補助情報と映像、音声、機器信号、治療工程表、および、確定可能治療工程の工程情報に基づき、治療工程を推定するための手掛かりとなる特徴情報を取り出し、治療工程推定部１５４に供給する。特徴情報抽出処理の詳細は後述する。 [Feature information extraction unit 153]
The feature information extraction unit 153 is a feature information to be a clue for estimating the treatment process based on the input feature extraction auxiliary information, the video, the audio, the device signal, the treatment process chart, and the process information of the determinable treatment process. Are supplied to the treatment process estimation unit 154. Details of the feature information extraction process will be described later.
　　［治療工程推定部１５４］
　治療工程推定部１５４は、入力される特徴情報と治療工程表、確定可能治療工程決定部で決定された工程情報に基づき、確定可能治療工程以外の治療工程の工程情報を推定し、その工程情報を確定可能治療工程決定部１５１で決定された工程情報と足し合わせ、工程情報推定部１５から出力する。ここで、確定可能治療工程以外の治療工程とは、治療工程表に記載の治療工程の内、確定可能治療工程決定部１５１で決定されていない治療工程を指す。すなわち、治療工程推定部１５４では、治療工程表に記載の治療工程から、確定可能治療工程決定部１５１で既に決定された治療工程を除いた残りの治療工程について、工程情報を推定する。 [The treatment process estimation unit 154]
The treatment process estimation unit 154 estimates the process information of the treatment process other than the determinable treatment process based on the input feature information and the treatment process table and the process information determined by the determinable treatment process determination unit, and the process information Are added together with the process information determined by the determinable treatment process determination unit 151, and output from the process information estimation unit 15. Here, the treatment process other than the determinable treatment process refers to a treatment process that is not determined by the determinable treatment process determination unit 151 among the treatment processes described in the treatment schedule. That is, in the treatment process estimation unit 154, process information is estimated for the remaining treatment processes excluding the treatment process already determined by the determinable treatment process determination unit 151 from the treatment processes described in the treatment schedule.
　治療工程推定部１５４では、確定可能治療工程決定部１５１で決定された工程情報に基づき、工程情報が決定されていない時刻について、治療工程を推定する。すなわち、確定可能治療工程決定部１５１において、確定可能治療工程が行われていると判断された時刻について、工程情報の推定を行う必要はない。図４を参照して具体的に言うと、確定可能治療工程である「麻酔」の治療工程が行われていた１４：００から１４：３０までの時刻と、同じく確定可能治療工程である「心膜切開」から「人工心肺停止」の治療工程が行われていた１４：５５から１６：５０までの時刻については、治療工程推定部１５４では治療工程の推定を行う必要はない。 The treatment process estimation unit 154 estimates a treatment process at a time when process information is not determined based on the process information determined by the determinable treatment process determination unit 151. That is, it is not necessary to estimate the process information at the time when the determinable treatment process determination unit 151 determines that the determinable treatment process is being performed. Specifically speaking with reference to FIG. 4, the time from 14:00 to 14:30 at which the treatment process of “anaesthesia” which is a determinable treatment process was performed, and “heart” which is also a determinable treatment process The treatment process estimation unit 154 does not have to estimate the treatment process for the time from 14:55 to 16:50 at which the treatment process of “cardiotomy” to “cardiopulmonary arrest” was performed.
　図３の治療工程の内、「肌切開」の治療工程について、具体的な工程推定の順序を説明する。「肌切開」の治療工程では、メス、ピンセット、電気メス切断モード、電気メス止血モードの順序で、器具が使用される。このため、上記の器具が上記の順序で使用されていると、特徴情報から判断できれば、「肌切開」の治療工程を推定できる。すなわち、順に、執刀医の指示音声や術部の映像よりメスが使用されていると検出され、同様にピンセットが使用されていると検出され、機器信号より電気メスが切断モードで使用されていると検出され、同様に電気メスが止血モードで使用されていると検出された場合、「肌切開」の治療工程が行われたと推定する。この場合、メスが使用され始めた時刻から、止血モードの電気メスの使用が終了した時刻まで、「肌切開」の治療工程が行われたとする。なお、必ずしも一つの治療工程内の全ての器具が検出される必要はない。上記で言えば、例えば、ピンセットの使用が検出されずとも、「肌切開」の治療工程が行われたと判断してもよい。 Among the treatment steps of FIG. 3, a specific order of step estimation will be described for the treatment step of “skin incision”. In the "skin incision" treatment process, the instruments are used in the order of a scalpel, forceps, an electrocautery cutting mode, and an electrocautery hemostasis mode. Therefore, if it can be judged from the feature information that the above-mentioned instruments are used in the above-mentioned order, the treatment process of "skin incision" can be estimated. That is, it is detected that the scalpel is used from the instruction voice of the surgeon and the image of the operation part in order, and it is similarly detected that the tweezers are used, and the electrosurgical knife is used in the cutting mode from the equipment signal If it is detected that the electrosurgical knife is being used in the hemostasis mode as well, it is presumed that the treatment process of "skin incision" has been performed. In this case, it is assumed that the treatment process of “skin incision” is performed from the time when the knife starts to be used to the time when the use of the electrosurgical unit in the hemostasis mode is finished. Note that it is not necessary that all instruments in one treatment process be detected. In the above, for example, it may be determined that the treatment process of "skin incision" has been performed even if the use of tweezers is not detected.
　また、既に確定可能治療工程決定部１５１で決定された工程情報、および、治療工程推定部１５４で推定された治療工程、治療工程表より、ある時刻に存在しうる特徴情報が限定される場合、特徴情報の抽出において、抽出され得ない結果を排除しても良い。例えば、図３の治療工程表で言うと、「人工心肺稼働」の治療工程の終了時刻と、「人工心肺停止」の治療工程の開始時刻が既に決定されている場合、それらの時刻の間に使用される器具はハサミ、ピンセット、電気メスのみとなる。使用される器具名を特徴情報とする場合、上記時刻において、上記の器具名のみが、使用される器具名を示す特徴情報になり得るとして、上記以外の器具名を特徴情報として扱わなくても良い。同様に、医師の発声内容を特徴情報とする場合に、医師が発声し得る音声が限定されるならば、それ以外の音声を特徴情報として扱わないものとしても良い。 Also, when the process information determined by the determinable treatment process determination unit 151, the treatment process estimated by the treatment process estimation unit 154, and the feature information that may exist at a certain time based on the treatment schedule are limited, In extracting feature information, results that can not be extracted may be excluded. For example, in the treatment process chart of FIG. 3, if the end time of the treatment process of "artificial cardiopulmonary operation" and the start time of the treatment process of "artificial cardiopulmonary arrest" have already been determined, the time is between them. The only instruments used are scissors, tweezers and electric scalpels. When the instrument name to be used is the feature information, only the above instrument name can be the feature information indicating the instrument name to be used at the above time, even if the instrument name other than the above is not treated as the feature information good. Similarly, when speech content of a doctor is used as the feature information, if speech that the doctor can utter is limited, other speech may not be treated as the feature information.
　また、治療工程の推定にあたり、既に確定可能治療工程決定部１５１で決定された工程情報、および、治療工程推定部１５４で推定された工程情報、治療工程表より、治療工程を推定してよい。例えば、図３の治療工程表で言うと、「人工心肺稼働」の治療工程の終了時刻と、「人工心肺停止」の治療工程の開始時刻が既に決定されている場合、それらの時刻の間に「心臓癒着部切除」の治療工程が行われたと判断してもよい。 Further, in estimating the treatment process, the treatment process may be estimated from the process information already determined by the determinable treatment process determination unit 151, the process information estimated by the treatment process estimation unit 154, and the treatment process table. For example, in the treatment process chart of FIG. 3, if the end time of the treatment process of "artificial cardiopulmonary operation" and the start time of the treatment process of "artificial cardiopulmonary arrest" have already been determined, the time is between them. It may be determined that the treatment process of “heart adhesion ablation” has been performed.
　また、同様の順序で器具を使用する治療工程が二つ以上ある場合、それらの治療工程の順序は、治療工程表に記載の治療順序に則ってよい。例えば、図３の治療工程表で言うと、「心膜縫合」と「肌縫合」の治療工程では、同様の順序で器具が使用されるが、両者が検出された場合、時刻の早い方を「心膜縫合」、遅い方を「肌縫合」だと判断してもよい。 Also, when there are two or more treatment steps using the devices in the same order, the order of the treatment steps may be in accordance with the treatment order described in the treatment schedule. For example, referring to the treatment process chart of FIG. 3, in the treatment steps of “pericardial suture” and “skin suture”, instruments are used in the same order, but when both are detected, the one with the earlier time is used. "Pericardial suture", it may be determined that the slower one is "skin suture".
　なお、ここで例に挙げて説明はしないが、確定可能治療工程決定部１５１で決定された工程情報を治療工程推定部１５４に供給せず、治療工程推定部１５４で、確定可能治療工程の工程情報を決定し、残りの治療工程について工程情報を推定し、出力しても良い。この場合、確定可能治療工程決定部１５１で決定された工程情報と、治療工程推定部１５４で推定された工程情報を足し合わせる必要はなく、治療工程推定部１５４で決定、および、推定された工程情報を、工程情報推定部１５の出力として良い。 Although not described here as an example, the process information determined by the determinable treatment process determination unit 151 is not supplied to the treatment process estimation unit 154, and the process of the determinable treatment process is performed by the treatment process estimation unit 154. Information may be determined, and process information may be estimated and output for the remaining treatment processes. In this case, there is no need to add the process information determined by the determinable treatment process determination unit 151 and the process information estimated by the treatment process estimation unit 154, and the process determined and estimated by the treatment process estimation unit 154. The information may be the output of the process information estimation unit 15.
　（工程情報推定部の動作フロー）
　工程情報推定部１５では以下の手順により工程情報を推定する。まず、確定可能治療工程決定部１５１において、入力される機器稼働信号と治療工程表より、確定可能治療工程の工程情報を決定し、特徴抽出補助情報取得部１５２、特徴情報抽出部１５３、治療工程推定部１５４に供給する。 (Operation flow of process information estimation unit)
The process information estimation unit 15 estimates the process information according to the following procedure. First, in the determinable treatment process determination unit 151, the process information of the determinable treatment process is determined from the input device operation signal and the treatment process table, and the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, the treatment process The information is supplied to the estimation unit 154.
　次に特徴抽出補助情報取得部１５２において、入力される工程情報と治療工程表、および、映像、音声、機器信号から特徴抽出補助情報を取り出し、特徴情報抽出部１５３に供給する。 Next, the feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input process information, the treatment process chart, the video, the audio, and the device signal, and supplies it to the feature information extraction unit 153.
　次に特徴情報抽出部１５３において、入力される特徴抽出補助情報、治療工程表、および、確定可能治療工程の工程情報に基づき、入力される映像、音声、機器信号の特徴情報を取り出し、治療工程推定部１５４に供給する。 Next, the feature information extraction unit 153 extracts the feature information of the input video, voice, and device signal based on the feature extraction auxiliary information, the treatment process chart, and the process information of the determinable treatment process, and the treatment process The information is supplied to the estimation unit 154.
　次に治療工程推定部１５４において、入力される特徴情報と治療工程表、確定可能治療工程決定部１５１で決定された工程情報に基づき、確定可能治療工程決定部１５１で決定されていない工程情報を推定し、出力する。 Next, in the treatment process estimation unit 154, the process information not determined by the determinable treatment process determination unit 151 based on the input characteristic information and the treatment process table, and the process information determined by the determinable treatment process determination unit 151. Estimate and output.
　最後に、確定可能治療工程決定部１５１で決定された工程情報と、治療工程推定部１５４の工程情報を足し合わせた工程情報を、工程推定部１５から出力し、工程情報推定を終了する。 Finally, the process information obtained by adding the process information determined by the determinable treatment process determination unit 151 and the process information of the treatment process estimation unit 154 is output from the process estimation unit 15, and the process information estimation ends.
　図４を参照し、工程情報推定部１５における治療工程推定のフローを具体的に説明する。図４は、図３の治療工程表が工程情報推定部１５に入力された場合に、各構成要素から出力される工程情報を示した図である。図４において、確定可能治療工程決定部の列は、確定可能治療工程決定部１５１から出力される工程情報を示す。治療工程推定部の列、工程推定部の列も同様である。また、それぞれの列に含まれる工程名の列と時刻の列は、その構成要素から出力される工程情報に含まれる治療工程名、および、治療工程の開始時刻と終了時刻を示す。 The flow of treatment process estimation in the process information estimation unit 15 will be specifically described with reference to FIG. 4. FIG. 4 is a diagram showing process information output from each component when the treatment process chart of FIG. 3 is input to the process information estimation unit 15. In FIG. 4, the row of the determinable treatment process determination unit indicates process information output from the determinable treatment process determination unit 151. The same applies to the row of the treatment process estimation unit and the sequence of the process estimation unit. The row of process names and the row of time included in each row indicate the treatment process name included in the process information output from the component, and the start time and end time of the treatment process.
　工程情報推定部１５では、まず、確定可能治療工程決定部１５１において、機器稼働信号で決定できる確定可能治療工程の工程情報が決定される。すなわち、図４の確定可能治療工程決定部の列に示すように、「麻酔」、「心膜切開」、「人工心肺稼働」、「心臓癒着部切除」、「人工心肺停止」の治療工程名と、治療工程の開始時刻と終了時刻が決定され、工程情報として出力される。次に、上記で決定された工程情報について、特徴抽出補助情報取得部１５２で特徴抽出補助情報が検出される。次に、その特徴抽出補助情報と治療工程表、および、確定可能治療工程の工程情報に基づき、特徴情報抽出部１５３で特徴情報が抽出される。次に、特徴情報と治療工程表、および、確定可能治療工程決定部１５１で決定された工程情報に基づき、確定可能治療工程決定部１５１で決定されていない治療工程の工程情報が推定される。すなわち、図４の治療工程推定部の列に示すように、「消毒」、「肌切開」、「胸骨切開」、「心膜縫合」、「胸骨縫合」、「肌縫合」の治療工程名と、治療工程の開始時刻と終了時刻が推定され、工程情報として出力される。なお、前述の通り、治療工程推定部１５４における治療工程推定では、確定可能治療工程決定部１５１で決定された治療工程が行われる時刻については工程情報の推定を行う必要はない。最後に、図４の工程推定部の列に示すように、確定可能治療工程決定部１５１で決定された工程情報と、治療工程推定部１５４で推定された工程情報を足し合わせた、全ての治療工程についての工程情報が、工程推定部１５から出力される。なお、図４では治療工程の開始時刻と終了時刻を、分単位までの表記で示しているが、時刻が分単位までと制限されることはない。 In the process information estimation unit 15, firstly, the determinable treatment process determination unit 151 determines process information of a determinable treatment process that can be determined by the device operation signal. That is, as shown in the row of the determinable treatment process determination unit in FIG. 4, the treatment process names of "anesthesia", "pericardiotomy", "artificial cardiopulmonary operation", "heart adhesion ablation", and "artificial cardiopulmonary arrest" The start time and the end time of the treatment process are determined and output as process information. Next, the feature extraction auxiliary information acquisition unit 152 detects feature extraction auxiliary information for the process information determined above. Next, the feature information extraction unit 153 extracts feature information based on the feature extraction auxiliary information, the treatment process chart, and the process information of the determinable treatment process. Next, based on the feature information and the treatment schedule, and the process information determined by the determinable treatment process determination unit 151, the process information of the treatment process not determined by the determinable treatment process determination unit 151 is estimated. That is, as shown in the column of the treatment process estimation unit in FIG. 4, the treatment process names of “disinfection”, “skin incision”, “sternum incision”, “pericardial suture”, “sternum suture”, and “skin suture” The start time and the end time of the treatment process are estimated and output as process information. As described above, in the treatment process estimation in the treatment process estimation unit 154, it is not necessary to estimate the process information at the time when the treatment process determined by the determinable treatment process determination unit 151 is performed. Finally, as shown in the column of the process estimation unit of FIG. 4, all treatments in which the process information determined by the determinable treatment process determination unit 151 and the process information estimated by the treatment process estimation unit 154 are added. Process information on the process is output from the process estimation unit 15. In addition, although the start time and completion | finish time of a treatment process are shown by the description to a unit of minute in FIG. 4, time is not restrict | limited to a unit of minute.
　（特徴抽出補助情報取得部１５２の詳細）
　続いて、特徴抽出補助情報取得部１５２の詳細を説明する。前述の通り、特徴抽出補助情報取得部１５２は、入力される工程情報と治療工程表に基づき、入力される映像、音声、機器信号から特徴抽出補助情報を取り出し、特徴情報抽出部１５３に供給する。 (Details of Feature Extraction Auxiliary Information Acquisition Unit 152)
Subsequently, the details of the feature extraction auxiliary information acquisition unit 152 will be described. As described above, the feature extraction auxiliary information acquisition unit 152 extracts the feature extraction auxiliary information from the input video, audio, and device signal based on the input process information and the treatment process chart, and supplies it to the characteristic information extraction unit 153. .
　（特徴抽出補助情報取得のフロー）
　図５を参照して、特徴抽出補助情報取得のフローを説明する。図５は、特徴抽出補助情報取得部１５２の動作フローの一例を示す図である。特徴抽出補助情報取得は、以下のＳ１５１～Ｓ１５５の手順で順次実行される。 (Flow of feature extraction auxiliary information acquisition)
The flow of feature extraction auxiliary information acquisition will be described with reference to FIG. FIG. 5 is a diagram showing an example of an operation flow of the feature extraction auxiliary information acquisition unit 152. As shown in FIG. The feature extraction auxiliary information acquisition is sequentially performed in the following steps S151 to S155.
　(Ｓ１５１)確定可能治療工程決定部１５１で決定された工程情報の中から、特徴抽出補助情報が未検出の治療工程を一つ選択して対象治療工程に設定する。次にＳ１５２を実行する。 (S151) From the process information determined by the determinable treatment process determination unit 151, one treatment process whose feature extraction auxiliary information is not detected is selected and set as a target treatment process. Next, S152 is executed.
　(Ｓ１５２)特徴抽出補助情報部１５２に入力された映像、音声、機器信号の情報から、特徴抽出補助情報が未検出の情報を一つ選択して対象情報に設定する。次にＳ１５３を実行する。 (S152) From the information of the video, audio, and device signal input to the feature extraction auxiliary information unit 152, one piece of information for which the feature extraction auxiliary information is not detected is selected and set as target information. Next, S153 is executed.
　(Ｓ１５３)上記対象治療工程と対象情報について、特徴抽出補助情報を取得する。特徴抽出補助情報には、サンプル情報と処理設定情報が含まれる。以下に、サンプル情報取得と処理設定情報取得の方法を順に説明する。なお、サンプル情報の取得と処理設定情報の取得は並行して行われても良いし、順に行われても良い。また、順に行われる場合、どちらを先に行っても良い。また、どちらか一方のみが行われても良い。次にＳ１５４を実行する。 (S153) The feature extraction auxiliary information is acquired for the target treatment process and the target information. The feature extraction auxiliary information includes sample information and process setting information. Below, the method of sample information acquisition and process setting information acquisition is demonstrated in order. The acquisition of the sample information and the acquisition of the processing setting information may be performed in parallel or sequentially. Moreover, when performed in order, you may perform whichever first. Also, only one of them may be performed. Next, S154 is executed.
　　［サンプル情報取得］
　上記対象治療工程と対象情報について、サンプル情報を取得し、特徴抽出補助情報とする。サンプル情報は、特定時刻、もしくは、特定期間に対応するメディアデータ（映像、音声、機器信号）のサンプル値の一部から構成される情報である。工程情報推定部１５で扱うサンプル情報とは、確定可能治療工程決定部１５１において決定された治療工程について、その治療工程が行われた時刻もしくはその近傍の時刻における、特徴抽出補助情報取得部１５２に入力された治療工程表、および、映像、音声、機器信号のサンプル値を集めたものであり、特徴情報抽出部１５３で参照される。サンプル情報は、特定の時刻の周囲状況を表す情報として利用できる。周囲状況とは、例えば、医師が出す指示の音声や、未使用の器具の数などである。異なる治療工程であっても、特定の周囲状況が一致する状況が生じ得る。そのような場合に、確定的に決定できる治療工程において観察された周囲状況とサンプル情報の対応関係は、同一の周囲状況で発生する別の治療工程の推定時に活用できる。工程情報推定部１５の場合、特徴情報抽出部１５３において、特定の周囲状況を表すサンプル情報と同一もしくは近いサンプル値が検出された場合、同じ周囲状況が発生していると判断する。これにより、特徴抽出の精度を向上できる。サンプル情報取得の具体例を以下に示す。 [Get sample information]
Sample information is acquired for the target treatment process and the target information, and is used as feature extraction auxiliary information. The sample information is information that is configured by a part of sample values of media data (video, audio, device signal) corresponding to a specific time or a specific period. The sample information handled by the process information estimation unit 15 refers to the feature extraction auxiliary information acquisition unit 152 at the time when the treatment process was performed or the time near the time when the treatment process was determined by the determinable treatment process determination unit 151. It is a collection of input treatment process charts, and video, audio, and sample values of device signals, and is referred to by the feature information extraction unit 153. The sample information can be used as information representing the surrounding situation at a specific time. The surrounding situation is, for example, a voice of an instruction issued by a doctor, the number of unused devices, and the like. Even different treatment steps may result in situations where certain circumstances are consistent. In such a case, the correspondence between the surrounding situation and the sample information observed in the treatment process that can be determined definitively can be utilized in estimating another treatment process occurring in the same surrounding situation. In the case of the process information estimation unit 15, the feature information extraction unit 153 determines that the same surrounding situation is occurring when a sample value identical or close to the sample information representing the specific surrounding situation is detected. Thereby, the accuracy of feature extraction can be improved. The specific example of sample information acquisition is shown below.
　　［サンプル情報の具体例］
　　１．術部映像の色情報
　サンプル情報の例としては、術部映像の色情報がある。図３の治療工程表にある「人工心肺稼働」の治療工程を例に説明する。人工心肺は切開された術部にとりつけられるため、「人工心肺稼働」の治療工程が行われている時は、術部は切開されている。このため、上記治療工程が行われている時の術部映像には、切開された術部が記録されている。ここから、「人工心肺稼働」の治療工程が対象工程となっている場合、術部映像を対象情報と設定し、その色情報をサンプル情報として取り出す。このサンプル情報は、術部が切開されているという周囲状況を表す。この情報に基づき、特徴情報抽出部１５３では、術部映像において、当該サンプル情報と一致もしくは類似した色情報が検出された場合、その時刻は術部が切開されている周囲状況が発生していると判断する。同様に、「麻酔」の治療工程が行われている時は、術部は閉じられている。このため、上記治療工程が行われている時の術部映像の色情報を、術部が閉じられている周囲状況を示すサンプル情報とする。この情報に基づき、特徴情報抽出部１５３では、術部映像において、当該サンプル情報と一致もしくは類似した色情報が検出された場合、その時刻は術部が閉じられている周囲状況が発生していると判断する。 [Specific example of sample information]
1. Color Information of Surgical Site Video As an example of sample information, there is color information of surgical site video. The treatment process of "artificial cardiopulmonary operation" in the treatment process chart of FIG. 3 will be described as an example. Since the heart-lung machine is attached to the incised operative part, the operative part is incised when the treatment process of "artificial cardio-pulmonary operation" is performed. For this reason, the operation part incised is recorded on the operation part image | video when the said treatment process is performed. From here, when the treatment process of "artificial cardiopulmonary operation" is a target process, the operation site image is set as target information, and the color information is extracted as sample information. This sample information represents the surrounding situation that the operative site is incised. When color information matching or similar to the sample information is detected in the operation site video based on this information, the surrounding situation in which the operation site is incised occurs at the time. I will judge. Similarly, when the "anesthetic" treatment process is being performed, the operative part is closed. Therefore, the color information of the operation site image when the above-mentioned treatment process is performed is set as sample information indicating the surrounding situation in which the operation section is closed. When color information matching or similar to the sample information is detected in the operation site video based on this information, the surrounding situation in which the operation site is closed occurs at that time. I will judge.
　また、この時、手術開始直後の切開されていない術部映像の色情報と、上記サンプル情報を比較することで、術部の領域を判断できる。両画像間で色情報に大きく差がある領域は、切開され体内が露出している領域だと言える。特徴情報抽出部１５３において、術部映像を処理する場合、処理範囲を当該領域に絞ることで、処理対象を術部のみに制限し、それ以外の領域を処理対象から除外することで誤検出を抑止しても構わない。上記説明は、次のようにも表現できる。すなわち、２つの異なる時刻において特定の周囲状況（前述の説明においては術部の切開状況）が変化する場合、両時刻のサンプル情報を比較し、一定以上の差分が発生する領域を比較することで、周囲状況の変化により影響を受けるサンプル情報の部分集合を特定できる。 At this time, the area of the operative site can be determined by comparing the color information of the unincised surgical site image immediately after the start of the operation with the sample information. It can be said that the area where there is a large difference in color information between the two images is the area in which the body is incised and exposed. When processing a surgical site video in the feature information extraction unit 153, the processing target is limited to the surgical part only by narrowing the processing range to the relevant area, and the false detection is eliminated by excluding the other areas from the processing target. You may suppress it. The above description can also be expressed as follows. That is, when a specific surrounding situation (in the above description, the incision situation of the operative part) changes at two different times, sample information of both times is compared, and a region where a difference of a certain level or more occurs is compared. , A subset of sample information affected by changes in ambient conditions can be identified.
　　２．医師の指示音声
　また、例えば、医師のピンセットを要求する指示音声をサンプル情報とする。つまり、医師がピンセットを要求している周囲状況を示すサンプル情報として音声のサンプル情報を用いる。図３の治療工程表に記載の「人工心肺停止」の治療工程は、確定可能治療工程決定部１５１で決定される治療工程である。治療工程表より、「人工心肺停止」の治療工程の直後に、「心膜縫合」の治療行為でピンセットが使用されることが分かる。従って、「人工心肺停止」の終了時刻より後に、一番初めに記録される音声は、医師がピンセットを要求する音声だと判断できる。ここから、「人工心肺停止」が対象工程である場合、医師の指示音声を記録した音声を対象情報とし、その音声情報の中で、対象工程が終了した時刻より後に、一番初めに検出される音声パターンをサンプル情報とする。当該サンプル情報は、上記した医師の指示音声を表している。すなわち、医師がピンセットを要求する音声パターンを表しているので、特徴情報抽出部１５３において、当該サンプル情報と一致もしくは類似した音声パターンが検出された場合、医師がピンセットを要求する周囲状況が発生していると判断する。 2. Doctor's Instruction Voice In addition, for example, the instruction voice requesting the doctor's tweezers is used as sample information. That is, voice sample information is used as sample information indicating the surrounding situation where the doctor is requesting tweezers. The treatment process of “cardiopulmonary arrest” described in the treatment schedule of FIG. 3 is a treatment process determined by the definable treatment process determination unit 151. From the treatment schedule, it can be seen that tweezers is used in the treatment operation of "pericardial suture" immediately after the treatment step of "cardiopulmonary arrest". Therefore, it can be determined that the voice recorded first at the end of the "cardiopulmonary arrest" end time is a voice for which the doctor requests tweezers. From here, when "artificial cardiopulmonary arrest" is the target process, the voice recording the instruction voice of the doctor is set as the target information, and the voice information is first detected after the time when the target process is finished. Voice pattern as sample information. The sample information represents the doctor's instruction voice described above. That is, since the voice pattern for which the doctor requests tweezers is represented, when the voice information pattern that matches or is similar to the sample information is detected in the feature information extraction unit 153, the surrounding situation for which the doctor requests tweezers occurs. It is determined that
　　［処理設定情報取得］
　対象治療工程に関する上記対象情報について、処理設定情報を取得し、特徴抽出補助情報とする。処理設定情報とは、特徴情報抽出部１５３において、映像、音声、機器信号の情報から特徴情報を抽出する特徴情報抽出処理の制御情報である。特徴情報抽出処理の制御情報には画像処理のパラメータが含まれる。画像処理のパラメータには、例えば画像の二値化処理における閾値がある。特徴情報抽出処理の結果は、その制御情報の設定により異なる。工程推定の精度を高めるためには、有効な特徴抽出を行う必要があり、すなわち、有効な特徴を抽出できる制御情報の設定が必要となるが、全ての制御情報について手動で設定を行うのは困難であり、また、適当な値に制御情報を設定したとして、有効な特徴抽出ができるとは限らない。 [Process setting information acquisition]
Process setting information is acquired about the said object information regarding an object treatment process, and it is set as feature extraction auxiliary information. The process setting information is control information of a feature information extraction process for extracting feature information from information of video, audio, and device signal in the feature information extraction unit 153. Control information of the feature information extraction process includes parameters of image processing. The parameters of the image processing include, for example, a threshold value in the binarization processing of the image. The result of the feature information extraction processing differs depending on the setting of the control information. In order to improve the accuracy of process estimation, it is necessary to perform effective feature extraction, that is, setting of control information that can extract effective features is required, but manually setting of all control information is necessary. It is difficult, and setting control information to an appropriate value does not necessarily enable effective feature extraction.
　そこで、治療工程表と上記対象治療工程を使用し、最適な制御情報の設定を検出する。治療工程表より、対象治療工程が行われた瞬間もしくはその近傍の時刻の周囲の状況は判断できる。上記時刻の対象情報について、特徴情報抽出部１５３と同様の特徴情報抽出処理を、制御情報の設定を変更しながら繰り返し行う。これにより得た複数の特徴情報について、治療工程表から判断される周囲の状況と、最も近しい特徴情報を抽出した制御情報を、処理設定情報として取得する。これにより得た処理設定情報を用いることで、より実際に近い特徴の抽出が行えるため、特徴抽出の精度を向上できる。 Therefore, the setting of the optimal control information is detected using the treatment schedule and the target treatment step. From the treatment process chart, it is possible to judge the situation around the moment when the target treatment process was performed or in the vicinity thereof. For the target information at the time, feature information extraction processing similar to that of the feature information extraction unit 153 is repeatedly performed while changing the setting of control information. About the several characteristic information obtained by this, the control information which extracted the surrounding condition judged from the treatment process chart, and the nearest characteristic information is acquired as process setting information. By using the process setting information obtained in this way, it is possible to extract features that are closer to actuality, so that the accuracy of feature extraction can be improved.
　上記の処理設定情報取得について、具体例を挙げて説明する。対象治療行為において、その治療行為もしくは前後の治療行為で、手術器具が使用されると治療工程表に記載されている場合、当該治療行為の瞬間もしくはその近傍の時刻の術部映像に特徴情報検出処理を加え、その医療器具が使用されたと判断する制御情報の設定を取り出す。 The above-described process setting information acquisition will be described with a specific example. In the target treatment practice, if it is described in the treatment flow chart that the surgical instrument is used in the treatment act or the treatment act before and after, feature information is detected in the operation part image of the time of the moment of the treatment act or its vicinity Add processing and take out the setting of control information that determines that the medical device has been used.
　例えば、図３の治療工程表では、「人工心肺稼働」の治療工程の直後に使用される器具はハサミとされている。従って、「人工心肺稼働」の治療工程が終了時刻以降で、一番早い時刻に検出される器具はハサミになると判断できる。ここから、「人工心肺稼働」の治療工程の終了時刻以降の時刻について、制御設定を変更しながら繰り返し画像処理と器具の検出を行い、その時刻で使用されている器具がハサミである可能性を、最も高く検出した制御設定を取り出し、ハサミ検出に用いる処理設定情報とする。以下に処理設定情報取得の一例を示す。 For example, in the treatment process chart of FIG. 3, the device used immediately after the “cardiopulmonary operation” treatment process is scissors. Therefore, it can be determined that the instrument detected at the earliest time is the scissors after the end time of the treatment process of "artificial cardiopulmonary operation". From here, for the time after the end time of the treatment process of "artificial cardio-pulmonary operation", the image processing and the device detection are repeated while changing the control setting, and it is possible that the device used at that time is scissors. The control setting detected the highest is taken out, and is taken as processing setting information used for scissors detection. An example of processing setting information acquisition is shown below.
　　［処理設定情報取得の具体例］
　　１．直線検出の処理設定情報取得
　手術器具の判別は様々な方法で行えるが、ここでは、特徴抽出補助情報取得部１５２に入力される術部映像から手術器具を識別する方法を考える。術部映像に映るものは、主に患者の体、医師の手、手術器具である。この内、直線的な形状をしているのは手術器具のみである。従って、術部映像に直線が存在するか否かを判別することで、手術器具が使用されているかどうかを検出できる。また、手術においてよく使用されるメス、ハサミ、ピンセットは全て二つの直線で構成されており、その構成は、それぞれ並行、交差、広がりとなっている。従って、二本の直線の相対位置を検出することで、手術器具の種類を識別できる。なお、開胸器など術部に長時間設置される器具も直線として検出されるが、長時間移動しない直線を処理対象から除外することで誤検出を抑止できる。 [Specific example of processing setting information acquisition]
1. Acquisition of Processing Setting Information of Straight Line Detection Although determination of a surgical instrument can be performed by various methods, here, a method of identifying a surgical instrument from a surgical site image input to the feature extraction auxiliary information acquisition unit 152 will be considered. What appears in the operation site video are mainly the patient's body, the hands of the doctor, and the surgical instruments. Of these, only the surgical instrument has a linear shape. Therefore, whether or not the surgical instrument is used can be detected by determining whether or not a straight line exists in the surgical site image. In addition, the scalpel, scissors and tweezers often used in surgery are all composed of two straight lines, and their configurations are parallel, crossing and spreading respectively. Therefore, the type of surgical instrument can be identified by detecting the relative position of the two straight lines. In addition, although the instrument installed in the operation part, such as a chest opener, is also detected as a straight line, an erroneous detection can be suppressed by excluding the straight line which does not move for a long time from a process target.
　直線検出には、例えばハフ変換が用いられる。術部映像にハフ変換を行うには、術部映像に二値化処理を施す必要がある。特徴情報抽出部１５３でも同様である。この場合、処理情報の設定としては、二値化処理では二値の閾値、ハフ変換では直線だと判断する閾値、変換画素ピッチがある。これらの設定によっては、手術器具を正しく識別できない可能性がある。そこで、上記処理情報の設定を変化させながら繰り返しハフ変換を行い、使用されている器具を識別する。全ての結果を比較し、治療工程表に記載されている使用器具と一致もしくは最も近い結果を出した処理情報の設定値を、処理設定情報として取り出す。この処理設定情報を特徴情報抽出部１５３において利用することで、より実際に近い検出結果を得られる。 For example, Hough transform is used for straight line detection. In order to perform Hough transform on the operation site image, it is necessary to perform binarization processing on the operation site image. The same applies to the feature information extraction unit 153. In this case, as setting of the processing information, there are a binary threshold value in the binarization processing, a threshold value determined to be a straight line in the Hough transform, and a conversion pixel pitch. Depending on these settings, the surgical instrument may not be identified correctly. Therefore, the Hough transform is repeatedly performed while changing the setting of the processing information to identify the device being used. All the results are compared, and the setting value of the processing information which gives the result that is the same as or closest to the used instrument described in the treatment schedule is taken out as the processing setting information. By using the process setting information in the feature information extraction unit 153, a detection result closer to actual can be obtained.
　(Ｓ１５４)特徴抽出補助情報取得部１５２に入力された全ての映像、音声、機器信号情報について、特徴抽出補助情報取得が完了しているかを判定する。完了している場合（Ｓ１５４yes）、Ｓ１５５を実行する。完了していない場合（Ｓ１５４no）、特徴抽出補助情報が未検出の映像、音声、機器信号の情報について対象情報選択Ｓ１５２から処理を行う。 (S154) It is determined whether acquisition of feature extraction auxiliary information has been completed for all video, audio, and device signal information input to the feature extraction auxiliary information acquisition unit 152. If it is completed (S154 yes), S155 is executed. If not completed (S154 no), processing is performed from target information selection S152 on the information of the video, audio, and device signal of which feature extraction auxiliary information is not detected.
　(Ｓ１５５)確定可能治療工程決定部１５１で決定された全ての治療工程について、特徴抽出補助情報の検出が完了しているかを判定する。完了している場合（Ｓ１５５yes）、特徴抽出補助情報取得を終了する。完了していない場合（Ｓ１５５no）、特徴抽出補助情報が未検出の治療工程について対象工程選択Ｓ１５１から処理を行う。 (S155) It is determined whether detection of feature extraction auxiliary information is complete for all treatment steps determined by the determinable treatment step determination unit 151. When it is completed (S155 yes), the feature extraction auxiliary information acquisition is ended. If not completed (S155 no), processing is performed from target process selection S151 for a treatment process in which the feature extraction auxiliary information is not detected.
　（特徴情報抽出部１５３の詳細）
　続いて、特徴情報抽出部１５３の詳細を説明する。前述の通り、特徴情報抽出部は、入力される特徴抽出補助情報と映像、音声、機器信号、治療工程表、および、確定可能治療工程の工程情報に基づき、特徴情報を抽出し、治療工程推定部１５４に供給する。ここで、特徴情報とは、映像、音声、機器信号からから特徴を抽出した情報であり、治療工程推定部１５４において治療工程の推定のために利用される。特徴情報は、あらかじめ設定された特徴抽出時刻ごとに抽出される。ただし、映像、音声、機器信号のサンプルが存在する全ての時刻において、特徴情報の抽出をする必要はない。例えば、特徴抽出時刻を映像のフレーム毎の時刻に設定しても良いし、２秒に一度など、既定の間隔に設定しても良い。また、特徴情報の抽出において、必ずしも全ての特徴情報を抽出する必要はない。例えば、音声が記録されていない時刻について特徴情報を抽出する場合、音声から抽出される特徴情報（例えば後述する医師の指示音声）を抽出しなくとも良い。映像、機器信号でも同様である。以下では、特定の特徴抽出時刻における特徴抽出処理の説明を行う場合、その特徴抽出時刻のことを特徴抽出対象時刻と呼ぶ。 (Details of feature information extraction unit 153)
Subsequently, details of the feature information extraction unit 153 will be described. As described above, the feature information extraction unit extracts feature information based on input feature extraction auxiliary information and video, audio, device signal, treatment process chart, and process information of determinable treatment process, and estimates treatment process It supplies to the part 154. Here, the feature information is information obtained by extracting features from video, audio, and device signals, and is used in the treatment process estimation unit 154 to estimate the treatment process. Feature information is extracted at each preset feature extraction time. However, it is not necessary to extract feature information at all times when samples of video, audio and device signals exist. For example, the feature extraction time may be set to the time of each frame of the video, or may be set to a predetermined interval such as once every two seconds. In addition, in the extraction of the feature information, it is not necessary to extract all the feature information. For example, when extracting feature information at a time when a voice is not recorded, it is not necessary to extract feature information (for example, an instruction voice of a doctor described later) extracted from the voice. The same applies to video and device signals. Hereinafter, when the feature extraction processing at a specific feature extraction time is described, the feature extraction time is referred to as a feature extraction target time.
　特徴情報抽出部１５３では、各特徴抽出対象時刻で特徴情報を抽出し、治療工程推定部１５４に供給する。特徴情報抽出部１５３では、特徴情報の抽出に特徴抽出補助情報を利用する。ただし、必ずしも特徴情報の抽出に特徴抽出補助情報を利用する必要はない。特徴情報の例を以下に示す。 The feature information extraction unit 153 extracts feature information at each feature extraction target time, and supplies the feature information to the treatment process estimation unit 154. The feature information extraction unit 153 uses feature extraction auxiliary information to extract feature information. However, it is not necessary to use feature extraction auxiliary information for feature information extraction. An example of feature information is shown below.
　＜特徴情報１：術部の切開状況＞
　特徴情報としては、例えば、手術における術部の切開状況がある。すなわち、術部を撮影した映像の色情報より、術部に肌色成分が多ければ術部は非切開であり、赤色が多ければ術部は切開されていると判断し、その切開／非切開の情報を特徴情報とする。 <Characteristic information 1: Incision situation of operation part>
As feature information, there is, for example, an incision status of a surgical site in surgery. That is, from the color information of the image obtained by photographing the operative site, if the operative site has more skin color components, the operative site is non-incising, and if the red is more, it is determined that the operative site is incised. Information is used as feature information.
　上記判断において、特徴抽出補助情報に含まれる術部映像の色情報のサンプル情報を利用する。このサンプル情報には、切開された術部の周囲状況を示すサンプル情報と、閉じられた術部の周囲状況を示すサンプル情報が含まれる。特徴抽出対象時刻における術部を撮影した映像の色情報が、切開された術部を示すサンプル情報と一致もしくは類似している場合、術部は切開されていると判断する。閉じられた術部を示すサンプル情報と一致もしくは類似した場合も同様である。 In the above determination, sample information of the color information of the surgical site image included in the feature extraction auxiliary information is used. The sample information includes sample information indicating the surrounding situation of the incision site and sample information indicating the surrounding situation of the closed surgery part. When the color information of the image obtained by photographing the operation part at the feature extraction target time matches or is similar to the sample information indicating the incision part, it is determined that the operation part is incised. The same is true for cases where the information matches or is similar to the sample information indicating the closed operation site.
　＜特徴情報２：医師の指示内容＞
　また、例えば、医師の発声を記録した音声に基づき検出された指示内容を特徴情報とする。すなわち、執刀医がメスなどの器具を渡すように助手に指示した場合、その器具名を特徴情報とする。この特徴情報より、執刀医は指示後に当該器具を使用すると推測できる。 <Feature information 2: Doctor's instruction content>
Further, for example, the instruction content detected based on the voice recording the doctor's utterance is used as the feature information. That is, when the surgeon instructs the assistant to deliver an instrument such as a scalpel, the instrument name is used as the feature information. From this feature information, it can be inferred that the surgeon uses the device after the instruction.
　上記の推測において、特徴抽出補助情報に含まれる医師の指示音声のサンプル情報を利用する。このサンプル情報は、医師が器具を要求する周囲状況を示す。特徴抽出対象時刻における医師の発声を記録した音声から、上記サンプル情報と一致もしくは類似した音声パターンが検出された場合、上記サンプル情報が示す周囲状況が発生していると判断する。すなわち、その周囲状況が示す器具が要求されたと判断する。 In the above estimation, the sample information of the instruction voice of the doctor included in the feature extraction auxiliary information is used. This sample information indicates the circumstances at which the physician requests the device. When a voice pattern matching or similar to the sample information is detected from the voice recording the doctor's utterance at the feature extraction target time, it is determined that the surrounding situation indicated by the sample information is generated. That is, it is determined that an appliance indicated by the surrounding situation has been requested.
　上記特徴情報１と特徴情報２は、まとめて次のようにも表現できる。すなわち、特徴抽出補助情報に特定の周囲状況を表すサンプル情報が含まれている場合に、特徴抽出対象時刻におけるサンプル情報と、前記周囲状況を表すサンプル情報を比較して一致性または類似性を評価することで、特徴抽出対象時刻においてある周囲状況が発生しているか否かを判定できる。 The feature information 1 and the feature information 2 can be collectively expressed as follows. That is, when the feature extraction auxiliary information includes sample information representing a specific surrounding situation, the sample information at the feature extraction target time is compared with the sample information representing the surrounding situation to evaluate the match or similarity. By doing this, it can be determined whether or not a certain surrounding situation occurs at the feature extraction target time.
　＜特徴情報３：医療器具置き場の状況＞
　また、例えば、医療器具の置き場を撮影した映像から推定される使用器具名を特徴情報とする。すなわち、医療器具の置き場を撮影した映像より、残りの器具の数、減った器具の個数を判別し、その時刻に使用されている器具を推定し、その器具名を特徴情報とする。上記器具の判別は、パターン認識による形状識別により行われても良いし、別の方法でもよい。治療工程表には、各治療行為について、使用されるメス等の手術器具の種類と順序が記載されている。このため、何の手術器具が使用されているのか、手術開始から順に追うことができれば、使用中の手術器具の種類、それまでに使用された手術器具の数から、現在の治療工程を推定できる。このため、器具置き場に置かれた残りの器具の個数を調べることで、今まで何の器具がどれだけ使用されたか判断できる。この情報からその時刻に使用されている器具の種類を推測し、その器具名を特徴情報とする。 <Feature Information 3: Status of Medical Device Storage>
In addition, for example, a used instrument name estimated from an image obtained by photographing a place of a medical instrument is used as the feature information. That is, the number of remaining devices and the number of reduced devices are determined from the image of the storage place of the medical device, the device used at that time is estimated, and the device name is used as the feature information. The determination of the device may be performed by shape identification by pattern recognition, or another method may be used. The treatment schedule describes the type and sequence of surgical instruments such as a scalpel used for each treatment action. For this reason, if it is possible to follow what kind of surgical instruments are used sequentially from the start of the surgery, it is possible to estimate the current treatment process from the type of surgical instruments being used and the number of surgical instruments used so far. . For this reason, it is possible to determine what amount of equipment has been used so far by examining the number of the remaining equipment placed in the equipment storage area. From this information, the type of device used at that time is inferred, and the device name is used as feature information.
　＜特徴情報４：術部で使用されている器具＞
　また、例えば、術部を撮影した映像から推定される使用器具名を特徴情報とする。すなわち、術部を撮影した映像より、各時刻に術部で使用されている器具を推定し、その器具名を特徴情報とする。上記使用されている器具の識別は、パターン認識による形状識別で行われても良いし、直線認識による識別でも良い。 <Feature information 4: Instruments used in the operation department>
In addition, for example, a used instrument name estimated from a video obtained by photographing an operation part is set as feature information. That is, an instrument used at the operation site is estimated at each time from the image obtained by photographing the operation site, and the instrument name is used as feature information. The identification of the used instrument may be performed by shape identification by pattern recognition or by linear recognition.
　直線認識による器具の識別において、特徴抽出補助情報に含まれる直線検出の処理設定情報を利用する。この処理設定情報には、直線検出による器具の識別を、好適に行える設定情報が含まれている。このため、特徴抽出対象時刻における術部を撮影した映像に対して、この処理設定情報に基づいた直線検出を適用することで、処理設定情報を使わない場合に比べ、より正確な器具の識別が可能となる。 In the identification of an instrument by straight line recognition, processing setting information of straight line detection included in the feature extraction auxiliary information is used. The process setting information includes setting information that can suitably identify the device by linear detection. For this reason, by applying straight line detection based on the processing setting information to an image obtained by photographing the operation part at the feature extraction target time, more accurate identification of the device is possible compared to the case where the processing setting information is not used. It becomes possible.
　＜特徴情報５：電子機器の使用状況＞
　また、例えば、各種電子機器の使用状況を特徴情報とする。すなわち、機器設定信号を特徴情報とする。例えば、心肺停止でない場合に、生体モニタに表示される心拍数の波形形状のような、判断基準が一意に定まらない機器設定信号を特徴情報とする。ただし、治療工程表に当該波形を取り出す指示があった場合に限る。 <Feature information 5: Usage of electronic devices>
Also, for example, the usage status of various electronic devices is used as feature information. That is, the device setting signal is used as feature information. For example, when cardiopulmonary arrest is not performed, a device setting signal such as a waveform shape of a heart rate displayed on a living body monitor and for which a determination criterion is not uniquely determined is used as feature information. However, this is limited to the case where there is an instruction to take out the waveform in the treatment schedule.
　上記で例に挙げた特徴情報はあくまで特徴情報の一例であり、上記以外の特徴情報が使用されても構わない。 The feature information exemplified above is merely an example of the feature information, and feature information other than the above may be used.
　特徴情報抽出部１５３では、確定可能治療工程決定部１５１で決定された工程情報に基づき、工程情報が決定されていない時刻についてのみ特徴情報を抽出しても良い。すなわち、確定可能治療工程決定部１５１において、確定可能治療工程が行われていると決定された時刻では、特徴情報の抽出を行う必要はない。それにより特徴情報抽出処理の処理量を低減できる。図４を参照して具体的に言うと、確定可能治療工程である「麻酔」の治療工程が行われていた１４：００から１４：３０までの時刻と、同じく確定可能治療工程である「心膜切開」から「人工心肺停止」の治療工程が行われていた１４：５５から１６：５０までの時刻については、特徴抽出対象時刻から除外しても良い。 The feature information extraction unit 153 may extract feature information only at times when the process information is not determined based on the process information determined by the determinable treatment process determination unit 151. That is, it is not necessary to extract feature information at the time when it is determined in the determinable treatment process determination unit 151 that the determinable treatment process is being performed. Thereby, the processing amount of the feature information extraction processing can be reduced. Specifically speaking with reference to FIG. 4, the time from 14:00 to 14:30 at which the treatment process of “anaesthesia” which is a determinable treatment process was performed, and “heart” which is also a determinable treatment process The time from 14:55 to 16:50 at which the treatment process from “membrane incision” to “cardiopulmonary arrest” is performed may be excluded from the feature extraction target time.
　また、確定可能治療工程決定部１５１で決定された治療工程と治療工程表より、ある時刻に抽出され得る特徴情報が限定される場合、特徴情報の抽出において、抽出され得ない結果を排除しても良い。抽出対象を絞り込むことで、より正確に特徴情報を抽出できる。例えば、図３の治療工程表で言うと、「人工心肺稼働」の治療工程の終了時刻と、「人工心肺停止」の治療工程の開始時刻が既に決定されている場合、それらの時刻の間に使用される器具はハサミ、ピンセット、電気メスのみとなる。使用される器具名を特徴情報とする場合、上記時刻において、上記の器具名のみが、使用される器具名を示す特徴情報になり得るとして、上記以外の器具名を特徴情報として扱わなくても良い。 Also, when the feature information that can be extracted at a certain time is limited from the treatment process and the treatment schedule determined by the determinable treatment process determination unit 151, excluding the result that can not be extracted in the extraction of feature information Also good. By narrowing down the extraction target, feature information can be extracted more accurately. For example, in the treatment process chart of FIG. 3, if the end time of the treatment process of "artificial cardiopulmonary operation" and the start time of the treatment process of "artificial cardiopulmonary arrest" have already been determined, the time is between them. The only instruments used are scissors, tweezers and electric scalpels. When the instrument name to be used is the feature information, only the above instrument name can be the feature information indicating the instrument name to be used at the above time, even if the instrument name other than the above is not treated as the feature information good.
　上記のように、時刻毎に抽出され得る特徴情報を示す情報を、特徴抽出制限情報とする。特徴抽出制限情報は、映像、音声、機器信号のサンプルが存在する全ての時刻において検出されても良いし、映像のフレーム毎の時刻に検出されても良いし、２秒に一度など、既定の間隔で検出されても良い。また、特徴抽出対象時刻に従って検出されても良い。 As described above, information indicating feature information that can be extracted for each time is taken as feature extraction restriction information. The feature extraction restriction information may be detected at all times when video, audio, or a sample of an equipment signal is present, may be detected at each time of a frame of the video, or may be predetermined at, for example, once every two seconds. It may be detected at intervals. Also, it may be detected according to the feature extraction target time.
　なお、ここで例に挙げて説明はしないが、確定可能治療工程決定部１５１で決定された工程情報を、特徴情報抽出部１５３に供給せず、特徴情報抽出部１５３において機器稼働信号を判別し、確定可能治療工程と、確定可能治療工程が行われていると判断された時刻を検出しても良い。 Although not described here as an example, the process information determined by the decidable treatment process determination unit 151 is not supplied to the feature information extraction unit 153, and the feature information extraction unit 153 determines the device operation signal. The determinable treatment process and the time when it is determined that the determinable treatment process is performed may be detected.
　（工程情報推定部の効果）
　以上説明した工程情報推定部１５は、入力された治療工程表と映像、音声、機器信号より、確定可能治療工程を判別し、そこから、サンプル情報と処理設定情報からなる特徴抽出補助情報を取り出し、その情報を利用して入力された映像、音声、機器信号より特徴情報抽出を行い、当該特徴情報を用いて治療工程推定を行う機能を備えている。 (Effect of process information estimation unit)
The process information estimation unit 15 described above discriminates a determinable treatment process from the input treatment process chart, video, audio, and device signal, and extracts feature extraction auxiliary information including sample information and process setting information therefrom. It has a function of extracting feature information from video, audio and device signals input using the information, and performing treatment process estimation using the feature information.
　特徴情報抽出にサンプル情報を利用することで、抽出した特徴情報が何を意味するのか、またその判断基準は何かを、実際の治療を記録した情報から取り出せるため、これらを予め設定する必要のある従来技術に比べ、特徴情報をより有効に利用できる。 By using sample information for feature information extraction, it is necessary to pre-set what the extracted feature information means, and also for the judgment criteria, because it can be taken out from the information that records the actual treatment. The feature information can be used more effectively than certain prior art.
　特徴情報抽出に処理設定情報を利用することで、特徴情報抽出の制御設定を予め設定する必要がある従来技術に比べ、より好適な制御設定を使用できるため、より有効な特徴情報抽出が可能となる。 By using processing setting information for feature information extraction, more preferable control setting can be used as compared with the prior art in which control setting for feature information extraction needs to be set in advance, and thus more effective feature information extraction is possible. Become.
　このため、従来の映像、音声、機器信号から特徴情報抽出を行い、治療工程推定を行う技術に比べ、特徴情報抽出に特徴抽出補助情報を利用することで、より有効な特徴情報抽出が行える。従って、より精度の高い治療工程推定が可能となる。 For this reason, more effective feature information extraction can be performed by using the feature extraction auxiliary information for feature information extraction, as compared with the conventional technique of extracting feature information from video, audio, and device signals and estimating treatment steps. Therefore, more accurate treatment process estimation is possible.
　〔付記事項１〕
　前述の治療工程推定装置１の説明では治療行為で定義された全ての治療工程を導出するとして装置の説明をしたが、必ずしも全ての治療工程を導出する必要はなく、一部の治療工程のみを導出する装置であってもよい。 [Appendix 1]
Although the above description of the treatment process estimation apparatus 1 has described the apparatus as deriving all treatment steps defined in the treatment practice, it is not necessary to derive all treatment steps, and only some treatment steps It may be an apparatus for deriving.
　〔付記事項２〕
　前述の治療工程推定装置１の説明では、予め記録の完了した、治療行為に関するメディアデータについて、当該治療行為の工程情報を推定するとして装置の説明をしたが、必ずしも予め記録の完了したメディアデータを扱う必要はなく、治療行為に関するメディアデータの記録と並行して、当該治療行為の工程情報を推定する装置であっても良い。 [Appendix 2]
In the description of the treatment process estimation apparatus 1 described above, the apparatus has been described on the assumption that the process information of the treatment action is estimated for the media data regarding the treatment action, the recording of which is completed in advance. It is not necessary to handle, and it may be an apparatus that estimates process information of the treatment action in parallel with the recording of media data regarding the treatment action.
　この場合、ある時刻に、工程情報推定部１５において、工程情報を推定する場合、その時刻より先の時刻のメディアデータを参照しないものとする。すなわち、工程情報の推定に未来の時刻のメディアデータを扱わない。具体的には、特徴抽出補助情報部１５２において、特徴抽出補助情報を取得する際、現在の時刻よりも過去の時刻のメディアデータのみを参照する。この場合、特徴抽出補助情報は、任意のタイミングで取得され直しても良い。 In this case, when the process information estimation unit 15 estimates the process information at a certain time, it is assumed that the media data of the time earlier than that time is not referred to. That is, the media data of the future time is not handled for estimation of process information. Specifically, when acquiring the feature extraction auxiliary information in the feature extraction auxiliary information unit 152, only media data of a time past the current time is referred to. In this case, the feature extraction auxiliary information may be acquired again at an arbitrary timing.
　また、ある時刻に、治療工程推定部１５４において工程情報を推定する場合、その時刻よりも未来の時刻のメディアデータは参照しないものとする。この場合、前述の治療工程推定装置１の説明と異なり、現在行われていると推定される治療工程に限り、その治療工程の終了時刻を推定する必要はなく、かつ工程情報にその治療工程の終了時刻を含めなくとも良い。確定可能治療工程決定部１５１における、確定可能治療工程の工程情報の決定についても同様である。ただし、終了時刻が確定していない治療工程は、治療工程を確定的に決定できない可能性があるため、確定可能治療工程として扱わないものとする。図４の工程情報を参照して具体的に説明する。1４：４６に、治療工程推定部１５４において、治療工程の推定を行う際、「消毒」の治療工程の終了時刻が推定されていれば、治療工程表より、次に行われる治療工程は「肌切開」の治療工程だと判断できる。そして、特徴情報よりメスの使用が検知されれば、「肌切開」の治療工程が行われていると推定する。 Also, when the treatment process estimation unit 154 estimates the process information at a certain time, media data of a time later than that time is not referred to. In this case, unlike the description of the treatment process estimation apparatus 1 described above, it is not necessary to estimate the end time of the treatment process only for the treatment process estimated to be currently performed, and the process information It is not necessary to include the end time. The same applies to the determination of the process information of the determinable treatment process in the determinable treatment process determination unit 151. However, a treatment process whose end time has not been determined should not be treated as a determinable treatment process because there is a possibility that the treatment process can not be determined definitively. This will be specifically described with reference to the process information of FIG. At 14: 46, when the treatment process estimation unit 154 estimates the treatment process, if the end time of the “disinfection” treatment process is estimated, the treatment process to be performed next is “skin” according to the treatment process chart. It can be judged that it is a treatment process of “incision” And if use of a scalpel is detected from feature information, it will presume that the treatment process of "skin incision" is performed.
　治療行為に関するメディアデータの記録と並行して、当該治療行為の工程情報を推定する場合、現在行われていると推定される治療工程の名称と詳細（名称、使用器具の順序など）や、次に行われると治療工程表に記載されている治療工程の名称や詳細を、映像や音声を用い、医師や医療スタッフに提示しても良い。なお、上記の説明では、現在の治療工程を推定するとして説明したが、必ずしも現在の治療工程について、工程情報を推定する必要はなく、治療工程の開始時刻と終了時刻を推定できる治療工程についてのみ、工程情報を推定しても良い。 When estimating the process information of the treatment action in parallel with the recording of media data regarding the treatment action, the name and details of the treatment step (name, order of instruments used, etc.) presumed to be currently performed, and the next If performed, the names and details of the treatment steps described in the treatment step chart may be presented to a doctor or medical staff using video and audio. In the above description, although the present treatment process is described as being estimated, the process information need not necessarily be estimated for the current treatment process, and only the treatment process which can estimate the start time and the end time of the treatment process. The process information may be estimated.
　治療行為に関するメディアデータの記録と並行して、当該治療行為の工程情報を推定することで、治療行為の完了と同時に、当該治療行為の工程情報推定を完了できる。すなわち、治療行為の記録完了後に治療工程推定を行う場合に比べ、より早い時刻に工程情報を出力できる。また、医師や医療スタッフに、現在の治療工程を提示することで、治療の進捗状況を確認できる。同様に、次に行われる治療工程の情報を提示することで、治療工程表が手元になくとも、次に行われる治療行為を確認できる。 By estimating the process information of the therapeutic action in parallel with the recording of the media data regarding the therapeutic action, it is possible to complete the process information estimation of the therapeutic action simultaneously with the completion of the therapeutic action. That is, the process information can be output at an earlier time than when the treatment process estimation is performed after the recording of the treatment action is completed. Also, by presenting the current treatment process to the doctor or medical staff, the progress of treatment can be confirmed. Similarly, by presenting information on the treatment process to be performed next, it is possible to confirm the next treatment action even if the treatment schedule is not at hand.
　〔付記事項３〕
　前述の治療工程推定装置１の説明では、治療行為の工程情報を推定するとして説明しているが、工程推定の対象は必ずしも治療行為に制限されない。すなわち、治療行為でない行為について、工程情報を推定する装置であっても良い。具体的には、治療行為ではないある行為について、当該行為に関するメディアデータ、および、治療工程表と同等の情報を持つ、当該工程に関する工程表（工程名、工程における使用器具の詳細、工程における電子機器の詳細）がある場合、治療行為の工程情報の推定と同様に、当該行為に関する工程情報の推定を行っても良い。以下に治療行為でない行為の一例を示す。 [Appendix 3]
Although the above description of the treatment process estimation apparatus 1 is described as estimating the process information of the treatment action, the target of the process estimation is not necessarily limited to the treatment action. That is, it may be an apparatus that estimates process information for an action that is not a treatment action. Specifically, a process schedule (process name, details of equipment used in process, process in process, electronic in process) having media data on the action and information equivalent to the treatment process chart, for an action that is not a treatment act When there is the details of the device, estimation of the process information related to the action may be performed as in the case of estimation of the process information of the treatment action. The following is an example of an act that is not a therapeutic act.
　治療行為ではない行為とは、例えば、演劇や音楽ライブ等のステージである。上記ステージは、一般に、予め工程の順序と詳細が定められた工程表が作成される。また、当該工程表には電子機器（スピーカ、照明など）の使用の詳細も記載されるため、治療行為の工程情報の推定と同様に、工程情報の推定が可能である。上記ステージについて推定した工程情報を利用することで、上記ステージを撮影した映像の閲覧が容易になる。これにより、例えば、上記ステージを撮影した映像を、工程（シーン）毎に分類し、任意のシーンにシークできる。 An action that is not a therapeutic action is, for example, a stage such as a play or live music. In the above stages, generally, a process chart in which the order and details of the processes are predetermined is created. In addition, since the details of the use of the electronic device (speaker, lighting, etc.) are also described in the process chart, it is possible to estimate the process information as in the case of the process information of the therapeutic action. By utilizing the process information estimated for the stage, it becomes easy to browse the video of the stage. Thus, for example, videos obtained by shooting the stage can be classified for each process (scene), and seek can be made to an arbitrary scene.
　また、例えば、製品の組み立てである。ただし、電子機器を利用するものに限る。一般に、製品の組み立ては、組み立て工程の順序と詳細が定められているため、治療行為の工程情報の推定と同様に、工程情報の推定が可能である。上記製品組み立てについて推定した工程情報を利用することで、上記製品組み立てを撮影した映像の閲覧が容易になる。これにより、例えば、上記製品組み立てを撮影した映像に、各時刻で行われている作業の名称を加えることで、教育用の映像として活用できる。また、各工程の作業時間を計測できる。 Also, for example, assembly of a product. However, it limits to what uses an electronic device. In general, since the assembly of the product is defined in the order and details of the assembly process, estimation of process information is possible as well as estimation of process information of a therapeutic action. By utilizing the process information estimated for the product assembly, it becomes easy to view the image of the product assembly. Thus, for example, by adding the name of the work being performed at each time to the image obtained by photographing the product assembly, it can be used as an educational image. Moreover, the working time of each process can be measured.
　〔付記事項４〕
　前述の治療工程推定装置１の説明では、工程情報を、治療工程の開始時刻と終了時刻、および、治療工程名を含む情報であるとして装置の説明をしたが、さらに、工程情報に重要度の情報を含む装置であっても良い。ここで、重要度の情報とは、工程情報を推定する各時刻において、その時刻におけるメディアデータの重要度を示す情報である。重要度の情報は、工程情報を推定する時刻毎に設定されても良いし、治療工程内の作業毎に設定されても良いし、治療工程ごとに設定されても良い。 [Appendix 4]
Although the apparatus has been described on the assumption that the process information is information including the start time and the end time of the treatment process and the treatment process name in the above description of the treatment process estimation apparatus 1, It may be an apparatus containing information. Here, the information on the importance is information indicating the importance of media data at each time at which the process information is estimated. The information on the degree of importance may be set for each time at which the process information is estimated, may be set for each operation in the treatment process, or may be set for each treatment process.
　重要度の情報は、予め治療工程表に記載されていれば、その情報を参照して決定しても良いし、メディアデータから導出されても良い。例えば、図３の治療工程表で言うと、機器稼働信号より、人工心肺が稼働完了してから停止作業に入るまでの時刻(「人工心肺稼働」の治療工程の終了時刻から「人工心肺停止」の治療工程が開始される時刻まで)は、術部を撮影した映像に重要なシーンが映っていると判断し、その時刻の重要度を高く設定する。また、治療工程の推定結果から導出されても良い。例えば、作業時間の長い治療工程は、重要であると設定する。 The information on the degree of importance may be determined with reference to the information if it is described in advance in the treatment schedule, or may be derived from media data. For example, in the treatment process chart of FIG. 3, according to the device operation signal, the time from the completion of the operation of the artificial heart to the start of the stop work (the “cardiopulmonary stop from the end time of the treatment process of It is judged that the important scene is shown in the image which photographed the operation part, and the importance degree of the time is set high until the time when the treatment process is started. Also, it may be derived from the estimation result of the treatment process. For example, a treatment process with a long working time is set as important.
　工程情報に重要度の情報が含まれる場合、工程情報蓄積部１６に蓄積される工程情報は、重要度による映像検索を行うためのインデックスであっても良い。なお、必ずしも全ての時刻、または作業、または治療工程の工程情報に、重要度の情報が含まれる必要はなく、一部の上記工程情報にのみ、重要度の情報が含まれても良い。また、メディアデータごとに個別の重要度の情報を設定しても良いし、全てのメディアデータで共通の重要度の情報を設定しても良い。 When information on importance is included in the process information, the process information accumulated in the process information accumulation unit 16 may be an index for performing a video search based on the importance. The information on the importance does not necessarily have to be included in the process information of all the times or work or treatment processes, and the information on the importance may be included in only a part of the process information. Further, information of individual importance may be set for each media data, or information of common importance may be set for all media data.
　工程情報に重要度の情報を含むことで、重要度による映像検索が可能となり、映像の確認を、より簡易にできる。 By including the information of the degree of importance in the process information, it becomes possible to search for a video according to the degree of importance, and the confirmation of the video can be made easier.
　〔付記事項５〕
　前述の治療工程推定装置１の説明では、治療工程について工程情報を推定するとして説明しているが、治療工程を構成する作業（器具の使用、電子機器の使用）について工程情報を推定する装置であっても良い。すなわち、治療工程を構成する作業毎に、開始時刻と終了時刻を推定し、作業名の情報と合わせ、工程情報としても良い。 [Item 5]
In the above description of the treatment process estimation apparatus 1, the treatment process is described as estimating the process information. However, the apparatus for estimating the process information with respect to the work (use of the instrument and use of the electronic device) constituting the treatment process It may be. That is, for each work constituting the treatment process, the start time and the finish time may be estimated, and may be combined with the information of the work name to be the process information.
　作業の工程情報の推定は、治療工程の工程情報の推定と同様に、工程情報推定部１５において行われる。治療工程を構成する作業は、治療工程と同様に、確定的に決定できる作業（確定可能作業）を含む。ここで、確定可能作業とは、電子機器を使用する作業であり、機器稼働信号によって工程情報を決定できる作業である。例えば、図３の治療工程表で言うと、「肌切開」の治療工程を構成する作業はメス、ピンセット（ガーゼ）、電気メス切断モード、電気メス止血モードである。この内、電気メス切断モード、電気メス止血モードの作業は、電子機器を使用する作業であり、機器稼働信号により作業の開始時刻と終了時刻を確定的に決定できるため、確定可能作業となる。以下で作業の工程情報の推定について説明する。 The estimation of the process information of the work is performed in the process information estimation unit 15 in the same manner as the estimation of the process information of the treatment process. The work that constitutes the treatment process includes, like the treatment process, a work that can be determined deterministically (determinable work). Here, the determinable work is a work using an electronic device, and is a work that can determine process information based on a device operation signal. For example, in the treatment process chart of FIG. 3, the work constituting the treatment process of “skin incision” is a scalpel, tweezers (gauze), electric scalpel cutting mode, and electric scalpel hemostasis mode. Among them, the operation of the electric scalpel cutting mode and the electric scalpel hemostasis mode is an operation using the electronic device, and since the start time and the end time of the operation can be determined definitely by the device operation signal, the operation can be determined. The estimation of work process information will be described below.
　作業の工程情報を推定する場合、まず確定可能治療工程決定部１５１において、確定可能作業の工程情報を決定する。すなわち機器稼働信号より工程情報を決定できる作業について、工程情報を決定する。なお、この際、確定可能治療工程でない治療工程を構成する作業であっても、確定可能作業であれば工程情報を決定する。例えば、図３の治療工程表で言うと、「肌切開」の治療工程は確定可能治療工程ではないが、この治療工程に含まれる確定可能作業（電気メス切断モード、電気メス止血モード）について工程情報を決定する。 When estimating the process information of the work, first, the determinable treatment process determination unit 151 determines the process information of the determinable work. That is, process information is determined for an operation that can determine the process information from the device operation signal. At this time, even if the operation constitutes a treatment process that is not a determinable treatment process, the process information is determined if it is an determinable operation. For example, in the treatment process chart of FIG. 3, although the treatment process of "skin incision" is not a determinable treatment process, it is a process about the determinable work (electric knife cutting mode, electric scalpel hemostasis mode) included in this treatment process. Determine the information.
　次に、治療工程推定部１５４において、特徴情報抽出部１５３で抽出された特徴情報に基づき、工程情報が決定されていない作業の工程情報を推定する。すなわち、確定可能作業ではない作業の工程情報を推定する。この時、治療工程の工程情報の推定と同様に、確定可能治療工程決定部１５１において、確定可能作業が行われていると判断された時刻について、作業の工程情報を推定する必要はない。また、治療工程の工程情報の推定と同様に、既に決定もしくは推定された作業の工程情報と治療工程表より、作業の工程情報を推定して良い。 Next, based on the feature information extracted by the feature information extraction unit 153, the treatment process estimation unit 154 estimates the process information of the work for which the process information is not determined. That is, the process information of the work which is not a determinable work is estimated. At this time, similarly to the estimation of the process information of the treatment process, it is not necessary to estimate the process information of the operation at the time when the determinable treatment process determination unit 151 determines that the determinable operation is performed. Further, similarly to the estimation of the process information of the treatment process, the process information of the operation may be estimated from the process information of the operation already determined or estimated and the treatment process chart.
　作業の工程情報を推定する場合、工程情報蓄積部１６に蓄積される工程情報は、作業名による映像検索を行うためのインデックスであっても良い。なお、必ずしも全ての作業の工程情報を導出する必要はなく、一部の作業の工程情報のみを導出しても良い。 When the process information of the work is estimated, the process information stored in the process information storage unit 16 may be an index for performing a video search by the work name. In addition, it is not necessary to necessarily derive the process information of all the operations, and only the process information of a part of the operations may be derived.
　工程情報に作業に関する工程情報を含むことで、作業名による映像検索が可能となり、映像の確認を、より簡易にできる。 By including the process information related to the work in the process information, the video search by the work name becomes possible, and the confirmation of the video can be simplified.
　〔変形例１〕
　前述の治療工程推定装置１の説明では、工程情報の推定の対象となる治療行為(推定対象治療行為)に関する情報のみを使用して、治療行為の工程情報を推定するとして説明しているが、当該治療行為以外の治療行為（非推定対象治療行為）の情報を利用して治療行為の工程情報を推定する装置であっても良い。すなわち、推定対象治療行為の治療工程情報の推定において、１つ以上の非推定対象治療行為に関する特徴情報抽出補助情報を、推定対象治療行為の特徴抽出を行う際に参照し、特徴情報を抽出する。そして、その特徴情報を用い、治療工程推定を行っても良い。なお、非推定対象治療行為とは、例えば、過去に行われた治療行為である。その場合、工程情報推定部１５には、推定済みの工程情報と、その導出時に用いた特徴抽出補助情報を記録する記録部を追加で備える。また、例えば、推定対象治療行為と並行して行われる治療行為である。 [Modification 1]
In the explanation of the treatment process estimation apparatus 1 described above, it is described that the process information of the treatment action is estimated using only the information on the treatment action (the estimation target treatment action) which is the target of the process information estimation. The apparatus may be an apparatus that estimates process information of a therapeutic action using information of a therapeutic action (non-estimated target therapeutic action) other than the therapeutic action. That is, in the estimation of the treatment process information of the estimation target treatment practice, the feature information extraction auxiliary information on one or more non-estimation target treatment practices is referred to when extracting the feature of the estimation target treatment practice, and the feature information is extracted. . Then, the treatment process may be estimated using the feature information. The non-estimated target treatment act is, for example, a treatment act performed in the past. In that case, the process information estimation unit 15 additionally includes a recording unit that records the estimated process information and the feature extraction auxiliary information used at the time of derivation thereof. Also, for example, it is a therapeutic action performed in parallel with the presumed target therapeutic action.
　非推定対象治療行為の特徴抽出補助情報は、過去に当該非推定対象治療行為の工程情報を推定した際に取得され、蓄積された情報であっても良いし、推定対象治療行為の特徴抽出補助情報を取得する際に、非推定対象治療行為に関する治療工程表とメディアデータから取得されても良い。 The feature extraction auxiliary information of the non-estimated target therapeutic action may be information acquired and accumulated when the process information of the non-estimated target therapeutic action is estimated in the past, or the feature extraction aid of the estimated target therapeutic action When acquiring information, it may be acquired from the treatment schedule and media data regarding the non-estimated target treatment practice.
　推定対象治療行為に関するメディアデータに機器稼働信号が含まれない場合など、特徴抽出補助情報が取得できない、もしくは取得が難しい場合であっても、上記のように、非推定対象治療行為の特徴抽出補助情報を利用することで、より有効な特徴情報抽出が行える。従って、より精度の高い治療工程推定が可能となる。 As described above, even if the feature extraction auxiliary information can not be acquired or acquisition is difficult, such as when the device operation signal is not included in the media data relating to the estimation target treatment, as described above, the feature extraction assistance of the non-estimation target treatment More effective feature information extraction can be performed by using the information. Therefore, more accurate treatment process estimation is possible.
　〔変形例２〕
　前述の治療工程推定装置１の説明では、推定した工程情報を工程情報蓄積部１６に蓄積するとして説明しているが、必ずしも工程情報を工程情報蓄積部１６に蓄積する必要はなく、工程情報を治療工程推定装置１から出力する装置であっても良い。 [Modification 2]
In the above description of the treatment process estimation apparatus 1, it is described that the estimated process information is stored in the process information storage unit 16, but it is not necessary to store the process information in the process information storage unit 16. It may be a device that outputs from the treatment process estimation device 1.
　この場合、治療工程推定装置１には、工程情報蓄積部１６を構成する記憶媒体が必ずしも必要でなくなる。また、推定した工程情報を、工程情報蓄積部１６を経由せず、出力することが可能となる。 In this case, the treatment process estimation apparatus 1 does not necessarily need a storage medium that configures the process information storage unit 16. Further, the estimated process information can be output without passing through the process information storage unit 16.
　〔変形例３〕
　図６を用いて特徴情報抽出部１５３の詳細構成を説明する。図６は特徴情報抽出部１５３の構成例を示した機能ブロック図である。図６に示すように、特徴情報抽出部１５３は、特徴抽出対象時刻設定部１５３１、特徴抽出制限情報設定部１５３２、メディアデータ処理部１５３３、メディアデータ比較部１５３４、特徴情報取得部１５３５を備える。 [Modification 3]
The detailed configuration of the feature information extraction unit 153 will be described with reference to FIG. FIG. 6 is a functional block diagram showing a configuration example of the feature information extraction unit 153. As shown in FIG. As shown in FIG. 6, the feature information extraction unit 153 includes a feature extraction target time setting unit 1531, a feature extraction restriction information setting unit 1532, a media data processing unit 1533, a media data comparison unit 1534, and a feature information acquisition unit 1535.
　特徴抽出対象時刻設定部１５３１は、入力される確定可能治療工程の工程情報と治療工程表に基づき、前述した特徴抽出対象時刻を設定し、その情報をメディアデータ処理部１５３３、および、メディアデータ比較部１５３４に供給する。 The feature extraction target time setting unit 1531 sets the feature extraction target time described above based on the input process information of the determinable treatment process and the treatment process chart, and the information is compared with the media data processing unit 1533 and media data comparison. It supplies to the part 1534.
　特徴抽出制限情報設定部１５３２は、入力される確定可能治療工程の工程情報と治療工程表に基づき、前述した特徴抽出制限情報を設定し、その情報をメディアデータ処理部１５３３、および、メディアデータ比較部１５３４に供給する。 The feature extraction restriction information setting unit 1532 sets the above-mentioned feature extraction restriction information based on the process information of the determinable treatment process and the treatment process chart inputted, and the information is compared with the media data processing unit 1533 and the media data comparison. It supplies to the part 1534.
　メディアデータ処理部１５３３は、入力される処理設定情報を含む特徴抽出補助情報、特徴抽出対象時刻情報に基づき、入力される映像、音声、機器信号に、画像処理などの処理を加え、その処理結果を特徴情報取得部１５３５に供給する。ここで、画像処理などの処理とは、特徴情報を抽出するためにメディアデータに対して行われる処理であり、例えば、術部を撮影した映像のフレームに対する二値化、ハフ変換、パターン認識、また、器具置き場を撮影した映像のフレームに対する二値化、パターン認識である。上記はあくまで一例であり、映像だけでなく、音声、機器信号に処理を行い、処理結果を取り出しても構わない。上記した各種処理の設定およびパラメータは、治療工程推定が行われる前に、予め設定される。ただし、メディアデータ処理部１５３３に入力される特徴抽出補助情報に、前述の処理設定情報が含まれる場合、当該処理設定情報を、各処理の設定、および、パラメータに反映する。なお、必ずしも全ての処理設定情報を反映する必要はない。処理結果には、その処理の対象となったメディアデータの時刻情報が付随される。 The media data processing unit 1533 adds processing such as image processing to the input video, audio, and device signal based on the feature extraction auxiliary information including the process setting information to be input and the feature extraction target time information, and the processing result Are supplied to the feature information acquisition unit 1535. Here, processing such as image processing is processing performed on media data to extract feature information, and, for example, binarization, Hough transform, pattern recognition on a frame of an image obtained by photographing a surgical site, In addition, it is binarization and pattern recognition for a frame of an image obtained by photographing an instrument storage area. The above is merely an example, and processing may be performed not only on video but also on audio and device signals, and processing results may be extracted. The settings and parameters of the various processes described above are preset before the treatment process estimation is performed. However, when the above-mentioned processing setting information is included in the feature extraction auxiliary information input to the media data processing unit 1533, the processing setting information is reflected on the setting of each processing and the parameter. Note that it is not necessary to reflect all the process setting information. The processing result is accompanied by time information of the media data that is the target of the processing.
　メディアデータ比較部１５３４は、入力されるサンプル情報を含む特徴抽出補助情報、特徴抽出対象時刻情報に基づき、入力される映像、音声、機器信号の情報を、基準サンプル情報と比較し、その比較結果を特徴情報取得部１５３５に供給する。基準サンプル情報とは、例えば、術部を撮影した映像に関しては、術部が切開されている時の術部の色情報である。この基準サンプル情報の色情報と、メディアデータに含まれる、術部を撮影した映像の色情報を比較し、どれだけ類似しているのかを数値で示したものを比較結果とし、その比較結果に基づき、特徴情報が決定される。この場合、特徴情報取得部１５３５において、上記比較結果が類似していれば、術部が切開されているとして特徴情報を設定する。類似していなければ、術部は閉じられているとして特徴情報を設定する、もしくは、基準サンプル情報と類似していないとして、特徴情報を設定しなくとも良い。上記した基準サンプル情報は、治療工程推定が行われる前に、予め設定される。ただし、メディアデータ処理部１５３３に入力される特徴抽出補助情報に、前述のサンプル情報が含まれる場合、当該サンプル情報を基準サンプル情報に設定し、そうでない場合は、規定値を基準サンプル情報と設定する。なお、必ずしも全ての基準サンプル情報を反映する必要はない。比較結果には、その比較の対象となったメディアデータの時刻情報が付随される。 The media data comparison unit 1534 compares the information of the input video, audio, and device signal with the reference sample information based on the feature extraction auxiliary information including the input sample information and the feature extraction target time information, and the comparison result Are supplied to the feature information acquisition unit 1535. The reference sample information is, for example, color information of the surgical site when the surgical site is incised with respect to an image obtained by photographing the surgical site. The color information of this reference sample information is compared with the color information of the image obtained by photographing the surgical site included in the media data, and a numerical result indicating how similar it is is regarded as a comparison result, and the comparison result is Based on the feature information is determined. In this case, if the comparison result is similar, the feature information acquisition unit 1535 sets feature information on the assumption that the surgical site is incised. If they are not similar, it is not necessary to set feature information as the operation part is closed, or not to set the feature information as not being similar to the reference sample information. The reference sample information described above is preset before the treatment process estimation is performed. However, when the above-described sample information is included in the feature extraction auxiliary information input to the media data processing unit 1533, the sample information is set as the reference sample information, and otherwise, the specified value is set as the reference sample information. Do. Note that it is not necessary to reflect all reference sample information. The comparison result is accompanied by time information of the media data which is the target of the comparison.
　特徴情報取得部１５３５は、入力される特徴抽出制限情報、処理結果、比較結果に基づき特徴情報を決定し、特徴情報抽出部１５３から出力する。 The feature information acquisition unit 1535 determines feature information based on the input feature extraction restriction information, the processing result, and the comparison result, and outputs the feature information extraction unit 153.
　なお、特徴抽出補助情報に処理設定情報のみを含む場合は、メディアデータ比較部１５３４で特徴抽出補助情報を受け取らず、基準サンプル情報に規定値を入れても良い。また、特徴抽出補助情報にサンプル情報のみを含む場合は、メディアデータ処理部１５３３で特徴抽出補助情報を受け取らず、処理の設定を規定値としても良い。 In the case where the feature extraction auxiliary information includes only the process setting information, the media data comparison unit 1534 may not receive the feature extraction auxiliary information, and may add a specified value to the reference sample information. When the feature extraction auxiliary information includes only the sample information, the media data processing unit 1533 may not receive the feature extraction auxiliary information, and the setting of the process may be set as the specified value.
　〔変形例４〕
　前述の治療工程推定装置１の説明では、工程情報推定部１５を、確定可能治療工程推定部１５１、特徴抽出補助情報取得部１５２、特徴情報抽出部１５３、治療工程推定部１５４で構成するとして説明しているが、必ずしもこの構成である必要はなく、特徴情報抽出部１５３と治療工程推定部１５４を、それらの機能を統合した治療工程推定部１５３aに置き換えて、工程情報推定部１５を構成する装置であっても良い。 [Modification 4]
In the description of the treatment process estimation apparatus 1 described above, it is assumed that the process information estimation unit 15 is configured by the determinable treatment process estimation unit 151, the feature extraction auxiliary information acquisition unit 152, the feature information extraction unit 153, and the treatment process estimation unit 154. Although it is not necessary to have this configuration, the process information estimation unit 15 is configured by replacing the feature information extraction unit 153 and the treatment process estimation unit 154 with a treatment process estimation unit 153 a that integrates their functions. It may be an apparatus.
　図７を用いて治療工程推定部１５３aの機能を説明する。図７は工程情報推定部１５の変形例を示した図である。図７に示すように、工程情報推定部１５は確定可能治療工程推定部１５１、特徴抽出補助情報取得部１５２、治療工程推定部１５３aより構成されても良い。確定可能治療工程推定部１５１と特徴抽出補助情報取得部１５２の機能は前述のとおりである。治療工程推定部１５３aは、入力される特徴抽出補助情報と映像、音声、機器信号、治療工程表、および、確定可能治療工程の工程情報に基づき、特徴情報を取り出す。さらに、上記特徴情報と治療工程表、確定可能治療工程の工程情報に基づき、確定可能治療工程以外の治療工程の工程情報を推定し、その工程情報を確定可能治療工程決定部１５１で決定された工程情報と足し合わせ、工程情報推定部１５から出力する。治療工程推定部１５３aにおける特徴情報の抽出は、上述した特徴情報抽出部１５３における特徴情報の抽出と同様に行われる。また、治療工程推定部１５３aにおける治療工程の推定は、上述した治療工程推定部１５４と同様に行われる。 The function of the treatment process estimation unit 153a will be described with reference to FIG. FIG. 7 is a view showing a modification of the process information estimation unit 15. As shown in FIG. 7, the process information estimation unit 15 may be configured of a determinable treatment process estimation unit 151, a feature extraction auxiliary information acquisition unit 152, and a treatment process estimation unit 153a. The functions of the determinable treatment process estimation unit 151 and the feature extraction auxiliary information acquisition unit 152 are as described above. The treatment process estimation unit 153a extracts the feature information based on the input feature extraction auxiliary information and the video, audio, device signal, treatment process chart, and process information of the determinable treatment process. Furthermore, the process information of the treatment process other than the determinable treatment process is estimated based on the feature information, the treatment process table, and the process information of the determinable treatment process, and the process information is determined by the determinable treatment process determination unit 151 The process information is added up with the process information and output from the process information estimation unit 15. Extraction of feature information in the treatment process estimation unit 153a is performed in the same manner as extraction of feature information in the feature information extraction unit 153 described above. Further, estimation of the treatment process in the treatment process estimation unit 153a is performed in the same manner as the treatment process estimation unit 154 described above.
　なお、前述の治療工程推定装置１の説明では、全ての特徴抽出対象時刻について特徴情報を抽出した後で、工程情報を推定するとして説明しているが、この構成の場合、必ずしも上記の順序に則る必要はなく、特徴情報の抽出と、工程情報の推定を並行して行っても良い。 In the above description of the treatment process estimation apparatus 1, the process information is estimated after extracting the feature information for all feature extraction target times, but in the case of this configuration, it is not always in the order described above. The extraction of feature information and the estimation of process information may be performed in parallel.
　〔ソフトウェアによる実現例〕
　治療工程推定装置１（特に工程情報推定部１５）は、集積回路（ＩＣチップ）等に形成された論理回路（ハードウェア）によって実現しても良いし、ＣＰＵ（Central Processing Unit）を用いてソフトウェアによって実現しても良い。 [Example of software implementation]
The treatment process estimation apparatus 1 (in particular, the process information estimation unit 15) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or software using a CPU (Central Processing Unit) It may be realized by
　後者の場合、治療工程推定装置１は、各機能を実現するソフトウェアであるプログラムの命令を実行するＣＰＵ，上記プログラムおよび各種データがコンピュータ（またはＣＰＵ）で読み取り可能に記録されたＲＯＭ（Read Only Memory）または記憶装置（これらを「記録媒体」と称する）、上記プログラムを展開するＲＡＭ（Random Access Memory）などを備えている。そして、コンピュータ（またはＣＰＵ）が上記プログラムを上記記録媒体から読みとって実行することにより、本発明の実施態様の目的が達成される。上記記録媒体としては、「一時的でない有形の媒体」、例えば、テープ、ディスク、カード、半導体メモリ、プログラマブルな論理回路などを用いることができる。また、上記プログラムは、該プログラムを伝送可能な任意の伝送媒体（通信ネットワークや放送波等）を介して上記コンピュータに供給されても良い。なお、本発明の一実施態様は、上記プログラムが電子的な伝送によって具現化された、搬送波に埋め込まれたデータ信号の形態でも実現され得る。 In the latter case, the treatment process estimation apparatus 1 is a CPU that executes instructions of a program that is software that realizes each function, a ROM (Read Only Memory) in which the program and various data are readably recorded by a computer (or CPU). Or a storage device (these are referred to as "recording media"), a RAM (Random Access Memory) for developing the program, and the like. The object of the embodiment of the present invention is achieved by the computer (or CPU) reading and executing the program from the recording medium. As the recording medium, a “non-transitory tangible medium”, for example, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit or the like can be used. Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program. Note that one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.
　〔補足〕
　本発明の一実施態様にかかる治療工程推定装置は、一つ以上の治療工程に関わる工程情報を決定する治療工程推定装置であって、前記治療工程に関する一つ以上のメディアデータを取得するメディアデータ取得部と、前記治療工程を示す治療工程表を取得する治療工程表取得部と、少なくとも前記メディアデータと前記治療工程表に基づき、確定可能な治療工程に関する工程情報を決定する確定可能治療工程決定部と、少なくとも前記確定可能な治療工程に関する工程情報、前記メディアデータおよび前記治療工程表に基づき、補助情報を取得する補助情報取得部と、少なくとも前記確定可能な治療工程に関する工程情報、前記メディアデータ、前記治療工程表および前記補助情報に基づき、前記工程情報を推定する治療工程推定部とを備える。 [Supplement]
A treatment process estimation apparatus according to an embodiment of the present invention is a treatment process estimation apparatus that determines process information related to one or more treatment processes, and media data for acquiring one or more media data related to the treatment process. An acquisition unit, a treatment process chart acquisition unit for acquiring a treatment process chart indicating the treatment process, and a determinable treatment process decision for determining process information on a determinable treatment process based on at least the media data and the treatment process chart An auxiliary information acquiring unit for acquiring auxiliary information based on at least the process information related to at least the determinable treatment process, the media data, and the treatment process chart, the process information at least regarding the determinable treatment process, the media data And a treatment process estimation unit configured to estimate the process information based on the treatment process chart and the auxiliary information. .
　なお、本発明の実施態様は、以下のような手段も講じてもよい。 The embodiment of the present invention may also take the following means.
　また、前記補助情報は、サンプル情報であることを特徴とする。 Further, the auxiliary information is sample information.
　また、前記補助情報は、処理設定情報であることを特徴とする。 Further, the auxiliary information is processing setting information.
　また、前記サンプル情報は、少なくとも前記確定可能な治療工程に関する工程情報と前記治療工程表に基づき、術部の映像から取得可能な情報であることを特徴とする。 Further, the sample information is characterized in that the sample information is information that can be acquired from an image of a surgical site based on at least process information on the determinable treatment process and the treatment process chart.
　また、前記サンプル情報は、少なくとも前記確定可能な治療工程に関する工程情報と前記治療工程表に基づき、治療中の指示を記録した音声情報から取得可能な情報であることを特徴とする。 Further, the sample information is characterized in that the sample information is information that can be acquired from voice information recording an instruction during treatment based on at least process information on the determinable treatment process and the treatment schedule.
　また、前記治療工程推定部は、少なくとも前記処理設定情報に基づく直線検出処理の結果と映像中に使用される医療器具を識別した識別結果に基づき、前記工程情報を推定することを特徴とする。 In addition, the treatment process estimation unit estimates the process information based on at least a result of a straight line detection process based on the process setting information and an identification result of identifying a medical device used in a video.
　また、前記確定可能治療工程決定部は、前記治療工程を構成する各作業の前記工程情報を決定し、前記治療工程推定部は、前記治療工程を構成する各作業の前記工程情報を推定することを特徴とする。 Further, the determinable treatment process determination unit determines the process information of each operation constituting the treatment process, and the treatment process estimation unit estimates the process information of each operation constituting the treatment process. It is characterized by
　また、前記治療工程推定部は、少なくとも前記治療工程表と、前記確定可能な治療工程に関する工程情報とを用いて、前記工程情報が未推定の時刻について、検出される医療器具の種類と数を含む情報を推定し、当該情報に基づき、前記工程情報を推定することを特徴とする。 In addition, the treatment process estimation unit uses at least the treatment process chart and process information on the determinable treatment process to determine the type and number of medical instruments detected for the time when the process information is not estimated. It is characterized in that information including the information is estimated, and the process information is estimated based on the information.
　また、前記治療工程推定部は、未推定の治療工程の前後の治療工程の前記工程情報が、推定または決定されている場合、少なくとも前記治療工程表と、前記確定可能な治療工程に関する工程情報と、推定された前記工程情報とに基づき、未推定の治療工程の情報を推定することを特徴とする。 In addition, when the process information of the treatment process before and after the unestimated treatment process is estimated or determined, the treatment process estimation unit determines at least the treatment process list and process information related to the determinable treatment process. And estimating the information on the unestimated treatment process based on the estimated process information.
　（関連出願の相互参照）本出願は、２０１５年１０月２０日に出願された特願２０１５－２０６０２４に対して優先権の利益を主張するものであり、当該出願を参照することにより、その内容の全てが本書に含まれる。 (Cross-reference to related applications) This application claims the benefit of priority over Japanese Patent Application No. 2015-206024 filed on October 20, 2015, the contents of which are hereby incorporated by reference. All of which are included in this book.
　１　　　　　　治療工程推定装置
　１１　　　　　映像取得部
　１２　　　　　音声取得部
　１３　　　　　機器信号取得部
　１４　　　　　治療工程表取得部
　１５　　　　　工程情報推定部
　１５１　　　　確定可能治療工程決定部
　１５２　　　　特徴抽出補助情報取得部
　１５３　　　　特徴情報抽出部
　１５３a　　　 治療工程推定部
　１５３１　　　特徴抽出対象時刻設定部
　１５３２　　　特徴抽出制限情報設定部
　１５３３　　　メディアデータ処理部
　１５３４　　　メディアデータ比較部
　１５３５　　　特徴情報取得部
　１５４　　　　治療工程推定部
　１６　　　　　工程情報蓄積部 DESCRIPTION OF SYMBOLS 1 treatment process estimation apparatus 11 image acquisition part 12 audio | voice acquisition part 13 apparatus signal acquisition part 14 treatment process chart acquisition part 15 process information estimation part 151 determinable treatment process determination part 152 feature extraction auxiliary information acquisition part 153 feature information extraction part 153a treatment Process estimation unit 1531 Feature extraction target time setting unit 1532 Feature extraction restriction information setting unit 1533 Media data processing unit 1534 Media data comparison unit 1535 Feature information acquisition unit 154 Treatment process estimation unit 16 Process information storage unit

Claims (9)

  1. 　一つ以上の治療工程に関わる工程情報を決定する治療工程推定装置であって、
    前記治療工程に関する一つ以上のメディアデータを取得するメディアデータ取得部と、
    　前記治療工程を示す治療工程表を取得する治療工程表取得部と、
    　少なくとも前記メディアデータと前記治療工程表に基づき、確定可能な治療工程に関する工程情報を決定する確定可能治療工程決定部と、
    　少なくとも前記確定可能な治療工程に関する工程情報、前記メディアデータおよび前記治療工程表に基づき、補助情報を取得する補助情報取得部と、
    　少なくとも前記確定可能な治療工程に関する工程情報、前記メディアデータ、前記治療工程表および前記補助情報に基づき、前記工程情報を推定する治療工程推定部と
    を備えることを特徴とする治療工程推定装置。     A treatment process estimation apparatus for determining process information related to one or more treatment processes, comprising:
    A media data acquisition unit for acquiring one or more media data related to the treatment process;
    A treatment process chart acquisition unit for acquiring a treatment process chart showing the treatment process;
    A determinable treatment process determination unit that determines process information on a determinable treatment process based on at least the media data and the treatment process chart;
    An auxiliary information acquisition unit that acquires auxiliary information based on at least process information related to the determinable treatment process, the media data, and the treatment process chart;
    What is claimed is: 1. A treatment process estimation apparatus comprising: a treatment process estimation unit configured to estimate the process information based on at least process information related to the determinable treatment process, the media data, the treatment process chart, and the auxiliary information.
  2. 　前記補助情報は、サンプル情報であることを特徴とする請求項１に記載の治療工程推定装置。 The treatment process estimation device according to claim 1, wherein the auxiliary information is sample information.
  3. 　前記補助情報は、処理設定情報であることを特徴とする請求項１に記載の治療工程推定装置。 The treatment process estimation apparatus according to claim 1, wherein the auxiliary information is process setting information.
  4. 　前記サンプル情報は、少なくとも前記確定可能な治療工程に関する工程情報と前記治療工程表に基づき、術部の映像から取得可能な情報であることを特徴とする請求項２に記載の治療工程推定装置。 The treatment process estimation apparatus according to claim 2, wherein the sample information is information that can be acquired from an image of a surgical site based on at least process information on the determinable treatment process and the treatment process chart.
  5. 　前記サンプル情報は、少なくとも前記確定可能な治療工程に関する工程情報と前記治療工程表に基づき、治療中の指示を記録した音声情報から取得可能な情報であることを特徴とする請求項２に記載の治療工程推定装置。 The sample information according to claim 2, wherein the sample information is information that can be acquired from voice information recording an instruction during treatment based on at least process information on the determinable treatment process and the treatment schedule. Treatment process estimation device.
  6. 　前記治療工程推定部は、少なくとも前記処理設定情報に基づく直線検出処理の結果と映像中に使用される医療器具を識別した識別結果に基づき、前記工程情報を推定することを特徴とする請求項３に記載の治療工程推定装置。 The treatment process estimation unit estimates the process information based on at least a result of a straight line detection process based on the process setting information and an identification result of identifying a medical device used in a video. The treatment process estimation device according to claim 1.
  7. 　前記確定可能治療工程決定部は、前記治療工程を構成する各作業の前記工程情報を決定し、
    　前記治療工程推定部は、前記治療工程を構成する各作業の前記工程情報を推定することを特徴とする請求項１に記載の治療工程推定装置。     The determinable treatment process determination unit determines the process information of each operation constituting the treatment process,
    The treatment process estimation apparatus according to claim 1, wherein the treatment process estimation unit estimates the process information of each work constituting the treatment process.
  8. 　前記治療工程推定部は、少なくとも前記治療工程表と、前記確定可能な治療工程に関する工程情報とを用いて、前記工程情報が未推定の時刻について、検出される医療器具の種類と数を含む情報を推定し、当該情報に基づき、前記工程情報を推定することを特徴とする請求項１に記載の治療工程推定装置。 The treatment process estimation unit uses information including at least the treatment process chart and process information related to the determinable treatment process, and information including the type and number of medical devices detected for the time when the process information is not estimated The treatment process estimation apparatus according to claim 1, wherein the process information is estimated based on the information.
  9. 　前記治療工程推定部は、未推定の治療工程の前後の治療工程の前記工程情報が、推定または決定されている場合、
    　少なくとも前記治療工程表と、前記確定可能な治療工程に関する工程情報と、推定された前記工程情報とに基づき、未推定の治療工程の情報を推定することを特徴とする請求項１に記載の治療工程推定装置。     The treatment process estimation unit may estimate or determine the process information of the treatment process before and after the unestimated treatment process.
    The treatment according to claim 1, wherein information of an unestimated treatment process is estimated based on at least the treatment schedule, the process information on the determinable treatment process, and the estimated process information. Process estimation device.
PCT/JP2016/074579 2015-10-20 2016-08-24 Medical treatment process estimation device WO2017068853A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2015206024A JP2018200493A (en) 2015-10-20 2015-10-20 Treatment process estimation device
JP2015-206024 2015-10-20

Publications (1)

Publication Number Publication Date
WO2017068853A1 true WO2017068853A1 (en) 2017-04-27

Family

ID=58556915

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2016/074579 WO2017068853A1 (en) 2015-10-20 2016-08-24 Medical treatment process estimation device

Country Status (2)

Country Link
JP (1) JP2018200493A (en)
WO (1) WO2017068853A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108932578A (en) * 2017-05-23 2018-12-04 希森美康株式会社 The technology evaluation system of form inspection

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001290885A (en) * 2000-04-04 2001-10-19 Hitachi Ltd Medical examination assisting system
JP2004280455A (en) * 2003-03-14 2004-10-07 Toshiba Corp Medical information system
JP2006164251A (en) * 2004-11-09 2006-06-22 Toshiba Corp Medical information system, medical information system program, and medical information processing method for performing information processing for management of medical practice
JP2006252286A (en) * 2005-03-11 2006-09-21 Gifu Univ Medical information processor synchronized with standard treatment program, and medical information processing method
JP2008220839A (en) * 2007-03-15 2008-09-25 Olympus Medical Systems Corp Image filing system and image display system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001290885A (en) * 2000-04-04 2001-10-19 Hitachi Ltd Medical examination assisting system
JP2004280455A (en) * 2003-03-14 2004-10-07 Toshiba Corp Medical information system
JP2006164251A (en) * 2004-11-09 2006-06-22 Toshiba Corp Medical information system, medical information system program, and medical information processing method for performing information processing for management of medical practice
JP2006252286A (en) * 2005-03-11 2006-09-21 Gifu Univ Medical information processor synchronized with standard treatment program, and medical information processing method
JP2008220839A (en) * 2007-03-15 2008-09-25 Olympus Medical Systems Corp Image filing system and image display system

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108932578A (en) * 2017-05-23 2018-12-04 希森美康株式会社 The technology evaluation system of form inspection
CN108932578B (en) * 2017-05-23 2024-01-23 希森美康株式会社 Technique evaluation system for morphological inspection

Also Published As

Publication number Publication date
JP2018200493A (en) 2018-12-20

Similar Documents

Publication Publication Date Title
KR102013837B1 (en) Method and apparatus for providing surgical video
JP4828919B2 (en) Medical information system
RU209329U1 (en) AUTOMATIC ADDITION OF THE ENDOSCOPIC VIDEO IMAGE
US20180366231A1 (en) System and method for analysis and presentation of surgical procedure videos
Stauder et al. The TUM LapChole dataset for the M2CAI 2016 workflow challenge
Blum et al. Modeling and online recognition of surgical phases using hidden markov models
JP7380557B2 (en) Information processing system, information processing device, and information processing method
JP2021099789A (en) Evaluation of surgical operation scene based on computer vision
CN106448403B (en) It is a kind of for simulating the training aids of thoracoscopic operation
US10271828B2 (en) Image file creation method, medium with image file creation program recorded and image file creation apparatus
WO2020116224A1 (en) Information processing device, presentation method, and surgical operation system
CN113039609A (en) Operation support system, data processing device, and method
WO2017068853A1 (en) Medical treatment process estimation device
WO2019239243A1 (en) Tool handedness determination for surgical videos
CN117727300A (en) Voice information processing method and device
JP2023181214A (en) Information processing device, information processing method, and computer program
JP7235212B2 (en) Handedness Tool Detection System for Surgical Video
JP2023175849A (en) Surgery details evaluation system, surgery details evaluation method, and computer program
CN117045348A (en) Methods, systems, and media for surgical automation
CN114025701A (en) Surgical tool tip and orientation determination
KR20220103656A (en) Apparatus and method for comparing surgical operation pattern based on artificial intelligence analytics
Nemani et al. A comparison of NOTES transvaginal and laparoscopic cholecystectomy procedures based upon task analysis
RU216092U1 (en) AUTOMATIC ADDITION OF THE ENDOSCOPIC VIDEO IMAGE
US20240095919A1 (en) Image processing device, endoscope system, image processing method, and information storage medium
JP2007075520A (en) Surgery analyzer

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16857171

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16857171

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP