WO2021152948A1

WO2021152948A1 - Computer program and image playback method

Info

Publication number: WO2021152948A1
Application number: PCT/JP2020/041295
Authority: WO
Inventors: 尚紀北村; 勇太熊頭; 直小林
Original assignee: アナウト株式会社
Priority date: 2020-01-30
Filing date: 2020-11-05
Publication date: 2021-08-05
Also published as: JP6887193B1; WO2021152785A1; JP2021121095A; JP2021119986A

Abstract

The present invention makes it possible to facilitate understanding of surgical events.　The present invention causes a computer to execute processing for: recording an operating field image obtained by capturing images of the operating field of an endoscopic surgery chronologically; determining the presence or absence of a predetermined amount or more of bleeding on the basis of the operating field image; and, when the presence of a predetermined amount or more of bleeding is determined, using a set playback method to play back, from among the recorded operating field image, a partial image depicting a time range containing from a time point prior to the start of bleeding to a time point after the bleeding has occurred.

Description

Computer program and video playback method

The present invention relates to a computer program and a video reproduction method.

Patent Document 1 proposes to reproduce a part of the surgical video.

Japanese Unexamined Patent Publication No. 2011-36370

In Patent Document 1, it is possible to confirm the necessary part by reproducing from the feature point, but even if the scene where the incident such as bleeding occurs is simply reproduced, the part causing the incident is grasped. Things are often difficult.

The present invention has been made in view of such a background, and an object of the present invention is to provide a technique for facilitating an understanding of an event during surgery.

The computer program according to one aspect of the present invention records a surgical field image obtained by photographing the surgical field under a microscope in a time series on a computer, and based on the surgical field image, bleeding of a predetermined amount or more. If the presence or absence is judged and it is judged that there is bleeding of a predetermined amount or more, the partial video of the time range including the time before the start of bleeding to the time after bleeding among the recorded surgical field images is set to the set playback method. To execute the process of playing.

According to the present invention, it is possible to make it easier to understand events during surgery.

It is a conceptual diagram explaining the outline of the surgical system which concerns on this Embodiment. It is a block diagram explaining the outline of the operation system which concerns on this embodiment. It is a block diagram explaining the hardware configuration example of the control unit 30. It is a block diagram explaining the software structure example of the control unit 30. It is a block diagram explaining the outline of the structure of the image M acquired from the endoscope camera 13. It is a figure explaining the occurrence of an event. It is a figure explaining the frame F1 at the time of an event occurrence. It is a figure which shows an example of the frame F1 at the time of event occurrence. It is a figure which shows the example of the display 21 which displayed the wipe image W. It is a figure explaining the process performed in the operation system of this embodiment. It is a figure which shows the software structure example of the control unit 30 which concerns on 2nd Embodiment. It is a figure which shows the flow of the process which concerns on 2nd Embodiment. It is a figure which shows the software structure example of the control unit 30 which concerns on 3rd Embodiment. It is a figure explaining the process flow of the control unit 30 which concerns on 3rd Embodiment. It is a figure which shows the software structure example of the control unit 30 which concerns on 4th Embodiment. It is a figure explaining the process flow of the control unit 30 which concerns on 4th Embodiment. It is a figure which shows the whole structure of the operation system 1 which concerns on 5th Embodiment. It is a figure explaining the process flow of the control unit 30 which concerns on 6th Embodiment. It is a figure explaining an example of a reproduction method. It is a figure explaining the process flow of the control unit 30 which concerns on 7th Embodiment. It is a figure which shows the display example of the thumbnail which shows the bleeding scene. It is a figure explaining the process flow of the control unit 30 which concerns on 8th Embodiment. It is a figure which shows the display example of the graph which shows the time-series change of the bleeding amount. It is a figure explaining the process flow of the control unit 30 which concerns on 10th Embodiment.

<Outline of the invention>
The contents of the embodiments of the present invention will be described in a list. The present invention includes, for example, the following configuration.
[Item 1]
A moving image acquisition unit that acquires moving images of the surgical field under arthroscopic surgery,
An event detection unit that analyzes the moving image and detects a predetermined event in the surgical field,
A partial moving image extraction unit that extracts a partial moving image during a period including the detection time of the event from the moving image, and a partial moving image extraction unit.
An event playback unit that reproduces the partial moving image by each of a plurality of playback methods,
A video playback system characterized by being equipped with.
[Item 2]
The video playback system according to item 1.
The reproduction method includes at least two of normal reproduction, slow reproduction, repeated reproduction, and reverse reproduction.
A video playback system featuring.
[Item 3]
The video playback system according to item 1.
The event reproduction unit selects the type of the plurality of regeneration methods according to at least one of the type of the organ to be operated on, the surgical method related to the treatment, and the parameter related to the event.
A video playback system featuring.
[Item 4]
The video playback system according to item 1.
The event regeneration unit has a regeneration speed according to the regeneration method and a period from the detection time according to at least one of the type of the organ to be operated on, the surgical procedure related to the treatment, and the parameters related to the event. To select at least one of the time to the start time, the time from the detection time to the end time of the period, and the reproduction direction.
A video playback system featuring.
[Item 5]
The video playback system according to item 1.
The event is bleeding from the surgical subject and
The event detection unit further detects the amount of bleeding and
The reproduction method includes at least the slow reproduction, and the reproduction method includes at least the slow reproduction.
The event reproduction unit determines the reproduction speed of the slow reproduction at least according to the amount of bleeding.
A video playback system featuring.
[Item 6]
The surgical support system according to item 1.
The event reproduction unit sets the reproduction speed of the partial moving image during the first period, which is the period, shorter than the first period, and for the second period including the detection time point. Playing lower than the period of
A surgical support system featuring.
[Item 7]
The surgical support system according to item 1.
At least the types of the plurality of reproduction methods, the reproduction speed according to the reproduction method, the time from the detection time to the start time of the period, the time from the detection time to the end time of the period, and the reproduction direction. To have a setting unit to set either,
A surgical support system featuring.
[Item 8]
The surgical support system according to item 7.
After the start of reproduction by the event reproduction unit, the setting unit includes the types of the plurality of reproduction methods, the reproduction speed related to the reproduction method, the time from the detection time to the start time of the period, and the detection time to the start time. To change at least one of the time to the end of the period and the playback direction,
A surgical support system featuring.

<System overview>
Hereinafter, the surgical system according to the embodiment of the present invention will be described. The surgery support system of the present embodiment supports arthroscopic surgery, and in particular, outputs a moving image of the surgical field and, for example, when an event such as local bleeding occurs, the event occurs. It can also be a moving image playback system that plays back the scene including the time of occurrence.

<First Embodiment>
FIG. 1 is a conceptual diagram illustrating an outline of a surgical system according to the present embodiment. As shown in the figure, the surgical system 1 includes a surgical unit 10, an operating unit 20, and a control unit 30 that controls the surgical unit 10 and the operating unit 20.

FIG. 2 is a block diagram illustrating an outline of the surgical system according to the present embodiment.

The surgical unit 10 performs surgical treatment on the patient 100 to be operated on. In the present embodiment, the surgical unit 10 is the instrument unit 11 which is the treatment unit, the sensor unit 12 which is also the treatment unit, and the image acquisition unit. A certain endoscopic camera 13 is provided.

The instrument unit 11 is composed of a movable arm and a surgical instrument attached to the tip of the movable arm. As the surgical instruments, for example, scalpels (electric scalpels, etc.), scissors, forceps, needle holders, tweezers, etc. are assumed, but various surgical instruments other than these are attached depending on the application.

For the sensor unit 12, for example, various sensors such as a pressure sensor, a gyro sensor, an acceleration sensor, and a temperature sensor for detecting the state of the instrument unit 11 are adopted.

The configuration of the endoscope camera 13 will be described later.

The operation unit 20 receives an operation from the surgeon 110 who is an operator for the operation unit 10, and in the present embodiment, the operation unit 20 includes a display 21, a controller 22, and a speaker 23, which are display units. Further, the operation unit 20 may be provided with a microphone for receiving a voice instruction from the surgeon 110.

The display 21 provides various information to the surgeon 110, and in the present embodiment, a technique for implementing a VR (Virtual Reality) HMD (Head Mount Display) is adopted, and the surgeon 110 It is possible to stereoscopically view the area to be visually recognized by using the parallax between the eyes.

In the present embodiment, the controller unit 22 is implemented by an input device such as a joystick or a foot pedal, and the speaker 23 provides various information to the surgeon 110 by voice or the like.

<Control unit 30 hardware>
FIG. 3 is a block diagram illustrating a hardware configuration example of the control unit 30. As shown in the figure, the control unit 30 includes a processor 31, a memory 32, a storage 33, a transmission / reception unit 34, and an input / output unit 35 as main configurations, and these are electrically connected to each other via a bus 36.

The processor 31 is an arithmetic unit that controls the operation of the control unit 30, controls the transmission and reception of data between each element, and performs processing necessary for executing an application program.

In the present embodiment, the processor 31 is, for example, a CPU (Central Processing Unit), and executes each process by executing an application program or the like stored in the storage 33 and expanded in the memory 32, which will be described later.

The memory 32 includes a main storage device composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory), and an auxiliary storage device composed of a non-volatile storage device such as a flash memory or an HDD (Hard Disk Drive). ..

While this memory 32 is used as a work area of the processor 31, the BIOS (Basic Input / Output System) executed when the control unit 30 is started, various setting information, and the like are stored.

The storage 33 stores application programs, data used for various processes, and the like. In this embodiment, a video processing program that performs various types of processing is stored. The video processing program is provided, for example, by a non-temporary recording medium RM in which the program is readablely recorded. The processor 31 may read a desired program from the recording medium RM using a reading device (not shown in the figure), and store the read program in the storage 33. Details of this video processing program will be described later.

The transmission / reception unit 34 connects the control unit 30 to the Internet. The transmission / reception unit 34 may be provided with a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

In the present embodiment, the control unit 30 is connected to the surgery unit 10 and the operation unit 20 via the transmission / reception unit 34.

Information input devices such as keyboards and mice and output devices such as displays are connected to the input / output unit 35 as needed.

The bus 36 transmits, for example, an address signal, a data signal, and various control signals between the connected processor 31, memory 32, storage 33, transmission / reception unit 34, and input / output unit 35.

<Control unit 30 software>
FIG. 4 is a block diagram illustrating a software configuration example of the control unit 30. As shown in the figure, the control unit 30 includes an operation input unit 311, a time point identification unit 312, a partial moving image extraction unit 313, a playback unit 314, a video recording unit 331, a parameter storage unit 332, and a setting storage unit 333.

The video recording unit 331 records the video M acquired by the endoscope camera 13 and displayed on the display 21 in real time as the recorded video M1.

FIG. 5 is a block diagram illustrating an outline of the configuration of the image M acquired from the endoscope camera 13. As shown in the figure, the endoscopic camera 13 captures the surgical procedure O for the patient 100 by the instrument unit 11 and acquires it as an image M. The image M acquired by the endoscope camera 13 is composed of a plurality of frames F, and is sequentially displayed on the display 21 in the acquired real time, and the image M displayed on the display 21 is recorded on the image recording unit 331. It will be recorded as video M1.

The surgical system 1 according to the present embodiment having such a configuration does not incise the body by making a plurality of small holes of about 3 mm to 10 mm in the body of the patient 100 so as to reach the abdominal cavity and the thoracic cavity of the patient 100. The surgical instruments of the endoscopic camera 13 and the instrument unit 11 are inserted through the holes, and endoscopic surgery is performed with minimal invasiveness under the endoscope.

The operation input unit 311 receives various operations input from the operation unit 20 by the surgeon 110.

The occurrence time identification unit 312 specifies the time when the event occurs from the recorded video M1 (hereinafter, referred to as the event occurrence time). The occurrence time identification unit 312 generates an event in the frame by, for example, performing image analysis on each frame F constituting the recorded image M1 and extracting a feature amount (for example, red color of bleeding) of the image indicating the event. It is possible to determine whether or not the event is occurring, and specify the time point of the earliest frame F in the period in which the same series of events are continuous as the event occurrence time point.

The time-of-occurrence identification unit 312 can be implemented by, for example, an artificial intelligence module that autonomously identifies event-related frames by machine learning. As a method for performing machine learning, various algorithms such as a neural network, a random forest, and an SVM (support vector machine) are appropriately used.

FIG. 6 is a diagram illustrating the occurrence of an event. FIG. 7 is a diagram for explaining the frame F1 when an event occurs.

As shown in FIG. 6, the time-of-occurrence identification unit 312 unexpectedly affects the patient 100 when performing surgical treatment on the affected area T of the patient 100 with the

surgical instruments

11a and 11b of the instrument unit 11 of the surgical unit 10. When a situation (event), for example, local bleeding occurs and bleeding B is displayed on the image M displayed in real time on the display 21, as shown in FIG. 7, the frame constituting the recorded image M1. From F, event F1 in which the occurrence of bleeding B is recorded can be identified. In the frame F1 where the event occurred, the state before the blood flowed out and the bleeding B spread, and the initial stage of the bleeding B where the blood had just exuded, that is, the state immediately after the bleeding B occurred was recorded. become.

The partial video extraction unit 313 extracts a partial video (frames included in the period) of the period including the time when the event occurs. In the example of FIG. 7, the partial moving image extraction unit 313 sets the time point of the frame FS at the time point i1 before the frame F1 at the event occurrence time t0 as the start frame, and is after the time i2 from the frame F1 at the event occurrence time point t0. Time point frame Each frame from the start frame to the end frame can be extracted as a partial moving image, with the time point of the FE as the end frame. That is, the partial moving image extraction unit 313 can extract the partial moving image from the recorded video M1 in the period from the start time t0-i1 to the end time t0 + i2.

The parameter storage unit 332 stores various parameters such as time i1 and i2. In the example of FIG. 4, the parameter storage unit 332 stores parameters such as the previous time, the later time, the reproduction speed, and the number of reproductions. For example, the front time is time i1 and the back time is time i2. The reproduction speed is a reproduction speed when a partial moving image is slowly reproduced by the reproduction unit 314 described later. The number of reproductions is the number of repetitions when the partial moving image is repeatedly reproduced by the reproduction unit 314 described later.

The playback unit 314 plays a partial moving image. In the present embodiment, the reproduction unit 314 reproduces the partial moving image by a plurality of reproduction methods. The plurality of reproduction methods include at least two of normal reproduction, slow reproduction, repeated reproduction, and reverse reproduction. That is, the reproduction unit 314 can reproduce the scene including the time when the event occurs a plurality of times while changing the reproduction method. Various parameters related to the reproduction method (for example, the reproduction speed of slow reproduction, the number of reproductions of repeated reproduction, etc.) are stored in the parameter storage unit 332.

The reproduction unit 314 can display a partial image as a wipe image on the display 21 viewed by the surgeon 110. FIG. 8 is a diagram showing an example of frame F1 at the time of event occurrence. FIG. 8 shows an example at the time when the occurrence of bleeding B was recognized. FIG. 9 is a diagram showing an example of a display 21 on which the wipe image W is displayed. In the example of FIG. 9, the frame F1 at the time of event occurrence is displayed as the wipe image W.

Further, the reproduction unit 314 can reproduce a partial moving image by a combination of different reproduction methods depending on the type of operation of the surgical instrument of the instrument unit 11. The playback method to be played back is stored in the setting storage unit 333, which will be described later.

In the example of FIG. 4, the setting storage unit 333 stores a plurality of playback methods (lists) in association with the operation type. The operation type includes, for example, excision, peeling, traction, suturing, suction, and the like, and these can be specified by, for example, image analysis using machine learning. Further, it may be specified by the type of surgical instrument attached to the instrument unit 11, the operation input signal of the surgeon 110 received by the operation input unit 311, or the like.

<Processing>
FIG. 10 is a diagram illustrating processing performed in the surgical system of the present embodiment. The figure shows the flow of operation of the control unit 30.

The occurrence time point specifying unit 312 specifies the time point (frame) at which the event occurred from the recorded video M1 (S401), and also specifies the operation type of the surgical instrument of the instrument unit 11 (S402). For example, the generation time identification unit 312 can specify the operation type by analyzing the image.

The partial moving image extraction unit 313 reads the parameter and playback method list from the parameter storage unit 332 and the setting storage unit 333 (S403), the time before (i1) and the time after (i2) of the read parameter, and the specified event occurrence time point. The period (t0-i1 to t0 + i2) to be extracted from t0 is calculated, and the partial moving image of the calculated period is extracted from the recorded video M1 (S404).

The playback unit 314 extracts the next playback method from the playback methods included in the playback method list (S405), and if there is a next playback method (S406: NO), reproduces the partial moving image by the next playback method (S406: NO). S407). Here, as the parameters of the reproduction method (reproduction speed of slow reproduction, number of reproductions of repeated reproduction, etc.), the above-read parameters are used. The reproduction unit 314 may superimpose and output a partial image as a wipe image W as shown in FIG. 9 on a real-time image from the endoscopic camera 13, or provide a sub-display or the like different from the display 21 on the sub-display. A partial image may be output.

As described above, according to the surgical system of the present embodiment, when an event (for example, bleeding) that occurs in the space (for example, abdominal cavity) to be operated on is detected, a partial moving image of a period including the time when the event occurs, That is, the event occurrence scene can be output. For example, as in the example of FIG. 6, in a state where bleeding B occurs and spreads, a partial moving image of a period including an initial time point (event occurrence time point) when bleeding B occurs is wiped as shown in FIG. By confirming with the image W, the surgeon 110 can easily identify the bleeding site where the bleeding B is occurring, for example. It is also possible to identify the cause of how bleeding B occurs, and take appropriate measures. This is due to the following reasons, for example, in the case of bleeding. That is, assuming that bleeding occurs after the tissue in the body is pinched by a resection device and activated and the tissue collapses, the movement until pinching and the time during activation can be grasped at a normal speed, but the tissue. Since the process of collapse and bleeding spreading from the bleeding point is an event that occurs in an extremely short time, slowing down the playback speed of this scene makes it easier to understand. In addition, by reverse regeneration, the spread blood is aggregated at one point, and it becomes easy to grasp the position of the bleeding point. Furthermore, by partially slowing down the regeneration speed or performing reverse regeneration in this way, it becomes easy to identify the location and cause of the event such as bleeding.

Further, according to the surgical system of the present embodiment, it is possible to confirm a partial moving image of a period including the time when an event such as bleeding B occurs by a plurality of reproduction methods. Therefore, as compared with the case where the past scene is simply confirmed once, the event occurrence position and the occurrence cause can be easily and surely grasped by performing the display a plurality of times as in the present embodiment. Further, by playing back the same scene with different playback methods as in the present embodiment, the same event can be confirmed from different viewpoints, so that the event occurrence location (for example, bleeding location) and the event occurrence factor (for example, bleeding) can be confirmed. The cause) can be grasped more easily and surely.

<Second Embodiment>
FIG. 11 is a diagram showing a software configuration example of the control unit 30 according to the second embodiment. In the first embodiment, the reproduction speed of slow reproduction (or fast-forward reproduction may be used) is set as a parameter in the parameter storage unit 332, but in the second embodiment, the reproduction speed is set according to the amount of bleeding. Can be changed.

The control unit 30 according to the second embodiment includes a bleeding amount detecting unit 315 in addition to the configuration of the control unit 30 of the first embodiment described above (see FIG. 4). The bleeding amount detection unit 315 analyzes the image of each frame F of the recorded image M1 and estimates the amount of blood contained in the image. The bleeding amount detection unit 315 may be, for example, a bleeding amount (or a bleeding area, that is, a ratio of blood covering in the target area of surgery, depending on the number of bleeding color pixels included in the image. ) Etc. can be calculated. In addition, the regeneration unit 314 of the second embodiment changes the regeneration speed according to the amount of bleeding. Specifically, the regeneration unit 314 can reduce the regeneration speed as the amount of bleeding (or bleeding area) increases. Further, the regeneration unit 314 is set to reduce the regeneration speed as the bleeding amount (or bleeding area) changes in time series (for example, the amount of increase in the bleeding amount per unit time) is larger (the bleeding rate is faster). Can be done.

FIG. 12 is a diagram showing a flow of processing according to the second embodiment. The difference from the process of the first embodiment (FIG. 10) will be described. In the second embodiment, the bleeding amount detection unit 315 detects the bleeding amount (or bleeding rate) from the partial moving image (S421), and the regenerating unit 314 regenerates as the bleeding amount (or bleeding rate) increases (or is faster). The speed can be reduced (S422). Then, in step S407, the partial moving image is reproduced at the adjusted reproduction speed. As a result, when a large amount of bleeding or high-speed bleeding occurs, the scene at the time of bleeding (at the time of event occurrence) can be reproduced more slowly, so that the bleeding site and the cause of bleeding can be more easily and surely identified.

Note that the reproduction unit 314 may change the reproduction speed only when the reproduction method is slow reproduction, or may change the reproduction speed for all the reproduction methods.

<Third Embodiment>
FIG. 13 is a diagram showing a software configuration example of the control unit 30 according to the third embodiment. In the first embodiment, the reproduction speed of the partial moving image is constant, but in the third embodiment, the reproduction speed can be reduced for a shorter period before and after the event occurrence even in the partial moving image.

In the control unit 30 according to the second embodiment, as compared with the configuration of the control unit 30 of the first embodiment described above (see FIG. 4), the parameters stored in the parameter storage unit 332 include the pre-time, the post-time, and the reproduction. Memorize 2 sets of speed. In the example of FIG. 13, the first set of the front time 1 (i1 (1)), the back time 1 (i2 (1)) and the playback speed 1 and the front time 2 (i1 (2)) and the back time 2 (i2) (2)) and the second set of the reproduction speed 2 are registered in the parameter storage unit 332. The period of the second set is shorter than the period of the first set, and both the period of the first set and the period of the second set include the event disclosure time point (frame F1). Further, the reproduction speed 2 of the second set is slower than the reproduction speed 2 of the first set.

FIG. 14 is a diagram illustrating a processing flow of the control unit 30 according to the third embodiment. The difference from the process of the first embodiment (FIG. 10) will be described. In the third embodiment, steps S441 to S443 are executed instead of the reproduction process of step S407. In step S441, the reproduction speed 1 of the first set is the period from the start time t0-i1 (1) similar to the start time t0-i1 of the first embodiment to the start time t0-i1 (2) of the second set. (S441), the period from the start time t0-i1 (2) of the second set to the end time t0 + i2 (2) of the second set is reproduced at the playback speed 2 of the second set (S442), and the second set. The period from the end time t0 + i2 (2) of the set to the end time t0 + i2 (1) of the first set is reproduced at the reproduction speed 1 of the first set (S443). As a result, the period of the second set, which is shorter than the period of the first set, including the time when the event occurs, can be reproduced at a slower speed.

In addition, also in the third embodiment, the bleeding amount detection unit 315 is provided as in the second embodiment, and the regeneration speed 2 is regenerated at a speed corresponding to the bleeding amount or the bleeding rate for the period related to the second set. You may do it.

It is also possible to set the playback speed 1 related to the first set to a speed faster than the normal speed (real time) so that the scenes other than the scene including the event occurrence time can be played back at high speed.

<Fourth Embodiment>
FIG. 15 is a diagram showing a software configuration example of the control unit 30 according to the fourth embodiment. In the fourth embodiment, various parameters related to the reproduction method (pre-time, post-time, reproduction direction, reproduction speed, number of reproductions, etc.) can be set and changed. In the fourth embodiment, the parameter storage unit 332'stores various parameters in association with the reproduction method ID indicating the reproduction method. Further, the control unit 30 includes a setting unit 316 that receives an input of a parameter stored in the parameter storage unit 332'. The setting unit 316 can register the parameters in advance, and can also change the parameters during the operation or the playback of the moving image.

FIG. 16 is a diagram illustrating a processing flow of the control unit 30 according to the fourth embodiment. The difference from the process of the first embodiment (FIG. 10) will be described. In the fourth embodiment, the setting unit 316 receives the input of the parameter for each reproduction method in advance and registers it in the parameter storage unit 332'(S461). Then, in step S403, only the reproduction method list is read (S403'), and before step S407 at the time of reproduction, the reproduction unit 314 reads and reads the parameters corresponding to the reproduction method from the parameter storage unit 332'(S462). A partial moving image is played back using the parameters (S407').

This makes it possible to freely set the playback speed, playback period, etc. according to the playback method.

<Fifth Embodiment>
FIG. 17 is a diagram showing an overall configuration of the surgical system 1 according to the fifth embodiment. In the fifth embodiment, it is assumed that the surgeon 110 directly operates the surgical instrument to the patient 100 instead of the remote control. It may be laparotomy or laparoscopic surgery. Even in such a case, the control unit 30 of each of the above-described embodiments can be used to reproduce the scene including the time when the event occurs a plurality of times while changing the reproduction method.

<Sixth Embodiment>
In the sixth embodiment, when a predetermined amount or more of bleeding is detected, a configuration for reproducing a partial moving image in a predetermined time range including the start time of bleeding will be described. Since the software configuration of the control unit 30 is the same as that of the second embodiment, the description thereof will be omitted.

FIG. 18 is a diagram illustrating a processing flow of the control unit 30 according to the sixth embodiment. The difference from the process of the first embodiment (FIG. 10) will be described. When an event (bleeding in this embodiment) is detected in step S401, the bleeding amount detection unit 315 analyzes the image of the frame F obtained from the endoscope camera 13 to detect the bleeding amount (S471). As described in the second embodiment, the bleeding amount detecting unit 315 may detect the bleeding amount according to the number of pixels having a color corresponding to bleeding and the area of the region. The bleeding amount detection unit 315 detects the bleeding amount each time the image of the frame F is input, and determines the presence or absence of bleeding of a predetermined amount or more by comparing with a preset threshold value (S472). When there is no bleeding of a predetermined amount or more (S472: NO), the control unit 30 returns the process to step S401.

When it is determined that there is bleeding of a predetermined amount or more (S472: YES), the control unit 30 executes the processes after step S402. That is, the control unit 30 reads out various parameters related to the reproduction method (for example, reproduction speed, reproduction number of repeated reproductions, etc.) and a reproduction method list according to the operation type such as excision, peeling, traction, suturing, and suction. , The partial moving image extracted from the recorded video M1 is played back based on the read parameters and the playback method list. The method for extracting the partial moving image is the same as that in the first embodiment. That is, the partial moving image extraction unit 313 is a frame included in the time range from the time point before the start of bleeding (t0-i1) to the time point after bleeding (t0 + i2) in the recorded video M1 recorded in the video recording unit 331. F may be extracted as a partial moving image. In this embodiment, t0 represents the time point at which bleeding begins. The times i1 and i2 may be set in advance, or may be set according to the operation type such as excision, peeling, traction, suturing, and suction, and the imaged scene.

The playback unit 314 plays a partial moving image according to a playback method including at least one of normal playback, slow playback, frame-by-frame playback, repeated playback, and reverse playback. The reproduction unit 314 may superimpose and output a partial moving image as a wipe image W as shown in FIG. 9 on a real-time image (surgical field image) from the endoscope camera 13, or may output a sub different from the display 21. A display or the like may be provided to output a partial moving image to the sub-display. Further, the reproduction unit 314 may repeatedly reproduce the partial moving image while switching at least two of normal reproduction, slow reproduction, frame advance reproduction, and reverse reproduction. Further, the reproduction unit 314 may reproduce the scene including the start time of bleeding at a speed lower than the reproduction speed of the scene before and after the start of bleeding.

FIG. 19 is a diagram illustrating an example of a reproduction method. In the recorded video M1 including a scene in which bleeding of a predetermined amount or more is detected, when time t = t0 is set as the bleeding start time, the partial moving image extraction unit 313 starts from time t = t0-i1 (1) to t = t0 + i2 ( The frame F in the time range up to 1) is extracted as a partial moving image. i1 (1) is set to, for example, 2 to 6 seconds depending on the operation type and the scene. For i2 (1), an appropriate time such as 2 seconds is set.

The playback unit 314 plays a partial moving image by switching the playback method in order, such as normal playback, partial slow playback, and partial slow reverse playback. The reproduction unit 314 may repeat normal reproduction, partial slow reproduction, and partial slow reverse reproduction after performing partial slow reverse reproduction.

In normal playback, for example, i1 (1) is set to 6 seconds and i2 (1) is set to 2 seconds. That is, the reproduction unit 314 reproduces the partial moving image for a total of 8 seconds from 6 seconds before the start of bleeding to 2 seconds after the start of bleeding at a normal reproduction speed (reproduction speed 1). By playing back the surgical field image captured before the start of bleeding for a little longer, the operator can grasp the operation leading to bleeding.

In the partial slow playback, for example, i1 (1) is set to 3 seconds, i2 (1) is set to 2 seconds, and i1 (2) and i2 (2) are set to 0.2 seconds, respectively. That is, the reproduction unit 314 reproduces a 2.8-second partial moving image from 3 seconds before the start of bleeding to 0.2 seconds before the start of bleeding at a normal reproduction speed (reproduction speed 1), and 0.2 at the start of bleeding. A 0.4 second partial moving image from a second before to a 0.2 second later is slowly played back at a playback speed (playback speed 2) that is 1/5 times the normal playback speed. The reproduction speed in slow reproduction is not limited to 1/5 times speed, and may be appropriately set. After the slow reproduction, the reproduction unit 314 reproduces the partial moving image for 1.8 seconds from 2 seconds to 2 seconds after the start of bleeding at a normal reproduction speed (reproduction speed 1). Partial slow regeneration has the advantage that the operator can easily recognize the bleeding site because the moment of bleeding is slow-regenerated. Also, since it changes to slow regeneration immediately before bleeding, the operator can predict that bleeding will occur in the future. Further, by adopting partial slow playback, it is possible to avoid a long playback time.

The time setting in partial slow reverse playback is the same as the time setting in partial slow playback. The reproduction unit 314 reversely reproduces the partial moving image from 0.2 seconds to 2 seconds after the start of bleeding at a normal reproduction speed (reproduction speed 1), and 0.2 seconds to 0.2 seconds before the start of bleeding. Partial moving images up to a second later are played back in slow reverse at a playback speed (playback speed 2) that is 1/5 times the normal playback speed. The reproduction speed in slow reverse reproduction is not limited to 1/5 times speed, and may be appropriately set. After the slow playback, the partial moving image from 3 seconds before the start of bleeding to 0.2 seconds before is played in reverse at the normal playback speed (playback speed 1). In partial slow reverse regeneration, the spread bleeding is displayed so as to gradually focus on one place, so that the operator can easily recognize the bleeding place by grasping the focusing point.

FIG. 9 describes a configuration in which a partial moving image is played back in the order of normal playback, partial slow playback, and partial slow reverse playback. Such a reproduction order and a reproduction method may be set in the control unit 30 by default. Further, the control unit 30 may accept changes in the reproduction order and the reproduction method through the operation unit 20. The reproduction order and the reproduction method may be arbitrarily set by the operator. For example, the surgeon may change the setting to play the partial video in the order of partial slow playback, partial slow reverse playback, and partial slow playback, partial slow playback, reverse playback, and partial slow reverse playback. You may change the settings to play the video. When the control unit 30 accepts a change in the setting of the reproduction order and the reproduction method, the control unit 30 reproduces the partial moving image according to the changed reproduction order and the reproduction method.

As described above, in the sixth embodiment, since the bleeding scene is displayed as a moving image only when the bleeding of a predetermined amount or more is detected, only the bleeding scene that the surgeon should recognize without interfering with the operation. Can be displayed.

In the sixth embodiment, the threshold value for the amount of bleeding is set in advance, but the threshold value may be set by an operator such as a surgeon through the operation input unit 311. When the threshold setting is accepted, the bleeding amount detection unit 315 may compare the bleeding amount estimated from the image with the threshold value after the setting to determine the presence or absence of bleeding of a predetermined amount or more.

<7th Embodiment>
In the seventh embodiment, the bleeding above the threshold value generated during imaging is recognized, and the recognized bleeding scene is displayed as a thumbnail on the display 21. When the control unit 30 accepts the selection for the thumbnail, it reproduces the moving image of the corresponding bleeding scene.

FIG. 20 is a diagram illustrating a processing flow of the control unit 30 according to the seventh embodiment. The difference from the process of the sixth embodiment (FIG. 18) will be described. When the control unit 30 detects a predetermined amount or more of bleeding by the same procedure as in the sixth embodiment (S481), the control unit 30 generates a thumbnail showing the scene of the bleeding, and superimposes the generated thumbnail on the surgical field image. It is displayed on the display 21 (S482). The thumbnail may display a still image or moving image that is representative of the bleeding scene. Further, the control unit 30 stores, for example, information at the bleeding start time t0 in the parameter storage unit 332 in association with the thumbnail as information for identifying the bleeding scene (S483).

FIG. 21 is a diagram showing a display example of a thumbnail showing a bleeding scene. FIG. 21 shows an example in which two thumbnails TH1 and TH2 are superimposed on the surgical field image and displayed on the display 21. The thumbnails TH1 and TH2 are preferably arranged so as to avoid the area near the center of the surgical field image so as not to interfere with the recognition of the treated portion. Therefore, for example, the control unit 30 may arrange and display the generated thumbnails TH1 and TH2 in the end region of the surgical field image. Alternatively, the control unit 30 may grasp the positions of the

surgical instruments

11a and 11b by image analysis and display the thumbnails TH1 and TH2 while avoiding the grasped positions of the

surgical instruments

11a and 11b. Further, the control unit 30 may add information on the detection time when bleeding of a predetermined amount or more is detected to each thumbnail TH1 and TH2.

The control unit 30 determines whether or not the selection for the thumbnail has been accepted through the operation input unit 311 (S484). The operation input unit 311 may accept selection for thumbnails using an input device such as a joystick or a foot pedal included in the controller unit 22, or may accept voice instructions for selecting thumbnails using a microphone. If the selection is not accepted (S484: NO), the control unit 30 returns the process to step S481.

The control unit 30 adds a new thumbnail to the surgical field image each time it detects bleeding exceeding a predetermined amount. If the number of thumbnails increases too much, it becomes difficult to confirm the affected area T in the surgical field image, so the number of thumbnails to be displayed may be limited. For example, the control unit 30 may select and display a predetermined number of thumbnails in order from the one with the newest detection time. Further, instead of the configuration in which the thumbnail is superimposed and displayed on the surgical field image, the surgical field image may be displayed on the display 21 and the thumbnail may be separately displayed on the sub-display.

When the selection operation for the thumbnail is accepted (S484: YES), the partial moving image extraction unit 313 reads the parameter and playback method list from the parameter storage unit 332 and the setting storage unit 333 for the bleeding scene indicated by the selected thumbnail (S403). ). At this time, the partial moving image extraction unit 313 can read out the necessary parameters and the reproduction method list using the information of the bleeding start time t0 associated with the thumbnail as a search key.

The processing after reading out the parameters and the playback method list is the same as that of the sixth embodiment. To play. The reproduction unit 314 may reproduce the partial moving image according to a reproduction method including at least one of normal reproduction, slow reproduction, frame advance reproduction, repeated reproduction, and reverse reproduction. The reproduction unit 314 may superimpose and output a partial moving image as a wipe image W as shown in FIG. 9 on a real-time image (surgical field image) from the endoscope camera 13, or may output a sub different from the display 21. A display or the like may be provided to output a partial moving image to the sub-display. Further, the reproduction unit 314 may repeatedly reproduce the partial moving image while switching at least two of normal reproduction, slow reproduction, frame advance reproduction, and reverse reproduction. Further, the reproduction unit 314 may reproduce the scene including the start time of bleeding at a speed lower than the reproduction speed of the scene before and after the start of bleeding.

As described above, since the surgical system 1 in the seventh embodiment has a configuration in which the moving image of the bleeding scene is reproduced only when the thumbnail is selected, only the bleeding scene desired by the surgeon such as the surgeon is used as the moving image. Can be provided.

<8th Embodiment>
In the eighth embodiment, when bleeding is detected in the end region of the surgical field region obtained from the endoscopic camera 13, a configuration for notifying that bleeding has been detected will be described.

FIG. 22 is a diagram illustrating a processing flow of the control unit 30 according to the eighth embodiment. The difference from the process of the sixth embodiment (FIG. 18) will be described. When the control unit 30 detects a predetermined amount or more of bleeding by the same procedure as in the sixth embodiment (S491), the control unit 30 determines whether or not the bleeding has occurred in the end region of the surgical field image (S492). .. The end region may be set as an region excluding the vicinity of the center of the surgical field image (area of interest). In one example, it is the upper, lower, left, and lower quarter areas of the surgical field image.

When it is determined that bleeding has occurred in the end region of the surgical field image (S492: YES), the control unit 30 notifies the information that bleeding has occurred (S493). The control unit 30 displays, for example, character information or an icon indicating that bleeding has occurred on the display 21. Alternatively, the control unit 30 may output voice information indicating that bleeding has occurred from the speaker 23.

As described above, in the eighth embodiment, when it is determined that bleeding has occurred in the end region of the surgical field image, information indicating that bleeding has occurred is notified, so that the operator is aware of the presence of bleeding that is difficult to notice. Can be recognized.

<9th embodiment>
In the ninth embodiment, a configuration for displaying a graph showing a time-series change in the amount of bleeding will be described.

FIG. 23 is a diagram showing a display example of a graph showing a time-series change in the amount of bleeding. The bleeding amount detection unit 315 of the control unit 30 can detect the bleeding amount at each time by analyzing the frame F at each time. The control unit 30 may generate a graph showing the time-series change in the amount of bleeding and display it together with the surgical field image. FIG. 23 shows an example in which a graph GR showing a time-series change in the amount of bleeding is displayed on the lower side of the surgical field image. The horizontal axis of the graph GR represents time, and the vertical axis represents the amount of bleeding. In this graph GR, the time Ta represents the bleeding start time point and the time Tb represents the hemostasis time point. Instead of the amount of bleeding, the time-series change in the number of pixels or the area corresponding to the bleeding may be displayed as a graph.

Further, as shown in FIG. 23, the control unit 30 may clearly indicate a point on the graph GR in which the amount of bleeding exceeds a predetermined amount by a mark MK or the like. Further, the control unit 30 may highlight the portion of the graph GR corresponding to the graph GR by displaying it in a thick line or changing the color in order to clearly indicate the portion exceeding the threshold value. Further, the control unit 30 may display the reproduction section (t0-i1 ≦ t ≦ t0 + i2) of the partial moving image together with the graph GR. Further, the control unit 30 may also display the graph GR during the reproduction of the partial moving image.

As described above, in the ninth embodiment, since the graph showing the time-series change in the amount of bleeding is displayed, the operator can grasp the state of bleeding that has occurred in the past.

<10th Embodiment>
In the tenth embodiment, when the surgical field image contains a resection device, the presence or absence of bleeding of a predetermined amount or more is determined, and when the surgical field image contains a hemostatic device, the determination is not performed. explain.

FIG. 24 is a diagram illustrating a processing flow of the control unit 30 according to the tenth embodiment. The difference from the process of the first embodiment (FIG. 10) will be described. When an event (bleeding in this embodiment) is detected in step S401, the image of the frame F obtained from the endoscopic camera 13 is analyzed to determine whether or not the surgical field image includes a resection instrument. (S501). The control unit 30 can determine whether or not the surgical field image includes a cutting instrument by determining whether or not a figure having a predetermined shape is included by using a technique such as template matching. can. If the excision device is not included (S501: NO), the control unit 30 returns the process to step S401.

When the excision device is included (S501: YES), the control unit 30 analyzes the image of the frame F to detect the amount of bleeding (S502). As described in the second embodiment, the bleeding amount detecting unit 315 may detect the bleeding amount according to the number of pixels having a color corresponding to bleeding and the area of the region. The bleeding amount detection unit 315 detects the bleeding amount each time the image of the frame F is input, and determines the presence or absence of bleeding of a predetermined amount or more by comparing with a preset threshold value (S503). If there is no bleeding greater than or equal to the predetermined amount (S503: NO), the control unit 30 returns the process to step S401.

When it is determined that there is bleeding of a predetermined amount or more (S503: YES), the control unit 30 executes the processes after step S402 described in the first embodiment. That is, the control unit 30 reads out various parameters related to the reproduction method (for example, reproduction speed, reproduction number of repeated reproductions, etc.) and a reproduction method list according to the operation type such as excision, peeling, traction, suturing, and suction. , The partial moving image extracted from the recorded video M1 is played back based on the read parameters and the playback method list. The method for extracting the partial moving image is the same as that in the first embodiment. That is, the partial moving image extraction unit 313 is a frame included in the time range from the time point before the start of bleeding (t0-i1) to the time point after bleeding (t0 + i2) in the recorded video M1 recorded in the video recording unit 331. F may be extracted as a partial moving image. In this embodiment, t0 represents the time point at which bleeding begins. The times i1 and i2 may be set in advance, or may be set according to the operation type such as excision, peeling, traction, suturing, and suction, and the imaged scene.

Further, the control unit 30 analyzes the image of the frame F obtained from the endoscopic camera 13 when playing back the partial moving image, and determines whether or not the surgical field image includes a hemostatic device (S504). .. The control unit 30 can determine whether or not a hemostatic device is included in the surgical field image by determining whether or not a figure having a predetermined shape is included by using a technique such as template matching. can. When the hemostatic device is not included (S504: NO), the control unit 30 plays a partial moving image (S407). On the other hand, when the hemostatic device is included (S504: YES), the control unit 30 ends the process according to this flowchart without reproducing the partial moving image.

As described above, in the tenth embodiment, when the surgical site is excised using the excision instrument, the partial moving image is reproduced only when the amount of bleeding increases to a predetermined amount or more. In addition, when a hemostatic device appears in the surgical field image, it is considered that the hemostatic site is already known, so that it is possible to control not to play the partial moving image.

Although the present embodiment has been described above, the above embodiment is for facilitating the understanding of the present invention, and is not for limiting and interpreting the present invention. The present invention can be modified and improved without departing from the spirit thereof, and the present invention also includes an equivalent thereof.

For example, in the above embodiment, the case where the event is bleeding B from a site near the affected part T of the patient 100 has been described, but for example, damage, mistreatment, or other abnormality may be detected.

This surgical system can be applied not only to surgery but also to all the procedures that may cause the above-mentioned accidents. For example, diagnosis, examination, diagnostic puncture / sample collection, imaging examination, medication / injection (including blood transfusion). ), Rehabilitation, anesthesia, radiotherapy, use of medical equipment, etc., as long as it can include accidents caused by or suspected to be caused by medical treatment.

The number of surgical systems in surgery is not limited to one, and there may be multiple. In this case, the movements of the operation unit and the work unit may be synchronized with each other, or they function independently (so that the movements of the first surgical device and the second surgical device are not linked). It may be a matter of fact.

In addition, the surgical target of this system is not limited to the human body, but may be an animal, an object as a training target, a plant, or the like.

Further, in each of the above-described embodiments, the same effect can be obtained by detecting digestive juice, bile, etc. discharged due to damage to organs, digestive tract, etc. instead of the amount of bleeding.

Further, in each of the above-described embodiments, it is assumed that the event is an incident such as bleeding, but for example, a specific operation with a surgical instrument may be detected as an event. For example, it is possible to detect the suturing / anastomosis with a stapler (stapler) as an event and reproduce the situation at the time of suturing / anastomosis, so that it is possible to confirm whether or not the treatment has been performed correctly. ..

Further, in the present embodiment, a real-time image during the operation for the patient 100 is assumed, but it is also useful in a situation where the recorded image M1 recording the operation situation is viewed later. For example, it can be used when the surgeon 110 browses the recorded video M1 of the surgery performed by himself or another surgeon and refers to the cause and location of the event for future surgery. Is.

1 Surgical system 10 Surgical unit 11 Instrument department (Treatment department)
13 Endoscopic camera (video acquisition section)
20 Operation unit 21 Display (display unit)
30 Control unit 33 Storage 311 Operation input unit 312 Occurrence time identification unit 313 Partial video extraction unit 314 Playback unit 331 Video recording unit 332 Parameter storage unit 333 Setting storage unit

Claims

On the computer
Record the surgical field image obtained by chronologically imaging the surgical field of arthroscopic surgery,
Based on the above-mentioned surgical field image, the presence or absence of bleeding exceeding a predetermined amount is determined.
When it is determined that there is bleeding of a predetermined amount or more, a process of playing back a partial video of the time range including the time before the start of bleeding to the time after bleeding in the recorded surgical field video by the set playback method is performed. A computer program to run.
The computer program according to claim 1, wherein the playback method includes at least one of normal playback, slow playback, frame-by-frame playback, repeated playback, and reverse playback.
The computer program according to claim 1, wherein the playback method includes a method of repeatedly playing back while switching at least two of normal playback, slow playback, frame-by-frame playback, and reverse playback.
The computer according to any one of claims 1 to 3, wherein the reproduction method includes a method of reducing the reproduction speed of the scene including the start time of bleeding to be lower than the reproduction speed of the scene before and after the start of bleeding. program.
On the computer
Claims 1 to 4 for displaying a surgical field image captured during the arthroscopic surgery in one display area and executing a process of reproducing a partial image in the time range in another display area. The computer program described in any one of the above.
On the computer
When multiple bleedings exceeding the specified amount are detected, thumbnails showing each bleeding scene are displayed.
The computer program according to any one of claims 1 to 5, wherein when the selection for the displayed thumbnail is accepted, the process of reproducing the partial image of the corresponding bleeding scene by the reproduction method is executed.
On the computer
The computer program according to any one of claims 1 to 6, for executing a process of notifying information that bleeding has been detected when bleeding is detected in the end region of the surgical field image.
On the computer
The computer program according to any one of claims 1 to 7, for executing a process of displaying a graph showing a time-series change in the amount of bleeding.
Using a computer
Record the surgical field image obtained by chronologically imaging the surgical field of arthroscopic surgery,
Based on the above-mentioned surgical field image, the presence or absence of bleeding exceeding a predetermined amount is determined.
If it is determined that there is bleeding of a predetermined amount or more, the partial video of the time range including the time before the start of bleeding to the time after bleeding is played back by the set playback method from the recorded surgical field video. Method.