JP2005110878A - Operation supporting system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an operation supporting system capable of easily and appropriately recognizing live images and relating images and executing appropriate supporting. <P>SOLUTION: The supporting device 104 of a conference room 103 is provided with functions such as an integrated image generating function 151, a full screen display / normal area display switching function 152, a thumbnail source image displaying function 153, a relation / live switching function 154, a thumbnail multi-display switching function 155, a moving image reproducing function 156, a linked information retrieving function 157, a thumbnail image generating function 158, and a data communication function 159. Also, a supporting monitor of the supporting device 104 is provided with a pointing device such as a mouse or a touch panel and a GUI is constructed. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a surgery support system that supports surgery in real time with reference to images.

  Usually, an operator in the operating room performs an operation on a patient. For example, in the case of an operation that requires a high degree of skill for an operator in the operating room, a remote place familiar with the operation (even in the hospital, the operating room and Connected to a supporter (remote support surgeon) in a remote location, for example, a conference room, via a line, and during surgery, the surgeon in the operating room performs an operation with advice on the surgical policy by remote support By doing so, a system for supporting a patient in the operating room so that more appropriate surgery can be performed is conceivable. Such a remote support system is proposed in, for example, Japanese Patent Laid-Open No. 2002-306509. Has been.

In such a support system, the support side performs support by referring to a related image such as a CT image or MR image of a patient together with a live image of a procedure. Further, in such a support system, a procedure image in the operating room is recorded. For example, an endoscopic image is recorded on an in-hospital server or the like when an affected area is observed using an endoscopic apparatus or a treatment is performed, and the post-operative operation is performed in the in-hospital conference room support apparatus in which the support system is constructed. Endoscopic images are played back, and subsequent guidance on procedures is performed.
JP 2002-306509 A

  However, in the conventional support system, if the live image and the related image are displayed on one monitor, the visual recognition of the image is hindered. Therefore, when referring to the image, it is necessary to use a monitor larger than necessary or a plurality of monitors. There is a problem that there is.

  The present invention has been made in view of the above circumstances, and an object thereof is to provide a surgery support system that can easily and appropriately recognize a live image and a related image and perform appropriate support.

  The surgical operation support system according to the present invention includes an image data obtaining unit for obtaining a live image during a procedure and a plurality of related images related to a procedure taken before the procedure, and a plurality of thumbnails of the live image and the plurality of related images. An integrated screen generating unit that generates an integrated screen composed of images, and a whole area display instruction unit that displays the live image or the related image over the entire display area of the integrated screen.

  According to the present invention, there is an effect that it is possible to recognize live images and related images easily and appropriately and perform appropriate support.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIGS. 1 to 21 relate to the first embodiment of the present invention, FIG. 1 is a block diagram showing the configuration of the surgery support system, and FIG. 2 is displayed on the operating room display device and conference room endoscope image monitor of FIG. FIG. 3 is a block diagram showing a functional configuration of the support apparatus of FIG. 1, FIG. 4 is an integrated screen generation function, link information search function, thumbnail image generation function, and data communication function of FIG. FIG. 5 is a diagram showing a link destination directory used for search by the link information search function of FIG. 3, FIG. 6 is a diagram showing an example of the configuration of IP addresses in the link destination directory of FIG. 7 is a diagram showing the prefecture number used for the IP address in FIG. 6, FIG. 8 is a diagram showing an integrated screen generated by the processing in FIG. 4, and FIG. 9 is a full screen display / normal area display switching in FIG. , A thumbnail original image display function, a related / live switching function, a thumbnail multi-display switching function, a moving image playback function, a data communication function, a flowchart for explaining the operation of the integrated screen, and FIG. 11 is a second view of the integrated screen for explaining the processing of FIG. 9, FIG. 12 is a third view of the integrated screen for explaining the processing of FIG. 9, and FIG. 13 is for explaining the processing of FIG. FIG. 14 is a fifth view of the integrated screen for explaining the processing of FIG. 9, FIG. 15 is a sixth view of the integrated screen for explaining the processing of FIG. 9, and FIG. FIG. 17 is an eighth diagram of an integrated screen for explaining the processing of FIG. 9, FIG. 18 is a ninth diagram of an integrated screen for explaining the processing of FIG. 19 is a tenth view of the integrated screen for explaining the processing of FIG. 9, and FIG. 20 is for explaining the processing of FIG. Figure 11 of the integrated screen, FIG. 21 is a twelfth diagram of integrated screen for explaining the processing of FIG.

  As shown in FIG. 1, the surgical operation support system employed in this embodiment is installed in a hospital having a base such as an operating room 100, a CT examination room 101, an MR examination room 102, a conference room 103, etc. The first controller 1 that obtains various medical information that is generated, and the operating room 100 are installed at remote locations, and the surgical unit server 2 that accumulates various medical treatment information obtained by the first controller 1; In-hospital server 3 as a patient information storage means that is installed at a base different from the surgical department server 2 and combines the medical procedure information obtained from the surgical department server 2 and the ordering information, and supports the procedure in the operating room 100 The operating room 100 is configured to include a support device 104 provided in a conference room 103 installed at a remote location, and the first controller 1, CT examination 101 CT examination apparatus, MR examination apparatus of MR examination room 102, support apparatus 104 and the operation section server 2 and between the operation section server 2 and the hospital server 3 are respectively the first and second Connected by communication lines 4 and 5.

  Each of the communication lines 4 and 5 may be any communication line such as a public line, a dedicated line, or a LAN (local communication network). Further, the in-hospital server 3 may be disposed at the same base as the surgical department server 2.

  The second controller 14 captures a video signal obtained by the endoscope apparatus 11 and an endoscope apparatus 11 that obtains a video signal including an endoscopic image by imaging a test site such as a body cavity of a patient. Display device 12 and a surgical device 13 such as an electrosurgical device, a pneumoperitoneum device, and an ultrasonic surgical device, for example, to control the surgical device 13 or to obtain endoscopic image data from the endoscope device 11 Surgical device information such as measurement information by the surgical device 13 is obtained.

  The second controller 14 is connected to surgical device control input means such as a touch panel 15 for inputting a control instruction or the like for the surgical device 13 to the second controller 14.

  Endoscopic image data, surgical device information, and patient information obtained from the living body monitoring device 19 provided from the second controller 14 are accumulated in the first controller 1 via a communication line, and set in another base. Is transmitted to the operating unit server 2 via the communication line 4.

  The video signal obtained from the endoscope apparatus 11 and the video signal obtained from the indoor camera 20 that captures the scenery in the operating room 100 are input to the signal transmission device 17 via the first controller 1. And transmitted to the outside via the communication line 6.

  Further, the operating room information obtained by the first controller 1 in the operating room, patient information obtained from the operating section server 2, integrated image data which is a support image obtained from the support apparatus 104 in the conference room 103, A display device 18 for displaying information received by the signal transmission device 17 from another base is provided.

  An input device 1 a is connected to the first controller 1. The input device 1a is a keyboard, a magnetic card reader, an optical card reader, an IC card reader, an RF-ID (IC tag) reader, or the like. Using this input device 1a, a chart number (ID number) and a name related to a patient Procedure information such as patient information, procedure name, and operator ID is input. In this case, patient information and procedure information are transmitted from the first controller 1 to the second controller 14.

  Note that the input device 1 a may be connected to the second controller 14. When the input device 1 a is connected to the second controller 14, patient information and procedure information are transmitted from the second controller 14 to the first controller 1.

  The surgical department server 2 is connected to the hospital server 3 and the WEB server 21 via the communication line 5. The WEB server 21 is also connected to the outside of the hospital via the Internet, and when the patient is examined at another medical institution outside the hospital, the examination image data (MR, CT, X-ray image, etc.) ) Is stored in the out-of-hospital servers 201, 202 and the like in the medical institution, but when the WEB server 21 accesses the out-of-hospital servers 201, 202 via the Internet, the support device 104 and the The inspection image data stored in the out-of-hospital servers 201 and 202 can be downloaded to the first controller 1.

  The in-hospital server 3 includes patient registration information from the patient registration terminal 22 of the hospital, examination images (MR, CT, X-ray images, etc.) such as preoperative images related to the patient, and surgical images (internal) Endoscope still images, endoscope moving images) are accumulated. Hereinafter, these inspection images and surgical images are collectively referred to as medical images.

  Prior to the operation, the surgical operation section server 2 reads out and stores registration information such as the name, date of birth, and medical record of the patient performing the operation and medical information related to the patient from the in-hospital server 3 via the communication line 5. Be able to. The medical information includes medical image data obtained from the in-hospital server 3, the IP addresses of the out-of-hospital servers 201 and 202, a folder name (link information) storing the medical image data file, and the like.

  The support device 104 provided in the conference room 103 includes an endoscope image monitor 111 that displays an endoscopic image obtained by the endoscope device 11 in the operating room 100 and a support image generated by the support device 104. And a support monitor 112 that displays a certain integrated screen (described later), and the endoscope image monitor 111 is configured to receive an endoscope through the second controller 14, the first controller 1, and the operating room communication line 4. The live endoscopic image captured by the apparatus 11 is displayed, and the live endoscopic image, the medical image relating to the patient stored in the surgical operation section server 2, and downloaded via the WEB server 21. The medical images stored in the out-of-hospital servers 201 and 202 are displayed on the support monitor 112 as an integrated screen composed of a plurality of windows.

  As shown in FIG. 2, the display device 12 in the operating room 100 and the endoscopic image monitor 111 in the conference room 103 have a display area in which a live endoscopic image (hereinafter, endoscopic live image) is a full screen. 121 is displayed.

  As shown in FIG. 3, the support apparatus 104 in the conference room 103 includes an integrated screen generation function 151 as an integrated image generation unit, a full screen display / normal area display switching function 152 as a whole area display instruction unit, and a thumbnail original image display function. 153, a related / live switching function 154, a thumbnail multi-display switching function 155, a video playback function 156, a link information search function 157, a thumbnail image generation function 158, a data communication function 159 as an image data acquisition means, etc. The support monitor 112 is provided with a pointing device (not shown) such as a mouse or a touch panel to construct a GUI. By storing various data in the data storage unit 150 and reading / writing, the above functions are supported by the support monitor. To perform processing based on 112 GUI That. Details of the processing of each function will be described later.

  The operation of this embodiment configured as described above will be described. As shown in FIG. 4, the support apparatus 104 uses the data communication function 159 to register the patient name, date of birth, medical record information, etc., and medical image information related to the patient who performs the operation from the surgical operation section server 2 in step S1. Obtain patient information consisting of

  Next, in step S2, the link information retrieval function 157 retrieves the link information of the link destination of the related image related to the patient from the medical image information. The search for the link information of the link destination of the related image is performed by reading the link destination directory having the structure as shown in FIG. In step S3, the IP addresses of the in-hospital server 3 and the (first) out-of-hospital server 201 and (second) out-of-hospital server 202 are acquired from the link destination directory. In step S4, the in-hospital server 3 and the out-of-hospital servers 201 and 202 are accessed. By designating the folder name, the related image data is downloaded from the in-hospital server 3 and the out-of-hospital servers 201 and 202 via the surgical department server 2. An example of the structure of the IP address specified in the link destination directory is shown in FIG. 6, and an example of the prefecture number used for the IP address is shown in FIG.

  When the downloading of the related image data is completed, a thumbnail image is generated from the related image data by the thumbnail image generation function 158 in step S5.

  Specifically, the related image data is composed of a plurality of types of medical image files (endoscopic moving image file, endoscope still image file, CT still image file, MR still image file, etc.). A file includes a single image file such as a still image and a plurality of frame images such as a moving image file.

  Therefore, the thumbnail image generation function 158 generates related image data, that is, a “thumbnail image” indicating a medical image file and a “related thumbnail image” that is a thumbnail indicating a predetermined frame image of each medical related image file data.

  In the present embodiment, for example, the “thumbnail image” is matched with the thumbnail of the first medical related image file data among a plurality of “related thumbnail images”.

  When thumbnails are generated in this way, the integrated screen generation function 151 generates an integrated screen 160 as shown in FIG. 8 and displays it on the support monitor 112 in step S6. In step S7, the data communication function 159 transmits the link information (link destination directory) including the link destination IP address and the image folder name to the first controller 1 and ends the process.

  When the first controller 1 receives link information (link destination directory) including the link destination IP address and the image folder name, the first controller 1 accesses the link destination IP address and performs the same processing as the above steps S4 to S6. The integrated screen 160 having thumbnails is generated and displayed on the display device 18.

  As shown in FIG. 8, the integrated screen 160 displays an image display area 161 for displaying an endoscope live image or the like in normal display, and a thumbnail for displaying a plurality of “thumbnail images” indicating each medical image file of related image data. An image display area 162, a related thumbnail image display area 163 for displaying a plurality of “related thumbnail images” that are thumbnails of the respective medical related image file data in the related image data, and a key area 164 including various keys SW are provided. The key area 164 is provided with at least a related / live switch SW165, a multiswitch SW166, and an enlargement SW167.

  Next, the operation on the integrated screen 160 will be described. As shown in FIG. 9, when the integrated screen 160 is displayed on the support monitor 112 in step S11, it is determined whether or not the cursor 168 has selected the enlargement SW 167 by the GUI of the support monitor 112 in step S12, and the enlargement SW 167 is selected. Then, in step S13, integration of the frame structure including the full screen display area 169 for enlarging the image displayed in the image display area 161 as shown in FIG. 10 by the full screen display / normal area display switching function 152 is performed. The screen 160a is generated and the integrated screen 160a is displayed on the support monitor 112, and the screen frame information of the integrated screen 160a is transmitted to the first controller 1 by the data communication function 159 in step S14. Here, the endoscope live image is displayed in full screen.

  The first controller 1 generates an integrated screen 160 a based on the transmitted screen frame information and displays it on the display device 18. Therefore, the display on the support monitor 112 matches the display on the display device 18. Here, in the integrated screen 160a, the enlargement SW 167 replaces the reduction SW 167a.

  In step S15, the cursor 168 waits for selection of the reduced SW 167a on the integrated screen 160a by the GUI of the support monitor 112. When the reduced SW 167a is selected, the frame structure integrated screen 160 shown in FIG. 8 is returned in step S16. In step S17, the data communication function 159 transmits the screen frame information of the integrated screen 160 to the first controller 1.

  The first controller 1 returns to the integrated screen 160 based on the transmitted screen frame information and displays it on the display device 18. Therefore, the display on the support monitor 112 matches the display on the display device 18.

  In step S18, the process waits for an instruction to end the integrated screen display (support end). If there is an instruction, the process ends. If there is no instruction, the process returns to step S12.

  If the enlargement SW 167 is not selected in step S12, it is determined in step S19 whether the cursor 168 has selected any “thumbnail image” in the thumbnail image display area 162 using the GUI of the support monitor 112, and “thumbnail image” is selected. Then, in step S20, the integrated screen 160 is generated as the original image of the “thumbnail image” in which the image in the image display area 161 is selected by the thumbnail original image display function 153, and the image in the image display area 161 is selected. The screen frame information as the original image of the “thumbnail image” is transmitted to the first controller 1.

  The first controller 1 generates the integrated screen 160 based on the transmitted screen frame information and displays it on the display device 18. Therefore, the display on the support monitor 112 matches the display on the display device 18.

  For example, as shown in FIG. 11, when an image in the thumbnail image display area 162 is selected as the CT still image file 1 thumbnail image 171, as shown in FIG. 12, the image in the image display area 161 is converted into the CT still image by the thumbnail original image display function 153. The original image of the image file 1 thumbnail image 171 is used. In the present embodiment, as described above, the “thumbnail image” matches the thumbnail of the first medical-related image file data among the plurality of “related thumbnail images”, so that the image display area 161 in FIG. The image is an original image of the CT still image file 1 related image 1 thumbnail image 172 in the related thumbnail image display area 163.

  Here, by displaying the frame of the CT still image file 1 thumbnail image 171 and the frame of the CT still image file 1 related image 1 thumbnail image 172 in a thick frame, it is possible to recognize which thumbnail is the original image.

  Next, in step S22, it is determined by the GUI of the support monitor 112 whether or not the cursor 168 has selected the related / live switching SW 165. The integrated live screen 160 that switches the live image of the endoscope and the original image of the selected “thumbnail image” into the image display area 161 is returned to step S21. If the related / live switch SW165 is not selected, the process returns to step S12.

  The first controller 1 generates the integrated screen 160 based on the transmitted screen frame information and displays it on the display device 18. Therefore, the display on the support monitor 112 matches the display on the display device 18.

  As a result, for example, when the related / live switching SW 165 is selected on the integrated screen 160 in the state of FIG. 12, the state is switched to the integrated screen 160 in the state of FIG. When SW165 is selected, the screen is switched to the integrated screen 160 in the state of FIG.

  12 returns to step S12 from step S22 on the integrated screen 160 in the state of FIG. 12, and when the enlargement SW 167 is selected as shown in FIG. 13, the “thumbnail image” selected as shown in FIG. Will be displayed in full screen. Further, as shown in FIG. 15, when the related / live switching SW 165 is selected in a state where the original image of the selected “thumbnail image” is displayed in full screen, the processing in step S23 is performed as shown in FIG. Endoscope live images are displayed in full screen.

  Further, when the “thumbnail image” is selected in step S19 using the related image as a moving image, the moving image is composed of a plurality of continuous scene moving images. Therefore, for example, as shown in FIG. 16, the endoscope moving image file 1 thumbnail. When the image 175 is selected, the image in the image display area 161 is made the original image of the endoscope moving image file 1 thumbnail image 175 by the thumbnail original image display function 153. At this time, in the related thumbnail image display area 163, the first frame images of a plurality of scene moving images constituting the moving image are displayed as “related thumbnail images”. As a result, in this embodiment, the image in the image display area 161 is the endoscope moving image file 1 frame image 1.

  In the integrated screen 160 in the state of FIG. 16, a moving image playback operation key area 180 is displayed in the lower part of the image display area 161. When the playback button 181 in the movie playback operation key area 180 is selected, the playback image of the endoscope movie file 1 is displayed as a movie in the image display area 161 by the movie playback function 156 as shown in FIG.

  When the enlargement SW 167 is selected as shown in FIG. 18 on the integrated screen 160 in the state of FIG. 17, the reproduction image of the endoscope moving image file 1 is displayed in full screen as shown in FIG. Will be. Further, when the related / live switch SW165 is selected in a state where the playback image of the endoscope moving image file 1 is displayed in full screen as shown in FIG. 19, the processing in step S23 causes the internal video to be displayed as shown in FIG. Display live endoscope images in full screen.

  Returning to FIG. 9, if “thumbnail image” is not selected in step S19, it is determined in step S24 whether or not the cursor 168 has selected the multi-switch SW 166 by the GUI of the support monitor 112, and when the multi-switch SW 166 is selected, In step S25, the integrated screen 160 in which the thumbnail display in the thumbnail image display area 162 is changed is generated by the thumbnail multi-display switching function 155, and the screen frame information of the integrated screen 160 is transmitted to the first controller 1 in step S25.

  The first controller 1 generates the integrated screen 160 based on the transmitted screen frame information and displays it on the display device 18. Therefore, the display on the support monitor 112 matches the display on the display device 18.

  For example, in the integrated screen 160 in FIG. 20, the thumbnail display in the thumbnail image display area 162 is displayed in three columns. When the multi-switch SW 166 is selected in this state, the thumbnail multi-display switch function 155 shows that in FIG. As described above, the integrated screen 160 in which the thumbnail display in the thumbnail image display area 162 is displayed in one column is generated.

  As described above, in this embodiment, by selecting the enlargement SW 167, the image displayed in the image display area 161 can be displayed in full screen, so that it is possible to easily and accurately obtain image information necessary for support. Become. In addition, by selecting the multi-switch SW 166, the thumbnail display in the thumbnail image display area 162 can be enlarged, so that it is easy to refer to, for example, an endoscope live image displayed in the image display area 161 and a thumbnail, and support is provided. Necessary image information can be obtained easily and accurately.

  In this embodiment, the integrated image is shared between the conference room side and the operating room side by operating the integrated image on the conference room side and transmitting the image frame information of the integrated image to the operating room side. However, the integrated image may be shared by operating the integrated image on the operating room side and transmitting image frame information of the integrated image to the conference room side.

  When operating the integrated image on the conference room side and transmitting the image frame information of the integrated image to the operating room side, it is suitable for assisting the operating room doctor when there is a supervisor on the conference room side, When operating the integrated image on the operating room side and transmitting image frame information of the integrated image to the conference room side, if there is a new doctor or medical student on the conference room side, the operating room support doctor will show the progress of the operating room This is suitable for explaining new doctors and medical students.

  The present invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the scope of the present invention.

The block diagram which shows the structure of the surgery assistance system which concerns on Example 1 of this invention. The figure which shows the live endoscopic image displayed on the display apparatus and conference room endoscopic image monitor of the operating room of FIG. The block diagram which shows the function structure of the assistance apparatus of FIG. Flowchart for explaining the operation of the integrated screen generation function, link information search function, thumbnail image generation function, and data communication function of FIG. The figure which shows the link destination directory used for the search by the link information search function of FIG. The figure which shows an example of a structure of the IP address in the link destination directory of FIG. The figure which shows the prefecture number used for the IP address of FIG. The figure which shows the integrated screen produced | generated by the process of FIG. FIG. 3 is a flowchart for explaining the operation of the integrated screen by the full screen display / normal area display switching function, thumbnail original image display function, related / live switching function, thumbnail multi-display switching function, video playback function, and data communication function. First view of the integrated screen for explaining the processing of FIG. Second view of the integrated screen for explaining the processing of FIG. Third view of the integrated screen for explaining the processing of FIG. 4th figure of the integrated screen explaining the process of FIG. The 5th figure of the integrated screen explaining the process of FIG. The 6th figure of the integrated screen explaining the process of FIG. FIG. 9 is a seventh view of the integrated screen for explaining the processing of FIG. 8th figure of the integrated screen explaining the process of FIG. FIG. 9 is a ninth diagram of the integrated screen for explaining the processing of FIG. FIG. 10 is a tenth view of the integrated screen for explaining the processing of FIG. The eleventh figure of the integrated screen explaining the process of FIG. 12th figure of the integrated screen explaining the process of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... 1st controller 1a ... Input device 2 ... Surgical department server 3 ... Hospital server 4,5 ... Communication circuit 11 ... Endoscope apparatus 12 ... Display apparatus 13 ... Surgical apparatus 14 ... 2nd controller 103 ... Conference room 104 ... support device 150 ... data storage unit 151 ... integrated screen generation function 152 ... full screen display / normal area display switching function 153 ... thumbnail original image display function 154 ... related / live switching function 155 ... thumbnail multi-display switching function 156 ... video playback Function 157 ... Link information search function 158 ... Thumbnail image generation function 159 ... Data communication function 160 ... Integrated screen Agent Patent attorney Susumu Ito

Claims (3)

Image data obtaining means for obtaining a live image during the procedure and a plurality of related images related to the procedure taken before the procedure;
An integrated screen generating means for generating an integrated screen composed of a plurality of thumbnail images of the live image and the plurality of related images;
An operation support system comprising: a whole area display instruction means for displaying the live image or the related image over the entire display area of the integrated screen. The surgery support system according to claim 1, further comprising changing means for changing a display form of the plurality of thumbnail images on the integrated screen. The surgery support system according to claim 2, wherein the changing unit changes the display format to display the enlarged thumbnail images.

Images