WO2023189510A1

WO2023189510A1 - Image processing device, image processing system, image display method, and image processing program

Info

Publication number: WO2023189510A1
Application number: PCT/JP2023/009725
Authority: WO
Inventors: 泰一坂本; 俊祐吉澤; クレモンジャケ; ステフェンチェン; トマエン; 亮介佐賀
Original assignee: テルモ株式会社; 株式会社ロッケン
Priority date: 2022-03-30
Filing date: 2023-03-13
Publication date: 2023-10-05

Abstract

This image processing device: causes a first display to display an operator screen that includes a first region for displaying a surgical assistance image including a biological tissue image which has been imaged on the basis of information obtained by sensing the structure of biological tissue in the body of a surgical subject, and a second region for displaying at least one operation element for controlling the display of the surgical assistance image in the first region; causes a second display, different from the first display, to display a viewer screen for displaying the surgical assistance image so as to match the display of the surgical assistance image in the first region; receives a user operation performed via the operator screen by using an input device; when the user operation is performed in the first region, mirrors said user operation display on the viewer screen; and when the user operation is performed in the second region, does not mirror the user operation display on the viewer screen.

Description

Image processing device, image processing system, image display method, and image processing program

The present disclosure relates to an image processing device, an image processing system, an image display method, and an image processing program.

Patent Document 1 discloses that image information of a subject is displayed on an operator's monitor and a surgeon's monitor, and image information independent of the image information displayed on the operator's monitor is displayed on the operator's monitor. An image diagnostic apparatus that displays images is disclosed.

Japanese Patent Application Publication No. 2007-275506

There are various types of monitor screen sizes, such as HD, NTSC, WXGA, and Cinemascope. "HD" is an abbreviation for high definition. "NTSC" is an abbreviation for National Television System Committee. "WXGA" is an abbreviation for Wide Extended Graphics Array. The HD screen ratio is 16:9 in width and 9 in height. The screen ratio of NTSC is 4:3 in width and 3 in height. The screen ratio of WXGA is 16:10 in width and 10 in height. The screen ratio of CinemaScope is 2.35 in width and 1 in height, that is, 2.35:1.

A screen size of 4:3 is often used in rooms where surgeons perform catheter treatment, such as angiography rooms. On the other hand, in a room where an operator operates software, it is desirable to adopt a screen with a wider screen size such as 16:9, which is suitable for menu operations and detailed settings. However, if the screen size of the operator's monitor is longer than that of the surgeon's monitor, the drawing size will change, so if the operator's screen is displayed on the surgeon's monitor, it will be compressed vertically. A problem arises in that a screen that has been modified is displayed. As a countermeasure to this problem, it is conceivable to display only information important to the surgeon on the monitor on the surgeon's side.

When the operator's screen is displayed on the surgeon's monitor, the surgeon can communicate smoothly with the operator while checking what operations are being performed on the operator's screen. be able to. On the other hand, if only clinically important information is displayed on the surgeon's monitor, the surgeon cannot see what operations are being performed on the operator's screen, and the It may become difficult to communicate smoothly.

The purpose of the present disclosure is to facilitate communication between a surgeon and an operator.

An image processing device according to an aspect of the present disclosure includes a first area that displays a surgical support image including a living tissue image that is imaged based on information obtained by sensing the structure of living tissue in the body of a surgical subject. and a second area displaying at least one operating element for controlling the display of the surgical support image in the first area, and displaying the operator screen in the first area on the first display. An image processing device that displays a viewer screen that displays the surgical support image on a second display that is different from the first display in accordance with the display of the surgical support image, the image processing device comprising: When the user operation is performed in the first area, the display of the user operation is reflected on the viewer screen, and when the user operation is performed in the second area. includes a control unit that does not reflect the display of the user operation on the viewer screen.

In one embodiment, when the user operation is performed in the second area, the control unit displays notification information on the viewer screen indicating that the user operation is performed in the second area. indicate.

In one embodiment, the control unit displays a setting menu including two or more setting items on the operator screen, and selects one of the two or more setting items from the setting menu using the input device. When an operation to select one is received, one or more operation elements corresponding to the selected setting item are displayed in the second area, and when an operation of the one or more operation elements is accepted as the user operation, As the notification information, information indicating the selected setting item is displayed on the viewer screen.

In one embodiment, the control unit displays two or more operation elements in the second area as the at least one operation element, and displays any one of the two or more operation elements as the user operation. When an operation is accepted, information indicating the operating element being operated is displayed on the viewer screen as the notification information.

In one embodiment, the input device is a pointing device, and the control unit displays a first pointer movable by the user operation on the operator screen, and the first pointer is in the first area. In this case, by displaying on the viewer screen a second pointer that moves in accordance with the movement of the first pointer, the display of the user operation is reflected on the viewer screen, and the first pointer moves in accordance with the movement of the second pointer. When the second pointer is in the area, the display of the user operation is stopped from being reflected on the viewer screen by hiding the second pointer on the viewer screen.

In one embodiment, when the control unit determines that the position of the first pointer has not changed for a certain period of time even when the first pointer is in the first area, the control unit displays the second pointer on the viewer screen. Hide.

In one embodiment, the input device is a pointing device, and the control unit displays a first pointer movable by the user operation on the operator screen, and the first pointer is in the first area. In this case, by displaying on the viewer screen a second pointer that moves in accordance with the movement of the first pointer, the display of the user operation is reflected on the viewer screen, and the first pointer moves in accordance with the movement of the second pointer. When the second pointer is in the area, the second pointer is displayed on the viewer screen with its position fixed, thereby stopping the display of the user operation from being reflected on the viewer screen.

In one embodiment, the second area includes a floating area that is movable using the input device and is allowed to overlap the first area, and the control unit is configured to control the floating area on the operator screen. overlaps at least a part of the surgery support image, the at least part of the surgery support image is hidden on the operator screen, and the at least one part of the surgery support image is hidden on the viewer screen. Continue to display the section.

In one embodiment, the control unit separately controls an image displayed in the first area as the surgical support image and an image displayed on the viewer screen based on the same information obtained by the sensing. Render.

An image processing system as one aspect of the present disclosure includes the image processing device, the first display, and the second display.

In one embodiment, the second display is installed in a radiation room where surgery is performed, and the first display is installed in a separate room where operations using the input device are performed.

An image display method as an aspect of the present disclosure includes a first area for displaying a surgical support image including a living tissue image that is imaged based on information obtained by sensing the structure of living tissue in the body of a surgical subject. and a second area displaying at least one operating element for controlling the display of the surgical support image in the first area, and displaying on the first display an operator screen including: An image display method that displays a viewer screen that displays the surgical support image on a second display different from the first display in accordance with the display of the surgical support image, the method comprising: When the user operation is performed in the first area, the display of the user operation is reflected on the viewer screen, and when the user operation is performed in the second area, the display of the user operation is reflected on the viewer screen. This means that the display of the user operation is not reflected on the viewer screen.

In one embodiment, when the user operation is performed in the second area, the image display method displays notification information on the viewer screen indicating that the user operation is performed in the second area. This means that it will be displayed on the screen.

An image processing program according to an aspect of the present disclosure includes a first area for displaying a surgical support image including a living tissue image formed based on information obtained by sensing the structure of living tissue in the body of a surgical subject. and a second area displaying at least one operating element for controlling the display of the surgical support image in the first area, and displaying the operator screen in the first area on the first display. A user operation performed on the operator screen using an input device is sent to a computer that displays a viewer screen that displays the surgical support image on a second display different from the first display in accordance with the display of the surgical support image. and when the user operation is performed in the first area, a display of the user operation is reflected on the viewer screen, and when the user operation is performed in the second area, A process for not reflecting the display of the user operation on the viewer screen is executed.

According to the present disclosure, communication between the surgeon and the operator can be facilitated.

FIG. 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of use of a first display and a second display according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of an operator screen displayed on a first display by the image processing device according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a viewer screen displayed on a second display by the image processing device according to the embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of an operator screen displayed on a first display by the image processing device according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a viewer screen displayed on a second display by the image processing device according to the embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of an operator screen displayed on a first display by the image processing device according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a viewer screen displayed on a second display by the image processing device according to the embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of an operator screen displayed on a first display by the image processing device according to an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of a viewer screen displayed on a second display by the image processing device according to the embodiment of the present disclosure. FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present disclosure. 3 is a flowchart showing the operation of the image processing device according to the embodiment of the present disclosure.

Hereinafter, one embodiment of the present disclosure will be described with reference to the drawings.

In each figure, the same or corresponding parts are given the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

With reference to FIG. 1, the configuration of an image processing system 10 according to the present embodiment will be described.

The image processing system 10 includes an image processing device 20, a first display 30, a second display 40 different from the first display 30, and an input device 50. The image processing device 20 is connected to the first display 30, the second display 40, and the input device 50 via a cable or a network, or wirelessly.

The image processing device 20 is, for example, a general-purpose computer such as a PC, a server computer such as a cloud server, or a dedicated computer. "PC" is an abbreviation for personal computer. The image processing device 20 may be installed in a medical facility such as a hospital, or may be installed in a facility other than the medical facility such as a data center.

The first display 30 and the second display 40 are, for example, LCDs or organic EL displays. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. The screen size of the first display 30 may be any screen size, but in this embodiment it is NTSC. That is, the screen ratio of the first display 30 may be any screen ratio, but in this embodiment, it is 4:3. The screen size of the second display 40 may be any screen size, but in this embodiment it is HD. That is, the screen ratio of the second display 40 may be any screen ratio, but in this embodiment it is 16:9. The input device 50 is, for example, a pointing device such as a mouse, a keyboard, or a touch screen provided integrally with the first display 30. As shown in FIG. 2, the second display 40 is installed in the radiation room 11 of a medical facility. The first display 30 and the input device 50 are installed in a separate room 12 of the medical facility. The radiation room 11 is visible from a separate room 12 through a window 13. In the radiation room 11, a surgery is performed by an operator 14 such as a doctor. In the separate room 12, an operator 15 such as a clinical engineer performs operations using the input device 50. The first display 30 and the second display 40 display surgical support images 16 such as IVUS images, OFDI images, OCT images, CT images, MRI images, echo images, or X-ray images. "IVUS" is an abbreviation for intravascular ultrasound. "OFDI" is an abbreviation for optical frequency domain imaging. "OCT" is an abbreviation for optical coherence tomography. "CT" is an abbreviation for computed tomography. "MRI" is an abbreviation for magnetic resonance imaging.

An overview of this embodiment will be described with reference to FIGS. 3 to 10.

The image processing device 20 displays an operator screen 31 as shown in FIGS. 3, 5, 7, and 9 on the first display 30, and also displays an operator screen 31 as shown in FIGS. A viewer screen 41 like this is displayed on the second display 40. The operator screen 31 includes a first area 32 that displays the surgical support image 16 and a second area 33 that displays at least one operating element for controlling the display of the surgical support image 16 in the first area 32. . The surgical support image 16 includes a living tissue image 17 that is an image formed based on information obtained by sensing the structure of living tissue within the body of the person undergoing surgery. The viewer screen 41 displays the surgical support image 16 in accordance with the display of the surgical support image 16 in the first area 32 . The image processing device 20 receives user operations performed on the operator screen 31 using the input device 50 . When a user operation is performed in the first area 32, the image processing device 20 reflects the display of the user operation on the viewer screen 41. When the user operation is performed in the second area 33, the image processing device 20 does not reflect the display of the user operation on the viewer screen 41.

In this embodiment, the surgical support image 16 displayed in the first area 32 of the screen of the operator 15 side is displayed on the monitor of the operator 14 side, and is displayed in the second area 33 of the screen of the operator 15 side. You can hide the operation elements that are displayed. Here, the surgical support image 16 is important information for the surgeon 14, but the operation elements are information unnecessary for the surgeon 14. That is, according to this embodiment, only clinically important information can be displayed on the monitor on the operator's side.

In this embodiment, when a user operation is performed in the first area 32, the display of the user operation is reflected on the monitor on the operator 14 side, but when the user operation is performed in the second area 33, the display of the user operation is reflected on the monitor on the operator 14 side. In this case, the display of the user's operation is not reflected on the monitor on the surgeon's side. Therefore, the surgeon 14 can confirm at least what kind of operation is being performed in the first area 32 of the screen on the operator 15 side. As a result, the surgeon 14 can communicate smoothly with the operator 15. That is, according to this embodiment, communication between the surgeon 14 and the operator 15 can be facilitated.

When the input device 50 is a pointing device, the image processing device 20 displays a first pointer 51 that can be moved by user operation on the operator screen 31. When the first pointer 51 is in the first area 32, the image processing device 20 displays a second pointer 52 that moves in accordance with the movement of the first pointer 51 on the viewer screen 41, thereby displaying user operations. is reflected on the viewer screen 41. The image processing device 20 hides the second pointer 52 on the viewer screen 41 when the first pointer 51 is in the second area 33, thereby preventing the display of the user's operation from being reflected on the viewer screen 41. Stop. In the example of FIG. 3, the first pointer 51 is located in the second area 33. Therefore, in the example of FIG. 4 corresponding to the example of FIG. 3, the second pointer 52 is hidden on the viewer screen 41. On the other hand, in the example of FIG. 5, the first pointer 51 is located in the first area 32. Therefore, in the example of FIG. 6 corresponding to the example of FIG. 5, the second pointer 52 is displayed on the viewer screen 41 so as to move in accordance with the movement of the first pointer 51.

When the first pointer 51 is in the second area 33, the image processing device 20 displays the second pointer 52 on the viewer screen 41 at a fixed position instead of hiding the second pointer 52. In this way, the display of user operations may be stopped from being reflected on the viewer screen 41. In the example of FIG. 7, the first pointer 51 is located in the second area 33. Therefore, in the example of FIG. 8 corresponding to the example of FIG. 7, the second pointer 52 is displayed on the viewer screen 41 with its position fixed. The second pointer 52 is fixed at a position on the viewer screen 41 corresponding to the position immediately before the first pointer 51 leaves the first area 32 . Therefore, the surgeon 14 can confirm at least in which direction the first pointer 51 has been moved on the screen on the operator 15 side. For example, assume that the surgeon 14 knows that the second area 33 is adjacent to the right side of the first area 32 on the operator screen 31 and that the setting menu 35 is arranged at the bottom of the second area 33. , since the second pointer 52 is located at the lower right corner of the viewer screen 41, it can be inferred that the operator 15 is operating the setting menu 35.

Although any operating elements may be displayed in the second area 33, in this embodiment, a 2D button 34a, a 2D/3D button 34b, an enlargement button 34c, a reduction button 34d, and a rotation button 34e are displayed.

The 2D button 34a is an operation element for selecting a mode in which a two-dimensional image is displayed in the first area 32 as the surgical support image 16. For example, by clicking the 2D button 34a, an IVUS image can be displayed in the first area 32 as the surgical support image 16. An IVUS image is a biological tissue image 17 in which an ultrasonic wave is transmitted from an ultrasonic transducer installed at the tip of a catheter inserted into the body of a patient undergoing surgery, a reflected wave is received, and the signal of the reflected wave is processed. The image includes a two-dimensional image of a cross-sectional structure of a living tissue such as a blood vessel or a heart of a person to be operated on based on information obtained by performing the surgery.

The 2D/3D button 34b is an operation element for selecting a mode in which a two-dimensional image and a three-dimensional image are displayed in the first area 32 as the surgical support image 16. For example, by clicking the 2D/3D button 34b, the latest IVUS image and a three-dimensional image obtained by stacking multiple IVUS images can be displayed in the first area 32 as the surgical support image 16. can. This three-dimensional image is generated as a biological tissue image 17 by sequentially imaging the cross-sectional structure of the patient's biological tissue while moving the ultrasound transducer by a pull-back operation, and generating a plurality of two-dimensional images. Contains a three-dimensional image obtained by stacking two-dimensional images to create a three-dimensional image.

The enlarge button 34c is an operation element for enlarging and displaying the surgical support image 16 in the first region 32. For example, by clicking the enlarge button 34c once, the IVUS image displayed in the first area 32 as the surgical support image 16 can be enlarged and displayed.

The reduction button 34d is an operation element for displaying the surgical support image 16 in a reduced size in the first area 32. For example, by clicking the reduction button 34d once, the IVUS image displayed in the first area 32 as the surgical support image 16 can be displayed in a reduced size.

The rotation button 34e is an operation element for rotating the surgical support image 16 in the first region 32. For example, by clicking the rotation button 34e once, the IVUS image displayed in the first area 32 as the surgical support image 16 can be rotated 90 degrees clockwise.

When the user operation is performed in the second area 33, the image processing device 20 displays notification information 42 indicating that the user operation is performed in the second area 33 on the viewer screen 41. In the example of FIG. 3, when the IVUS image is displayed in the first area 32 as the surgical support image 16, the rotation button 34e is clicked in the second area 33. Therefore, in the example of FIG. 4 corresponding to the example of FIG. 3, an icon indicating that the rotation button 34e has been clicked in the second area 33 is displayed on the viewer screen 41 as the notification information 42. In the example of FIG. 9, when the IVUS image is displayed in the first area 32 as the surgical support image 16, a calibration operation is performed in the floating area 37 included in the second area 33. Therefore, in the example of FIG. 10 corresponding to the example of FIG. 9, a label indicating that a calibration operation is being performed in the second area 33 is highlighted on the viewer screen 41 as the notification information 42.

The image processing device 20 displays a setting menu 35 on the operator screen 31. In this embodiment, the image processing device 20 displays the setting menu 35 only in the second area 33 and does not display it on the viewer screen 41. As a modification of this embodiment, the image processing device 20 may display the setting menu 35 not only in the first area 32 but also on the viewer screen 41. Setting menu 35 includes two or more setting items 36. When the image processing device 20 receives an operation to select any one of two or more setting items 36 from the setting menu 35 using the input device 50, the image processing device 20 selects one or more operation elements according to the selected setting item. is displayed in the second area 33. When the image processing device 20 receives an operation of the one or more operation elements as a user operation, the image processing device 20 displays information indicating the selected setting item on the viewer screen 41 as notification information 42. In the example of FIG. 5, the settings menu 35 is closed. In the example of FIG. 7, when the setting menu 35 is opened and two or more setting items 36 are displayed, the setting item corresponding to the calibration operation is clicked. Therefore, in the example of FIG. 9, a window for performing a calibration operation is displayed in the floating area 37 included in the second area 33 as an operation element corresponding to the clicked setting item. In the example of FIG. 9, a window displayed in the floating area 37 is being manipulated. Therefore, in the example of FIG. 10 corresponding to the example of FIG. 9, a label indicating that the calibration operation has been selected is highlighted on the viewer screen 41 as the notification information 42.

The image processing device 20 displays two or more operation elements in the second area 33. When the image processing device 20 receives an operation of any one of the two or more operation elements as a user operation, the image processing device 20 displays information indicating the operated operation element on the viewer screen 41 as notification information 42. In the example of FIG. 3, when the enlarge button 34c, the reduce button 34d, and the rotate button 34e are displayed in the second area 33, the rotate button 34e is clicked. Therefore, in the example of FIG. 4 corresponding to the example of FIG. 3, an icon indicating that the rotation button 34e has been clicked is displayed on the viewer screen 41 as the notification information 42.

The floating area 37 is an area that can be moved using the input device 50 and is allowed to overlap the first area 32 . When the floating area 37 overlaps at least a portion of the surgical support image 16 on the operator screen 31, the image processing device 20 hides at least a portion of the surgical support image 16 on the operator screen 31, and , continues to display at least a portion of the surgical support image 16 on the viewer screen 41. In the example of FIG. 9, the floating area 37 overlaps at least a portion of the surgical support image 16 on the operator screen 31. Therefore, in the example of FIG. 9, at least a part of the surgical support image 16 is hidden on the operator screen 31, but in the example of FIG. At least a portion of the support image 16 is displayed.

The configuration of the image processing device 20 according to this embodiment will be described with reference to FIG. 11.

The image processing device 20 includes a control section 21, a storage section 22, a communication section 23, an input section 24, and an output section 25.

The control unit 21 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for specific processing. "CPU" is an abbreviation for central processing unit. “GPU” is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. “ASIC” is an abbreviation for application specific integrated circuit. The control unit 21 executes processing related to the operation of the image processing device 20 while controlling each part of the image processing system 10 including the image processing device 20.

The storage unit 22 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, RAM, ROM, or flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, SRAM or DRAM. “SRAM” is an abbreviation for static random access memory. “DRAM” is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. The magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 22 functions as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 22 stores data used for the operation of the image processing device 20 and data obtained by the operation of the image processing device 20.

The communication unit 23 includes at least one communication interface. The communication interface is, for example, an interface compatible with a wireless LAN communication standard such as IEEE802.11, or an interface compatible with a wired LAN communication standard such as Ethernet (registered trademark). "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. The communication unit 23 receives data used for the operation of the image processing device 20 and transmits data obtained by the operation of the image processing device 20.

The input unit 24 includes at least one input interface. The input interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface. The input unit 24 receives an operation for inputting data used for the operation of the image processing device 20. Input section 24 is connected to input device 50 .

The output unit 25 includes at least one output interface. The output interface is, for example, a USB interface, an HDMI (registered trademark) interface, or an interface compatible with a short-range wireless communication standard such as Bluetooth (registered trademark). The output unit 25 outputs data obtained by the operation of the image processing device 20. The output unit 25 is connected to the first display 30 and the second display 40.

The functions of the image processing device 20 are realized by executing the image processing program according to the present embodiment by a processor serving as the control unit 21. That is, the functions of the image processing device 20 are realized by software. The image processing program causes the computer to function as the image processing device 20 by causing the computer to execute the operations of the image processing device 20 . That is, the computer functions as the image processing device 20 by executing the operations of the image processing device 20 according to the image processing program.

The program may be stored on a non-transitory computer-readable medium. The non-transitory computer-readable medium is, for example, a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. Distribution of the program is performed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM that stores the program. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

For example, a computer temporarily stores a program stored on a portable medium or a program transferred from a server in its main storage device. Then, the computer uses a processor to read a program stored in the main memory, and causes the processor to execute processing according to the read program. A computer may read a program directly from a portable medium and execute processing according to the program. The computer may sequentially execute processing according to the received program each time the program is transferred to the computer from the server. Processing may be performed using a so-called ASP type service that implements functions only by issuing execution instructions and obtaining results without transferring programs from the server to the computer. “ASP” is an abbreviation for application service provider. The program includes information similar to a program that is used for processing by an electronic computer. For example, data that is not a direct command to a computer but has the property of regulating computer processing falls under "something similar to a program."

A part or all of the functions of the image processing device 20 may be realized by a programmable circuit or a dedicated circuit as the control unit 21. That is, some or all of the functions of the image processing device 20 may be realized by hardware.

The operation of the image processing device 20 according to this embodiment will be described with reference to FIG. 12. This operation corresponds to the image display method according to this embodiment.

In step S1, the control unit 21 receives a user operation performed on the operator screen 31 using the input device 50. The operator screen 31 is displayed on the first display 30. In the first area 32 of the operator screen 31, a surgical support image 16 including a biological tissue image 17 is displayed. In the second area 33 of the operator screen 31, at least one operating element for controlling the display of the surgical support image 16 in the first area 32 is displayed.

Specifically, the control unit 21 receives user operations via the input unit 24. In the example of FIG. 3, as a user operation, the control unit 21 uses the mouse to move the first pointer 51 to the rotation button 34e among the two or more operation elements displayed in the second area 33, and rotates it. An operation of clicking the button 34e with a mouse is accepted. In the example of FIG. 5, the control unit 21 uses the mouse to move the first pointer 51 on the surgical support image 16 displayed in the first area 32 as a user operation, and moves the first pointer 51 on the surgical support image 16 displayed in the first area 32 to Accepts a mouse click operation to mark a position. In the example of FIG. 7, the control unit 21 displays a setting menu 35 including two or more setting items 36 on the operator screen 31 via the output unit 25. The control unit 21 uses the mouse to move the first pointer 51 displayed in the second area 33 as an operation for selecting any one of two or more setting items 36 from the setting menu 35 using the input device 50. The setting menu 35 is moved to the setting menu 35, and the setting menu 35 is opened, and an operation of clicking a setting item corresponding to the calibration operation among the two or more setting items 36 with the mouse is accepted. The control unit 21 displays a window including a group of GUI parts for performing a calibration operation in the floating area 37 of the second area 33 as one or more operation elements according to the selected setting item. “GUI” is an abbreviation for graphical user interface. In the example of FIG. 9, the control unit 21 uses the mouse to move the first pointer 51 to the floating area 37 of the second area 33 as a user operation, and displays the GUI components in the window displayed in the floating area 37. Accepts calibration operations performed by operating with the mouse. Since the floating region 37 overlaps at least a portion of the surgical support image 16 on the operator screen 31, the control unit 21 hides at least a portion of the surgical support image 16 on the operator screen 31.

When a user operation is performed in the first area 32, the process of step S2 is executed. When a user operation is performed in the second area 33, the processes of step S3 and step S4 are executed.

In step S2, the control unit 21 reflects the display of the user's operation on the viewer screen 41. A viewer screen 41 is displayed on the second display 40. The viewer screen 41 displays the surgical support image 16 in accordance with the display of the surgical support image 16 in the first area 32 .

Specifically, the control unit 21 reflects the display of the user operation on the viewer screen 41 via the output unit 25. In the example shown in FIG. 6 corresponding to the example shown in FIG. to stop the second pointer 52 at a position corresponding to the position clicked in the first area 32 on the viewer screen 41. That is, the control unit 21 reflects the display of the user's operation on the viewer screen 41 by displaying the second pointer 52 on the viewer screen 41 so as to move in accordance with the movement of the first pointer 51.

In step S3, the control unit 21 stops reflecting the display of the user's operation on the viewer screen 41.

Specifically, the control unit 21 stops reflecting the display of the user's operation on the viewer screen 41 via the output unit 25. In the example of FIG. 4 corresponding to the example of FIG. 3, the control unit 21 stops reflecting the display of the user operation on the viewer screen 41 by hiding the second pointer 52 on the viewer screen 41. . In the example shown in FIG. 8, which corresponds to the example shown in FIG. Stop reflection.

In step S4, the control unit 21 displays the notification information 42 on the viewer screen 41.

Specifically, the control unit 21 displays the notification information 42 on the viewer screen 41 via the output unit 25. In the example of FIG. 4 corresponding to the example of FIG. 3, the control unit 21 displays, as notification information 42, an icon indicating that the rotation button 34e has been clicked on the viewer screen 41. That is, the control unit 21 displays information indicating the operating element being operated on the viewer screen 41 as the notification information 42. In the example of FIG. 10 corresponding to the example of FIG. 9, the control unit 21 highlights, as the notification information 42, a label indicating that the calibration operation has been selected on the viewer screen 41. For example, the control unit 21 displays a group of labels corresponding to two or more setting items 36 included in the setting menu 35 at the top of the viewer screen 41, and also displays a group of labels corresponding to two or more setting items 36 included in the setting menu 35. Display the corresponding label in a different color tone than other labels. That is, the control unit 21 displays information indicating the selected setting item on the viewer screen 41 as the notification information 42. Although the floating area 37 overlaps at least a portion of the surgical support image 16 on the operator screen 31, the control unit 21 continues to display at least a portion of the surgical support image 16 on the viewer screen 41.

As a method for drawing the surgical support image 16 on the operator screen 31 and the viewer screen 41, for example, information obtained by sensing is stored in the storage unit 22 as 2D memory information, and based on this 2D memory information, the light source and After setting the camera position, rendering is performed on the operator screen 31, and the two-dimensional image generated as the surgical support image 16 is drawn in the first area 32 of the operator screen 31, and the scale is adjusted. It is conceivable to draw the image on the viewer screen 41 as well. However, in this embodiment, the screen ratios of the first area 32 of the operator screen 31 and the viewer screen 41 are different. Therefore, it is desirable to render the operator screen 31 and viewer screen 41 separately. For example, as a more desirable method, the light source and camera positions are set based on 2D memory information, and then rendering is performed on the operator screen 31 according to the screen ratio, and the 2D image generated as the surgical support image 16 is A dimensional image is drawn in the first area 32 of the operator screen 31, and rendering is separately performed on the viewer screen 41 according to the screen ratio, and another two-dimensional image generated as the surgical support image 16 is displayed to the viewer. It is conceivable to draw on the screen 41. That is, the control unit 21 of the image processing device 20 separates the image displayed in the first area 32 as the surgical support image 16 and the image displayed on the viewer screen 41 based on the same information obtained by sensing. You can render it to . Each rendering is repeated at regular intervals, so if the operator 15 changes the position of the light source or camera, the change is automatically reflected on both the operator screen 31 and the viewer screen 41.

The case where a three-dimensional image is displayed as the surgical support image 16 is the same as the case where a two-dimensional image is displayed. That is, as a method for drawing the surgical support image 16 on the operator screen 31 and the viewer screen 41, for example, 3D space information is stored in the storage unit 22, and based on this 3D space information, the positions of the light source and camera are After setting, volume rendering is performed on the operator screen 31, and the three-dimensional image generated as the surgical support image 16 is drawn in the first area 32 of the operator screen 31, and the scale is adjusted and viewed. It is conceivable to draw the image on the user screen 41 as well. However, in this embodiment, the screen ratios of the first area 32 of the operator screen 31 and the viewer screen 41 are different. Therefore, it is preferable to use a method that performs volume rendering twice rather than a method that performs volume rendering only once. For example, a more desirable method is to set the light source and camera positions based on 3D space information, perform volume rendering on the operator screen 31 according to the screen ratio, and generate the surgical support image 16. At the same time, a three-dimensional image is drawn in the first area 32 of the operator screen 31, and volume rendering is separately performed on the viewer screen 41 according to the screen ratio, and another three-dimensional image is generated as the surgical support image 16. It is conceivable to draw this on the viewer screen 41. Each volume rendering is repeated at regular intervals, so if the operator 15 changes the position of the light source or camera, the change is automatically reflected on both the operator screen 31 and the viewer screen 41.

As described above, in this embodiment, information unnecessary for the surgeon 14, such as operation elements for controlling the display of the surgical support image 16, is displayed only on the operator screen 31, and such information is displayed on the viewer screen 41. This allows you to omit unnecessary information. Therefore, the visibility of the viewer screen 41 is improved.

In this embodiment, instead of displaying a copy of the operator screen 31 on the viewer screen 41, clinically necessary information displayed on the operator screen 31 can be displayed on the viewer screen 41. Therefore, even if the floating area 37 overlaps at least a portion of the surgical support image 16 on the operator screen 31, only the operator screen 31 is covered with at least a portion of the floating area 37, and the surgical support image 16 is hidden on the viewer screen 41. You can continue to display the entire image.

In this embodiment, when the mouse cursor position on the operator screen 31 is within the first area 32 displaying clinical information, the position is specified and the mouse cursor is moved to the corresponding position on the viewer screen 41. can be displayed. Therefore, the operation status of the operator 15 can be communicated to the operator 14.

In this embodiment, communication between the operator 14 in the radiation room 11 and the operator 15 in the separate room 12 is carried out via the notification information 42 while minimizing the number of people exposed to radiation in the radiation room 11. It can be made smoother.

As a modified example of the present embodiment, when the control unit 21 of the image processing device 20 determines that the position of the first pointer 51 has not changed for a certain period of time even when the first pointer 51 is in the first area 32, The second pointer 52 may be hidden on the viewer screen 41. For example, if the first pointer 51 does not move in the first area 32 for three seconds, the control unit 21 may hide the second pointer 52 on the viewer screen 41. According to this modification, the second pointer 52 can be prevented from interfering with the surgeon 14 checking the structure of the living tissue. As a further modification, when the control unit 21 determines that the position of the first pointer 51 has not changed for a certain period of time, it not only hides the second pointer 52 on the viewer screen 41 but also hides the second pointer 52 on the operator screen 31. The first pointer 51 may be hidden.

The present disclosure is not limited to the embodiments described above. For example, two or more blocks depicted in the block diagram may be combined, or one block may be divided. Instead of performing two or more steps described in a flowchart chronologically as described, they may also be performed in parallel or in a different order, depending on the processing power of the device performing each step or as necessary. good. Other changes are possible without departing from the spirit of the present disclosure.

10 Image processing system 11 Radiology room 12 Separate room 13 Window 14 Surgeon 15 Operator 16 Surgery support image 17 Biological tissue image 20 Image processing device 21 Control unit 22 Storage unit 23 Communication unit 24 Input unit 25 Output unit 30 First display 31 Operation User screen 32 First area 33

Second area

34a

2D button 34b 2D/3D button

34c Enlarge button

34d Reduce button 34e Rotate button 35 Setting menu 36 Setting item 37 Floating area 40 Second display 41 Viewer screen 42 Notification information 50 Input device 51 1st pointer 52 2nd pointer

Claims

a first area for displaying a surgical support image including a biological tissue image imaged based on information obtained by sensing the structure of a biological tissue in the body of a surgical subject; an operator screen that includes a second area that displays at least one operating element for controlling the display on the first display, and displays the surgical support image in accordance with the display of the surgical support image in the first area. An image processing device that displays a viewer screen that displays on a second display different from the first display,
A user operation performed on the operator screen is accepted using an input device, and when the user operation is performed in the first area, the display of the user operation is reflected on the viewer screen, and the user operation is The image processing apparatus includes a control unit that does not reflect the display of the user operation on the viewer screen when the user operation is performed in the second area.
Claim 1: When the user operation is performed in the second area, the control unit displays notification information indicating that the user operation is performed in the second area on the viewer screen. The image processing device described in .
The control unit displays a setting menu including two or more setting items on the operator screen, and selects any one of the two or more setting items from the setting menu using the input device. When the operation of the one or more operation elements is accepted as the user operation, one or more operation elements corresponding to the selected setting item are displayed in the second area, and when the operation of the one or more operation elements is accepted as the user operation, the notification information includes: The image processing apparatus according to claim 2, wherein information indicating the selected setting item is displayed on the viewer screen.
When the control unit displays two or more operation elements in the second area as the at least one operation element, and receives an operation of any one of the two or more operation elements as the user operation, The image processing apparatus according to claim 2, wherein information indicating an operating element being operated is displayed on the viewer screen as the notification information.
The input device is a pointing device,
The control unit displays a first pointer movable by the user operation on the operator screen, and when the first pointer is in the first area, the control unit displays a first pointer that moves in accordance with the movement of the first pointer. By displaying two pointers on the viewer screen, the display of the user operation is reflected on the viewer screen, and when the first pointer is in the second area, the second pointer is displayed on the viewer screen. The image processing apparatus according to any one of claims 1 to 4, wherein display of the user operation is stopped from being reflected on the viewer screen by hiding the image.
The control unit hides the second pointer on the viewer screen when determining that the position of the first pointer has not changed for a certain period of time even when the first pointer is in the first area. The image processing device according to claim 5.
The input device is a pointing device,
The control unit displays a first pointer movable by the user operation on the operator screen, and when the first pointer is in the first area, the control unit displays a first pointer that moves in accordance with the movement of the first pointer. By displaying two pointers on the viewer screen, the display of the user operation is reflected on the viewer screen, and when the first pointer is in the second area, the second pointer is fixed in position. The image processing device according to any one of claims 1 to 4, wherein the display of the user operation is stopped from being reflected on the viewer screen by displaying the user operation on the viewer screen.
The control unit hides the second pointer on the viewer screen when determining that the position of the first pointer has not changed for a certain period of time even when the first pointer is in the first area. The image processing device according to claim 7.
The second area includes a floating area that is movable using the input device and is allowed to overlap the first area,
When the floating area overlaps at least a portion of the surgical support image on the operator screen, the control unit hides at least a portion of the surgical support image on the operator screen, and The image processing device according to any one of claims 1 to 8, wherein at least a part of the surgical support image continues to be displayed on the viewer screen.
1 . The control unit separately renders an image displayed in the first area as the surgical support image and an image displayed on the viewer screen based on the same information obtained by the sensing. The image processing device according to claim 9.
An image processing device according to any one of claims 1 to 10,
the first display;
An image processing system comprising: the second display.
The second display is installed in a radiation room where surgery is performed,
The image processing system according to claim 11, wherein the first display is installed in a separate room where operations using the input device are performed.
a first area for displaying a surgical support image including a biological tissue image imaged based on information obtained by sensing the structure of a biological tissue in the body of a surgical subject; an operator screen that includes a second area that displays at least one operating element for controlling the display on the first display; and a second area that displays at least one operation element for controlling the display; An image display method for displaying a viewer screen on a second display different from the first display, the method comprising:
accepting user operations performed on the operator screen using an input device;
When the user operation is performed in the first area, the display of the user operation is reflected on the viewer screen, and when the user operation is performed in the second area, the display of the user operation is reflected on the viewer screen. An image display method that does not reflect the display on the viewer screen.
14. The image display according to claim 13, wherein when the user operation is performed in the second area, notification information indicating that the user operation is performed in the second area is displayed on the viewer screen. Method.
a first area for displaying a surgical support image including a biological tissue image imaged based on information obtained by sensing the structure of a biological tissue in the body of a surgical subject; an operator screen that includes a second area that displays at least one operating element for controlling the display on the first display, and displays the surgical support image in accordance with the display of the surgical support image in the first area. a computer that displays a viewer screen that displays on a second display different from the first display;
a process of accepting a user operation performed on the operator screen using an input device;
When the user operation is performed in the first area, the display of the user operation is reflected on the viewer screen, and when the user operation is performed in the second area, the display of the user operation is reflected on the viewer screen. An image processing program that executes processing that does not reflect a display on the viewer screen.