JP2023071677A - Medical image processing device and medical image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a medical image processing device and the like which can display a three-dimensional medical image and perform various display processing while having the excellent workability and keeping cleanliness of an operator.

SOLUTION: An image processing device (301) is equipped with: a display (361); a control unit (350) which is connected to the display; a speech input device (370); and a motion sensor (380). The control unit (350) has: an image display part (355a) for displaying a three-dimensional medical image on the display; a mode selection part (355d) for switching, upon recognizing speech input through the speech input device (370), the mode of display related to the three-dimensional medical image in response to the speech; and a display processing part (355c) for changing, upon recognizing via the motion sensor a motion input by an operator, the display of the three-dimensional medical image in response to the motion.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a medical image processing apparatus and a medical image processing program. More particularly, the present invention relates to a medical image processing apparatus capable of displaying a three-dimensional medical image obtained by imaging a patient and performing various display processing with good workability and keeping the operator clean. .

Currently, known medical image diagnostic apparatuses include a CT (Computed Tomography) apparatus, an MRI (Magnetic Resonance Imaging) apparatus, a PET (Positron Emission Tomography) apparatus, an ultrasonic diagnostic apparatus, an angiography imaging apparatus, and the like. there is

In recent years, for example, a simulation may be performed before performing an operation on an organ such as the liver in which blood vessels are intricately intertwined. The simulation is performed, for example, by performing a contrast-enhanced CT examination, preparing a fluoroscopic captured image of a region to be treated, and confirming it on a display. Such simulations are useful for examining treatment plans. For example, Patent Literature 1 discloses a technique of displaying an organ such as a liver on a tablet terminal and simulating a surgical operation.

JP 2014-54358 A

The technology disclosed in Patent Document 1 is useful in that it can be carried anywhere to carry out preoperative simulations because it uses a tablet terminal. Moreover, since it can be brought into the operating room, it is useful in that some kind of confirmation can be made during the operation.

However, if it is inconvenient for the operator who uses the terminal, there is a possibility that the provided function will not be used. In addition, it is desirable that the operator can operate while keeping cleanliness. This is because a medical image processing apparatus may be operated by a doctor or the like during an operation.

The present invention has been made by paying attention to these problems. The purpose is to display a three-dimensional medical image obtained by imaging a patient, and to perform various display processing with good workability and keeping the operator clean, such as a medical image processing apparatus and an image processing program. is to provide

A medical image processing apparatus according to one aspect of the present invention for solving the above problems is as follows:
a display;
a control unit (processor) connected to the display;
an audio input device;
a motion sensor;
A medical image processing apparatus comprising
The control unit (processor)
a: an image display unit for displaying a three-dimensional medical image on the display;
b: a mode selection unit that recognizes voice input using the voice recognition device and switches a mode related to display of the three-dimensional medical image accordingly;
c: a display processing unit that recognizes an operator's motion input via the motion sensor and changes the display of the three-dimensional medical image accordingly;
A medical image processing apparatus having

In other words, in the device of one embodiment of the present invention, the control unit (processor) is
- causing the display to display a three-dimensional medical image;
- recognizing input speech using the speech recognition device, and switching the mode for displaying the three-dimensional medical image accordingly;
- configured to recognize operator motion input via said motion sensor and modify the display of said three-dimensional medical image accordingly.

(Explanation of terms)
- "Anatomical structure" refers to objects (eg, organs, bones, blood vessels, etc.) that can be recognized within the subject, and includes fat, lesions such as tumors, and the like.
- "Terminal" refers to an information processing device that is connected to a network or used stand-alone and performs data processing. Any peripheral device may be connected to the information processing device. Basically, devices such as tablet terminals and laptop computers in which various functions are provided in one device are preferable, but depending on the case, some functions may be It can also be configured by distributing functionally or physically in units.
・"Connection" refers to the case where two elements are directly connected, as well as the case where an element and another element are indirectly connected via some intermediate element within the scope of the present invention. Including cases where It also includes both wired and wireless connections.

In this specification, a component represented by "function"+"unit" corresponds to a functional block that performs a predetermined function. Functional blocks do not necessarily imply a division between hardware circuits. Thus, for example, one or more functional blocks can be implemented in a single piece of hardware, but can also be implemented in multiple pieces of hardware.

According to the present invention, there is provided a medical image processing apparatus, etc., capable of displaying a three-dimensional medical image obtained by imaging a patient, and performing various display processing with good workability and keeping the operator clean. can do.

1 illustrates a medical image processing apparatus according to one embodiment; FIG. 2 is a diagram showing an example of a block diagram of the image processing apparatus of FIG. 1; FIG. It is a figure which shows an example of the component of a hospital system. 2 is a flowchart of an operation example of the image processing apparatus of FIG. 1; FIG. 4 is a diagram showing an example of a three-dimensional medical image displaying the liver and blood vessels around it; It is an example of a graphical image when recognizing the position of an operator's hand. It is a figure which shows an example of a three-dimensional medical image. FIG. 10 is a diagram for explaining how two points are specified with respect to an arbitrary target on a three-dimensional medical image; FIG. 10 is a diagram for explaining how the position of a specified point is finely adjusted; It is a figure which shows the example which divides|segments a liver. It is a figure for demonstrating a mode that a part of liver which divided|segmented is touched. It is a figure which shows the state which did not display only a part of divided liver. It is a flowchart which shows the flow of an operating procedure as an example. FIG. 10 is a diagram for explaining a procedure for specifying an area; FIG. FIG. 10 is a diagram for explaining a procedure (an example) of specifying an area; 4 is a three-dimensional medical image showing an example of displaying blood vessels with different thresholds; FIG. 10 is a diagram showing an example of a screen displaying a stereo image; FIG. It is a figure which shows the example of the screen regarding anonymization of patient information. It is an operation flow for displaying patient information in a predetermined case. FIG. 10 is a table showing that each part of a fluoroscopic image of a patient is separately and independently managed in advance in a data server; FIG. It is a figure which shows typically an example of arrangement|positioning of a foot switch. It is an example of a screen displayed when voice input is ON.

Embodiments of the present invention will be described below with reference to the drawings.
[Section A: A medical image processing apparatus capable of performing various display processes while maintaining good workability and keeping the operator clean]
1. Configuration The medical image processing apparatus 301 of this embodiment is, for example, a portable computer device such as a tablet terminal. Alternatively, it may be a laptop PC (notebook PC) having a touch panel display. FIG. 1 shows an example of a tablet terminal, which may be configured by installing an image processing program according to one embodiment of the present invention in a commercially available tablet terminal. In the following description, the medical image processing apparatus is simply referred to as an image processing apparatus. Regarding tablet terminals and notebook PCs, although not particularly limited, it is preferable in one embodiment that the screen size is 9 inches or more, or 10 inches or more. In one embodiment, the thickness is preferably 20 mm or less or 15 mm or less. In one aspect, the mass is preferably within 2 kg or within 1.5 kg.

As shown in FIG. 1, in this example, the image processing apparatus 301 has a thin housing 301a, and a touch panel display 360 is provided on one surface of the housing 301a. The touch panel display 360 is composed of a display 361 (see FIG. 2) and a touch panel 363 (see FIG. 2).

Such an image processing device 301 can be connected to a hospital system network, for example, as shown in FIG. The hospital system of this example includes the following devices connected to a network 30: an imaging device 1, a liquid injection device 10, a hospital information system HIS (Hospital Information System) 21, and a radiology department information system. RIS (Radiology Information System) 22, PACS (Picture Archiving and Communication Systems) 23 which is an image storage communication system, workstation 24 and printer 25, and the like. Note that not all of these are essential components, and some of them can be omitted. Each of the above elements may be singular or plural. The connection to the network may of course be a wired connection or a wireless connection.

Examples of the imaging device 1 include imaging devices such as a CT device, an MR device, and an angiography device. Other types of imaging devices may be used, or multiple imaging devices of the same type or different types may be used. A three-dimensional medical image, which will be described later, may be created using a plurality of modality images, such as combining an image captured by a CT device and an image captured by an MR device.

The drug solution injection device 10 may be a contrast medium injection device that injects at least a contrast medium. A control circuit for controlling may also be provided. As an example, a contrast injection device with an injection head and console may be utilized. The drive mechanism may be a piston drive mechanism, a roller pump, or the like.

Please refer to the block diagram of FIG. The image processing apparatus 301 includes a display 361, a touch panel 363, an input device 365, a communication section 367, an interface 368, a slot 369, a control section 350, a storage section 359, and the like. Note that not all of these are essential, and some may be omitted.

Devices such as a liquid crystal panel and an organic EL panel can be used as the display 361 . A touch panel type display in which the touch panel 363 is integrally provided can also be used. As the touch panel, touch panels of resistive film, capacitance, electromagnetic induction, surface acoustic wave, infrared rays, and the like can be used. As a specific example, a multi-touch that is a touch at a plurality of positions, such as a capacitive method, may be detected. A touch operation can be performed using a user's finger, a touch pen, or the like. The touch panel may detect the start of a touch operation on the touch panel, the movement of the touch position, the end of the touch operation, and the like, and output the detected touch type and coordinate information.

As will be described later, the image processing apparatus 301 of this embodiment can perform operations related to image display using voice input and motion input. Therefore, the touch panel 363 may be omitted in some cases.

Examples of the input device 365 include general devices such as a keyboard and a mouse.

The storage unit 359 may be configured by a hard disk drive (HDD), a solid state drive (SDD), and/or memory, etc., and may store OS (Operating System) programs and , a medical image processing program (including algorithm data, graphical user interface data, etc.) according to one embodiment of the present invention may be stored.

It also stores other programs, tables, databases, etc. used for various processes as needed. The computer program is executed by being loaded into the memory of the control unit, cooperates with hardware such as a CPU, and thereby constitutes a control unit having the functions of the present embodiment. .

The computer program may be downloaded in whole or in part from an external device via any network when necessary. The computer program may be stored in a computer-readable recording medium, and the "recording medium" includes a memory card, USB memory, SD card (registered trademark), flexible disk, magneto-optical disk, ROM, EPROM, EEPROM, It shall include any “portable physical media” such as CD-ROM, MO, DVD, and Blu-ray (registered trademark) Disc. The medical image processing apparatus of this embodiment may be provided with a slot 369 for reading the above storage medium.

A communication unit 367 is a unit for enabling communication with an external network or device by a wired system or a wireless system. The communication unit 367 may have a transmitter for transmitting data to the outside and a receiver for receiving data from the outside. The interface 368 is for connecting various external devices and the like, and although only one interface is shown in the drawing, a plurality of interfaces may be provided as a matter of course.

Slot 369 is for reading data from a computer readable medium. An interface 368 is a portion for connecting external devices and the like.

The image processing apparatus 301 of this embodiment includes a microphone 370 as an audio input device. The microphone may be built in the housing, or may be an external microphone that is separate from the housing and connected to the terminal by wire or wirelessly. It is also possible to use a device that integrates a motion sensor 380 and a microphone described below.

In order to perform speech recognition, speech recognition software is installed in the image processing device 301, and a speech recognition unit 351 is configured by this.

(motion sensor)
The motion sensor 380 is a sensor that three-dimensionally detects the motion of at least a part of the operator's body in a non-contact manner. Motion recognition software is installed in the image processing device 301, and a motion recognition unit 353 is configured by this.

As the motion sensor 380, for example, a Leap Motion controller (manufactured by Leap Motion, "Leap Motion" is a registered trademark) or the like can be used. This leap motion controller is an input device that can recognize the position, shape and movement of the operator's fingers and/or the position and movement of the palm in real time without contact. The leap motion controller is configured as a sensor unit containing an infrared irradiation section, a CCD camera, and the like. The area above the sensor unit is the recognition area. This sensor unit is used by connecting to a tablet terminal or a laptop PC by wire or wirelessly.

As the motion sensor 380, for example, Kinect (manufactured by Microsoft Corporation, registered trademark) can also be used. Motion sensor 380 may comprise one or more cameras and one or more range sensors. Alternatively, only one of them may be provided. The motion sensor 380 unit may have a built-in microphone. As for the accuracy of the detection target (for example, hand) of the motion sensor 380, for example, it is preferable that the detection is possible with an accuracy of 5 mm or less, and it is more preferable that the detection is possible with an accuracy of 1 mm or less.

The detection principle of the motion sensor 380 is also not limited to a specific one. As one aspect, a system called Light Coding can be used. In this method, a large number of dot patterns are emitted from an infrared light emitting unit, and a camera reads the amount of change (distortion) when the dot pattern hits a detection target (person). As another aspect, a method called TOF (Time Of Flight) can be used. This has a relatively short recognition range and is suitable for usage such as sensing fine finger movements. In the TOF method, the distance is measured by analyzing the time it takes for the irradiated infrared rays to hit the object and return. In general, the recognition accuracy is higher than that of the light coding method, and the decrease in accuracy due to distance is small. As still another aspect, a method of recognizing movement by photographing the reflection of light emitted from an infrared LED onto an object with two cameras may be used.

(control part)
Refer to FIG. 2 again. The control unit 350 has hardware such as a central processing unit (CPU) and memory, is installed with a computer program, and performs various arithmetic processes. The control unit 350 has, as a conceptual configuration, an image display unit 355a, an operation determination unit 355b, a display processing unit 355c, and a mode selection unit 355d. It also has the voice recognition section 351 and the motion recognition section 353 as described above.

The image display unit 355a causes the display 361 to display a three-dimensional medical image for medical use. For example, the image display unit 355a displays each of the anatomical structures such as the liver and blood vessels as independent objects. Alternatively, each anatomical structure may be displayed in a different color. The color to display each anatomical structure may be manually input and set by the operator, but is not necessarily limited to this. As will be described later, when colors are assigned in advance on the side of a predetermined data server (using a table or the like), display may be performed according to the assigned colors.

The operation determination unit 355b receives input operations and the like on the input device 365 and the touch panel 363 .

The display processing unit 355c performs various image processing. for example,
- rotation of the three-dimensionally displayed image,
- translation of the three-dimensionally displayed image,
- enlargement/reduction of images displayed in 3D,
- Change the transparency of the image displayed in 3D,
- Show/hide certain objects,
- Ability to cut (split) a given object,
- Ability to designate regions of a given object, etc.
is. The specific contents of these functions will be described in detail in a series of operations to be described later.

The voice recognition unit 351 performs various voice recognitions. For example, the following words are recognized.
- "Rotate" command for changing the display mode
- "move" command for changing display mode
- "Multi", the command for changing the display mode
- "Stop" which is a command for changing the display mode
- "Cut" which is a command for processing
- "Box" which is a command for processing
- Names of anatomical structures (eg names of organs such as "liver" or names of blood vessels such as "portal vein", "hepatic artery").

The motion recognition unit 353 performs various motion recognition processes. For example, assuming that the motion sensor detects a hand, it detects the position and movement of the hand (fingers) within the detection space.

It should be noted that, with respect to inventions having image processing or other data processing as a main feature, the hardware configuration is not limited to the specific ones disclosed in the above-described embodiments, and various aspects can be used. Therefore, it should be noted that, for example, not only a tablet terminal or a notebook PC, but also processing performed by other computer means can be a target of one aspect of the present invention. In addition, it should be understood that the inventions disclosed below mainly as explanations of "operations" can also be grasped by those skilled in the art as inventions of products or inventions of computer programs with different categorical expressions. . Accordingly, this specification also discloses such an invention.

2. Operation Next, an example of image display operation in the image processing apparatus 301 of this embodiment will be described. An example of displaying and operating a three-dimensional medical image as illustrated in FIG. 5 will be described below. A liver 371 and blood vessels 375 are included in this three-dimensional medical image.

First, as shown in the flowchart of FIG. 4, three-dimensional medical image data is acquired in step S11. The "three-dimensional medical image data" may be created based on data obtained by tomography of a patient using an imaging device. In particular, it may be volume data obtained by volume rendering. The data format of the three-dimensional image is not particularly limited, and various formats can be used. For example, an STL (Standard Triangulated Language) file format can be used.

The image data may be stored in a predetermined data storage area such as a predetermined database server, PACS, DICOM server or workstation. As an example, the image processing apparatus 301 reads the data from a predetermined data storage area on the network and saves it in the storage unit 359 within the apparatus.

Next, the image processing device 301 causes the display 361 to display the three-dimensional medical image (step S12). A three-dimensional medical image can be basically created using a known method. An image creation flow according to one embodiment of the present invention will be described later with reference to the drawings. The created three-dimensional medical image data may be stored in the image processing apparatus 301 and/or may be stored in an external server (for example, a cloud server).

Here, various display modes are prepared for displaying medical images. for example,
- displaying certain anatomical structures in a translucent state;
- display certain anatomical structures in an opaque state;
- color-coding certain anatomical structures,
- displaying certain anatomical structures with shading;
at least one of: displaying an image of three-dimensional coordinate axes (or its equivalent, such as a cube) on the screen.

Regarding transparency, for example, the liver is displayed in a translucent state and the blood vessels are displayed in an opaque state. If there is a tumor, the tumor may also be displayed in an opaque state. According to such a display mode, by translucently displaying the liver, it is possible to confirm the positions and running states of internal blood vessels that are hidden behind the liver and cannot be visually recognized. Such confirmation is very useful, for example, in laparoscopic surgery in which a part of the liver is excised, in that the positional relationship of the liver, blood vessels, tumors, etc. can be well confirmed. The image processing apparatus 301 of this embodiment is portable, and therefore can be operated in the operating room to confirm three-dimensional medical images.

Regarding color coding, for example, the liver and blood vessels may be displayed in different colors. If there is a tumor, the tumor may be displayed in yet another color. More specifically with regard to blood vessels, the liver, portal vein, and hepatic artery may be displayed in different colors. When blood vessels are grouped, they may be collectively displayed in the same color.

Next, in step S13, the image processing device 301 detects the hand position so that the distance between the motion sensor 380 and the operator's hand is appropriate. Specifically, the operator positions his hand above the motion sensor 380 . An appropriate distance between the sensor and the operator's hand (the height from the sensor to the hand) is set in advance to a range of h1 (mm) to h2 (mm), for example. The reason why the appropriate range is set in advance is that if the operator's hand is too close or too far from the sensor, the operator's hand movement may not be properly recognized.

Regarding the detection of the hand position, there is no particular limitation on what kind of image is displayed on the screen, but it may be as shown in FIG. 6 as an example. In this example, a reference circle (first circle) 391 of a predetermined size is displayed approximately in the center of the screen. The first circle 391 is always fixedly displayed with a constant size regardless of the position of the operator's hand. On the other hand, a second circle 393 is also displayed on the screen.

The center of the second circle 393 corresponds to the position of the operator's hand. That is, when the operator's hand is directly above the motion sensor 380 (for example), the center of the second circle 393 is the same as the center of the circle 391 at the second position. That is, two circles 391 and 393 are displayed as concentric circles.

When the operator's hand is displaced in a predetermined direction (to the right as an example) from the position directly above the motion sensor 380, the second circle 393 is correspondingly displaced in the same direction (to the right as an example) and displayed in real time. be done.

With such a display mode, the operator can check whether his/her hand is in the proper position (horizontal position) with respect to the motion sensor 380 while looking at the positional relationship between the two circles 391 and 393 on the screen. can do.

The position in the height direction can be confirmed as follows. That is, the diameter of the second circle 393 corresponds to the height of the operator's hand. For example, if the height of the hand is the reference height ((h1+h2)/2 in one example), the second circle 393 is adjusted to have the same diameter as the first circle 391. Is displayed. The higher the hand position, the smaller the size of the second circle 393 correspondingly, and conversely, the lower the hand position, the larger the size of the second circle 393 correspondingly. With such a display mode, the operator can check whether the height of his or her hand is appropriate while looking at the size relationship between the two circles 391 and 393 on the screen.

In order to make it easier to confirm the correct position, the following display may be used. That is, the second circle 393 may be displayed in a special display when the horizontal position of the hand, the vertical position, or a combination thereof, enters a predetermined proper position. For example, the second circle 393 may be displayed in different colors depending on whether it is out of the proper range as shown in FIG. 6(a) or within the proper range as shown in FIG. 6(b), In one embodiment, it is preferable to switch between blinking display and lighting display.

As described above, the steps for adjusting the distance between the motion sensor and the operator's hand are completed (step S13).

Next, in step S14, voice input for selecting the display mode is accepted. An example of speech input may be recognition of words such as:
- "move"
-"Stop"
-"rotate"
- "Multi"

The voice recognition function may be turned on triggered by the position of the operator's hand entering a predetermined proper range in step S3. In this configuration, the voice recognition function is turned off when the position of the operator's hand is out of the predetermined proper range, and is turned on only when it is within the proper range. In this way, according to the configuration in which the voice recognition function is turned ON only under a predetermined condition, it is possible to prevent voice input due to erroneous recognition that is not intended by the operator.

As schematically shown in FIG. 1, in order to inform the operator that the voice recognition function is ON, a display such as "Voice recognition in progress" may be displayed on the screen. preferable.

Note that in one aspect of the present invention, the hand position detection step of S13 may be omitted.

<Zoom (enlargement and reduction)/Pan>
To change the size or position of the displayed three-dimensional medical image, follow the procedure below.

First, the operator utters "move" while the voice recognition function is ON. The image processing device 301 analyzes the voice input from the microphone 370 by the voice recognition unit 351 and recognizes the word "move". Accordingly, a transition is made to the "zoom/pan" mode (step S15).

In the "zoom/pan" mode, the image processor 301 then waits for motion input by the operator's hands. The image processing device 301 uses the motion sensor 380 and the motion recognition unit 353 to recognize the position and motion of the operator's hand in real time. Then, when the operator moves the hand upward (that is, when the hand moves from the initial height _h0 to a higher height _hh ), the image gradually shrinks according to the movement. On the other hand, when the operator moves the hand downward (moves from the initial height _h0 to a lower _hL ), the medical image is gradually enlarged according to the movement.

When the hand is horizontally moved, the three-dimensional medical image is panned (translated) in accordance with the direction and amount of movement.

As described above, in this mode, moving the hand up and down reduces or enlarges the image, and moving the hand horizontally translates the image. According to the configuration of the present embodiment, compared to input methods such as voice recognition and numeric input, motion input that enables intuitive analog input is used to zoom/pan an image (further described below). rotation, etc.) can be implemented. Therefore, it is intuitive for the operator, has excellent operability, and contributes to practical use.

Further, since the motion sensor 380 can perform input without touching the device, the operator can perform input while keeping the operator's hands clean. Such a configuration is extremely advantageous in that, for example, a doctor performing an operation can use the image processing device in the operating room to check the image.

In particular, taking the liver as an example, the liver has a plurality of branched blood vessels. Therefore, in order not to damage the blood vessels more than necessary, it is necessary to fully confirm the positional relationship between the blood vessels and the tumor when part of the liver parenchyma is excised. In this respect, according to the image processing apparatus of the present embodiment, it is possible to confirm the positional relationship of blood vessels and the like while viewing a three-dimensional medical image during surgery. Also, in a three-dimensional medical image, for example, there may be a blood vessel on the near side and a tumor hidden behind it (the tumor is not shown, but see FIG. 5 for reference). Even in such a case, the image processing apparatus of this embodiment can rotate the image in the "rotation" mode to confirm the tumor. Above all, in the configuration of the present embodiment, it is possible to freely (steplessly) rotate by an arbitrary angle by motion input, instead of rotating by a predetermined angle, so that good observation can be performed. becomes possible.

In one embodiment, it is preferable that the three-dimensional medical image is not "rotated" (detailed below) during the "zoom/pan" mode. When using this mode, it is often the case that you only want to perform scaling or translation. Therefore, it is easier for the operator to use when the rotation is prohibited and the enlargement, reduction, and translation are performed while maintaining the desired viewing posture.

Although the above description describes a mode in which both zooming and panning can be performed, the present invention is not limited to this. A mode in which only one of them can be performed is also possible.

<Rotation>
If you want to rotate the displayed 3D medical image, follow the steps below:
First, the operator utters "stop" to cancel the above "zoom/pan" mode. The image processing apparatus 301 accepts this with the voice recognition function, cancels the "zoom/pan" mode, and transitions to a state of accepting another mode.

In this state, the operator utters "rotation". The image processing device 301 accepts this with the voice recognition function, and transitions to the "rotation" mode. The image processing device 301 then waits for motion input by the operator's hand.

In the "rotation" mode, the image processing device 301 recognizes the position and movement of the operator's hand in real time. Then, the image processing apparatus 301 rotates the three-dimensional medical image around predetermined rotation axes (X-axis, Y-axis, Z-axis) in accordance with the motion of the operator's hand. Specifically, horizontal movement of the operator's hand or movement of the hand along the surface of the virtual sphere is recognized. Then, the three-dimensional medical image is rotated by a predetermined angle in accordance with the moving direction, moving speed, and moving amount.

Also in this rotation mode, it is preferable in one aspect that only rotation is permitted, and panning (translation) and zooming (enlargement/reduction) are prohibited. As a result, for example, it is possible to rotate the image in a desired direction while maintaining the predetermined image size, and then perform predetermined image processing and observation.

Also when canceling the "rotation mode", the operator utters "stop" as described above.

In the above description, when one of "move" and "rotate" is performed, the other is not performed. may be provided. In this mode, "zoom", "pan" and "rotation" are all performed according to the operator's hand movements.

Specifically, when the operator utters "multi", the image processing apparatus 301 recognizes this and transitions to the "multi" mode. The three-dimensional medical image is rotated, moved, and enlarged/reduced according to the motion input of the operator's hand.

It should be noted that a function that can rotate a three-dimensional medical image only by voice input instead of motion input may be implemented. For example, the image processing device 301 recognizes "rotation", "left", and "15 degrees" by uttering them. Then, the image is rotated by 15° around a predetermined rotation axis (for example, the Z axis extending in the vertical direction of the screen). If you want to rotate upward by 15° about a laterally extending axis, you can say, for example, “rotate”, “up”, “15°”.

<Other audio inputs>
The image processing apparatus 301 of this embodiment can also select an anatomical structure and change the transmittance of the selected one by voice input. Note that this will be explained again after explaining the operation through the touch panel.

(Various functions through touch panel input)
The image processing device 301 displays anatomical structures in the three-dimensional medical image as independent objects. This allows you to select each of them individually and switch their display on and off. Blood vessels, for example, the hepatic artery, the portal vein, and the hepatic vein may be grouped so that they can be collectively selected, or they can be individually selected.

(Rotation function)
Rotation of the three-dimensional medical image can also be performed by an operation on the touch panel. When the operator touches the touch panel and moves the finger, the image processing device 301 rotates the three-dimensional medical image accordingly.

(enlargement/reduction function)
Also, when the operator touches two points on the screen and performs an operation (pinch-out operation, pinch-in operation) to separate or bring the two points closer together, the image processing apparatus 301 , the image may be enlarged or displayed.

(Display transparency switching function)
The image processing device 301 may change the display density when the operator touches any anatomical structure (for example, the liver). Specifically, it may be such that switching is performed between two states, a normal opaque display state and a translucent display state. That is, as an example. It may be such that when touched once, it becomes translucent, and when touched again, it returns to the normal display state.

As another aspect, for example, the transparency is set in a plurality of stages such as 0%, 30%, 70%, and 100% (non-display), and the display density is sequentially switched in a loop each time the touch screen is touched. It may be. In this case, 100% transparency (ie non-visible state) may be excluded from this loop. Naturally, the specific numerical value of the transmittance can be changed as appropriate. In short, it is sufficient that the degree of transparency is set in at least a plurality of stages and that they are switched.

According to such a display transparency switching function, the display transparency switching function can be exhibited simply by touching an arbitrary anatomical structure. Therefore, the operation can be performed simply and intuitively, compared with the method in which it is necessary to separately select some icon or the like or select a command in order to switch the degree of transparency.

Moreover, even when the display is switched in a loop as described above, it is preferable in that it is not necessary to separately select an icon or the like to return to the original display state. Furthermore, since such loop-shaped display switching can be realized only by changing the display color of the selected object, it is preferable in that image processing can be simplified and arithmetic processing can be performed with a small amount of memory.

Gestures for changing transparency are not limited to those described above. For example, when a finger is swiped vertically or horizontally (an example), the image processing device accepts the input, and the transparency is changed accordingly. It is good also as a structure changed. In this case, the transparency may be set in several steps such as 0%, 30%, 70%, 100% (hidden), or may be set continuously (continuously). A configuration in which the degree of transparency changes may be employed.

(display/hide switching function)
When the operator touches a predetermined anatomical structure for a certain period of time or more (one example), the image processing apparatus 301 puts the anatomical structure in a “selected state”. In order to indicate the "selected state", the anatomical structure (for example, the liver) may be displayed in a color different from that of the initial state, or displayed blinking.

The image processing apparatus 301 performs a swipe operation, a drag operation, or the like when the operator moves the fingertip (example) toward the edge of the screen while touching the selected anatomical structure (for example, the liver). ) to hide that anatomical structure. In this example, the liver is not displayed, and only the three-dimensional image such as blood vessels remains.

Such functionality is useful when the operator wishes to see only the desired anatomical structures. In addition, according to the method in which the anatomical structure can be hidden by simply selecting it directly and moving it as in the present embodiment, the display can be switched on and off only by selecting, for example, some icon. It is easier and more intuitive to operate compared to modes.

(Cut function)
The cut function works as follows. An example of cutting the liver and then hiding part of it will be described below. FIG. 8 is a flow chart of a series of operations.

The image processing apparatus 301 first displays a three-dimensional medical image as shown in FIG. 7A in step S1. Then, when the operator touches two points on an arbitrary anatomical structure (liver 71 in this case) as shown in FIG. (step S2). As for the timing, the two points may be touched at the same time or substantially at the same time.

Next, the image processing device 301 determines whether or not two points have been touched for a certain period of time or longer in order to make the so-called long press function work (step S3).

If the image processing apparatus 301 determines in step S3 that the touched points have been continued for a predetermined time or longer, the image processing apparatus 301 displays on the screen in a predetermined display mode that the two touched points P1 and P2 have been designated. The "predetermined display mode" may be any one, but for example, (i) both the points P1 and P2 and the line L1 connecting them are displayed, or (ii) only the points P1 and P2 are displayed. Alternatively, only the line L1 may be displayed. As for the points P1 and P2, rather than simple small dots, slightly larger graphical images (for example, circular, rectangular, polygonal, star-shaped, etc.) images such as those shown in FIG. , but a circular shape is exemplified here).

The image processing apparatus 301 may display the designated points P1 and P2 as shown in FIG. 7C even after the operator takes his/her hand off the screen. Further, it may be configured to accept fine adjustment of the positions of the points P1 and P2. For example, the circular graphical images of P1 and P2 and/or the line L1 may blink so that the user can see that the mode is for accepting fine adjustments. In FIG. 7C, as an example, the point P2 is slightly moved and finely adjusted to the point P2'.

This fine adjustment may be performed by the operator, for example, by moving the graphical images of the points P1 and P2 with his/her finger (operation on the touch panel). Alternatively, motion input may be used to finely adjust the positions of the points P1 and P2 without contacting the device. Displaying the cutting reference line L1 by voice input will be described later. Here, first, the cut function and the like of the present embodiment will be described on the premise of touch panel operation.

After the points P1 and P2 have been specified in this manner, the operator touches a predetermined icon (for example, an "OK" input icon) on the screen in step S4. Then, the cut function cuts the liver along a line L1 connecting points P1 and P2 as shown in FIG. 7D (step S5).

A first portion 71-1 and a second portion 71-2 divided into two across the line L1 can be operated as independent anatomical structures. In addition, as a method other than the above operation, for example, (i) by touching a predetermined area on the screen instead of touching the icon, or (ii) by touching multiple times (double tap in one example). You may make it perform the said function by operating outside. Voice input may be used.

Since it can be operated as an independent anatomical structure, for example, when the first region 71-1 is touched (step S6), the function described above causes the region to be displayed as shown in FIG. 7E. is selected. Then, the display density is switched. Specifically, only the first portion 71-1 is translucent. Touch again to return to the original display.

Further, for example, when the first portion 71-1 is long-pressed and a swipe operation or a drag operation is performed toward the peripheral portion of the screen, the portion is hidden, and the second portion 71-2, the blood vessel 73, and the like are displayed. Only 75 remain. Those that are hidden may be displayed as thumbnail images 66 as illustrated in FIG. 7F.

(Region designation)
The image processing device 301 designates a region of a part of the anatomical structure by the operator's operation as follows. FIG. 9A shows a state in which two points P1 and P2 are touched as in the operation described with reference to FIG. (although not shown).

From this state, for example, as shown in FIG. The positions of the latter two points P1' and P2' are specified, and based on them, a substantially rectangular area is specified. Specifically, a quadrangle surrounded by four points, two points P1 and P2 before movement and two points P1' and P2' after movement, is designated as an area.

Here, as in the above case, even if the operator releases his or her hand, the substantially rectangular area designated on the screen remains, and the positions of the points P1, P2, P1', and P2' are individually moved to change the position. It is also preferable that the image processing device 301 is configured so that fine adjustments can be made. As a method of confirming the specified area, for example, the operator may touch a predetermined icon (for example, an "OK" input icon) on the screen.

When the positions of four points P1, P2, P1' and P2' are specified as shown in FIG. 9B, circular points P1, P2, P1' and P2' (an example) are drawn so that each position can be finely adjusted. and a line connecting them may blink.

The designated area Sa1 (see FIG. 9B) is divided from other parts and can be operated as an independent object. Therefore, it is possible to change the display density of only the area or to switch the display ON/OFF. According to such a function, it is possible to observe the inner blood vessels 73 and 75 and confirm the relationship between the blood vessels 73 and 75 and the liver 71 by hiding only the region Sa1, for example.

It should be noted that the area specification does not necessarily have to be a quadrilateral, and the area may be specified in the form of a triangle or a polygon with pentagons or more.

In the above description, a medical image of the liver and its surroundings was taken as an example, but of course, the anatomical structure is not limited to a specific one in the present invention. For example, a medical image of the examiner's head may be displayed and various image processing may be performed thereon.

[Other functions by voice input]
(display/emergency switching)
It is also preferable that the display/non-display switching function and the cut function as described above can be executed only by voice input or the like without inputting to the touch panel.

First, regarding the selection of a predetermined anatomical structure, the selection is made when the target is spoken and recognized by voice, not by touch. For example, when the operator says "liver", the image processing device 301 recognizes it by voice and selects the liver. In order to indicate the "selected state", the anatomical structure (for example, the liver) may be displayed in a color different from that of the initial state or displayed blinking.

When the operator wants to change the liver permeability, for example, the operator utters "permeate". The image processing device 301 recognizes it by voice and switches the display of the liver to a translucent state. At this discharge, other anatomical structures (blood vessels and tumors, for example) remain opaque and visible. According to such a configuration, it is possible to check blood vessels and the like that are hidden behind the liver and cannot be seen under normal circumstances.

As one embodiment, for example, a configuration may be adopted in which the transmittance is changed according to the distance from the motion sensor to the operator's hand. In other words, the transmittance may gradually increase (or decrease) as the hand approaches the motion sensor, and the transmittance gradually decreases (or increases) as the hand moves away from the motion sensor. . Specifically, when the operator utters "transmission" and the image processing apparatus 301 recognizes it by voice, the motion input reception mode as described above is entered. The device then detects the distance from the motion sensor to the operator's hand and changes the transparency accordingly.

(line cut/box cut)
When performing a line cut, the operator utters, for example, "line cut". The image processing device 301 recognizes it by voice and displays the cutting reference line on the screen. This reference line may be something like line L1 in FIG. 7B.

Then, the image processing device 301 waits for motion input. The operator can change the position, length, direction, etc. of the cutting reference line by motion input. Thereby, the reference line can be set at a predetermined position without contact.

Then the area to the right of the reference line is removed, for example by saying "right cut". Alternatively, saying "left cut" removes the area to the left of the reference line. Translucent display may be used instead of removal. FIG. 5(b) shows an example, where the portion 371-2 on the right side of the reference line remains opaque and the portion 371-1 on the left side is translucent.

When performing a box cut, for example, the operator utters "box cut". Although detailed illustration is omitted, the image processing apparatus 301 recognizes the voice and displays a rectangle (an example) as a reference for resection on the screen. The size of the rectangle may be only one predetermined size, or may be prepared in a plurality of sizes such as large, medium, and small.

The box cut resects the target anatomy at a predetermined depth. The image processing apparatus 301 waits for motion input while displaying a rectangle serving as a reference for resection on the screen. The size and shape of the quadrangle may be fixed, or may be changeable. For example, it is possible to change the size and shape of the quadrangle by moving the positions of the corners of the initially displayed default quadrangle. Motion input can be used to move the position of the corners.

In this state, for example, when the operator moves his/her hand closer to the motion sensor 380, a substantially rectangular parallelepiped hole having a predetermined depth corresponding to the movement distance of the hand and having the outline of the rectangle is formed in the liver. It will be done. As a result, it is possible to obtain an image for medical observation in which a part of the liver is excised while the internal blood vessels are not excised.

It should be noted that it is also possible to rotate the entire three-dimensional medical image by a predetermined angle in a state in which the resection portion of a predetermined depth is formed in this manner. For example, when the operator utters "up", "15°", etc. in the rotation mode, the image processing apparatus 301 recognizes this and rotates the medical image in which the hole is formed by 15°. . Such a configuration is useful because it allows observation of the internal configuration of the hole (for example, the liver is partially excised but the blood vessels are displayed) from different angles.

Although the above describes removal with rectangular contours, it should be appreciated that the contours may be defined by triangles, polygons, circles, ellipses, or any other geometric shape.

In the above description, the area specified as the box is cut off, but conversely, only the area specified as the box may be left and the other areas may be hidden.

[Section B: Collective Handling of Multiple Anatomical Structures]
1. Problems of the Invention in this Section Volume data and the like are used as a three-dimensional medical image as illustrated in FIG. 5, as described above. By the way, in such a three-dimensional medical image (that is, one including several different types of anatomical structures), the liver and blood vessels have different CT values (signal values) of the data. there is Even in the same blood vessel, for example, arteries, veins, and portal veins have different CT values (signal values).

This is because, in fluoroscopic imaging using a contrast agent, image data is acquired by performing fluoroscopic imaging after a predetermined time has passed since the injection of the contrast agent. This is because there are differences in CT values (signal values) of various parts such as arteries, veins, portal veins, and liver parenchyma. Conventional methods create volume data for each part by setting thresholds and performing filter processing for each blood vessel and organ.

However, even for the same blood vessel, if the arteries, veins, and portal veins are individually registered due to differences in CT values (signal values), in the following cases, it is relatively time-consuming work. may be required. That is, in the viewer function of the three-dimensional medical image, for example, it is possible to switch between display with emphasis on blood vessels and display without emphasis. By being able to do this, for example, it is possible to switch between display and non-display of the peripheral part of the blood vessel (low CT value (signal value)) as necessary, and it is possible to observe as necessary. becomes (see FIG. 10). However, in the case of a three-dimensional medical image including a plurality of blood vessels of different types, the display mode of all blood vessels is uniformly changed unless the operator resets the display threshold or performs filtering for each blood vessel. Therefore, there is a problem that it is difficult to easily change the display.

On the other hand, in actual observation, it is sometimes preferable that the vascular system is separated from the vascular system, and the parenchymal system is separated from the parenchymal system, so that it can be handled. Therefore, in the invention of this section, the image processing apparatus has the following functions.

2. Functions and Operations The image processing apparatus of this embodiment reads volume data based on information obtained by imaging a patient (see also step S1 in FIG. 3).

Then, the signal value, CT value, and standard deviation (SD) in the volume are analyzed. For example, if the average CT value is 300 HU or more, it is automatically determined to be an artery, and if the average CT value is 100 HU or less, it is automatically determined to be an organ. In addition to CT values, the shape of a histogram of CT values is also recognized. In general, arteries tend to have a high peak and a narrow width (distribution), while portal veins and veins tend to have a low peak and a wide width (distribution). Therefore, based on such factors, automatic recognition of vessel type may be achieved.

As described above, it is possible to automatically discriminate arteries, veins, portal veins, etc. having different CT values (signal values) by the image processing apparatus and register the data. Alternatively, different colors may be automatically assigned to arteries, veins, portal veins, and the like to display them in different colors.

In addition, although there is originally one blood vessel histogram for each artery, vein, portal vein, etc., each is integrated and normalized, and one histogram is used for all blood vessels (in another embodiment, any blood vessel may be two or more). Specifically, as an example, the average value and the center of gravity of each histogram may be calculated, and one histogram may be created by shifting the whole so that the lower ones are matched with the higher ones.

In this way, when a plurality of blood vessels (or other anatomical structures) are integrated into a single histogram, the histogram can be collectively manipulated only by manipulating the histogram instead of performing the manipulation on the histogram of each blood vessel. You can change the display, etc. That is, for example, when the peripheral portion of a blood vessel does not need to be displayed, the image processing apparatus accepts a predetermined input from the operator and does not display portions below a certain reference value (or are not displayed), peripheral portions such as arteries, veins, and portal veins can be collectively hidden (see, for example, FIG. 10B). On the other hand, if it is desired to emphasize and display even the peripheral portion of the blood vessel, the lower limit of the CT value to be displayed should be set lower as shown in FIG. .

According to this configuration, it is necessary to change the display modes of the arteries, veins, portal veins, etc., so that the operability is very good and it contributes to practical use.

The above-described "predetermined input from the operator" may be performed by operating image buttons such as icons, cursors, and sliders on the screen, for example. Alternatively, it may be performed based on recognition of a predetermined gesture of the operator's finger on the touch panel.

For example, in a mode for changing the display of blood vessels, the operator may touch the touch panel with several fingers and simultaneously move the fingers in a predetermined direction to change the display of blood vessels. . More specifically, when several fingers are moved upward (first direction) on the screen at the same time, even peripheral parts of the blood vessels are displayed, and conversely, when the fingers are moved downward (second direction), A configuration may be adopted in which, when moved, the peripheral portion of the blood vessel (more precisely, the vicinity of the outer edge of the thick portion of the blood vessel as well) disappears.

It is also preferable that the above operation can be performed by motion input via the motion sensor 380 instead of operation on the touch panel. In other words, with this configuration, it is possible to switch the display of blood vessels (for example, other anatomical structures may be used) only by voice input and motion input, so there is no need to touch the touch panel, and cleanliness is maintained. A three-dimensional medical image can be observed as it is.

[Section C: Other Functions]
(1) 3D-pointer As an example of how to use a 3D medical image as illustrated in FIG. and the like.

In the case of a two-dimensional medical image, for example, by displaying a pointer as an image on the screen and positioning it at a predetermined site, the user can gaze at the site. However, in a three-dimensional medical image, it is necessary to place the pointer not within a plane but at an arbitrary site within a three-dimensional space. Such an operation of moving the pointer to any part in the dimensional space is relatively difficult to perform with an input interface such as a mouse or a touch panel.

Therefore, in the present embodiment, motion input may be used to arrange the pointers three-dimensionally. Specifically, the medical image processing apparatus 301 first receives an input of "pointer" (one example) with the speech recognition function. Then, as an example, a stereoscopic pointer is displayed on the screen.

As for the vertical and horizontal directions on the screen, the pointer may be moved according to the movement of the operator's hand in the horizontal plane. As for the depth direction, when the operator brings the hand closer to the motion sensor 380, the pointer moves in the depth direction of the three-dimensional medical image, and when the operator moves the hand away, the pointer moves in the front direction.

Regarding the display of the pointer, a display mode may be employed in which the display size gradually decreases as the pointer moves in the depth direction, and the display size gradually increases as the pointer moves in the front direction.

(2) Writing Schema Image In diagnosis and examination, a two-dimensional picture called a schema image representing body parts of a patient is sometimes used. Therefore, the image processing apparatus according to one aspect of the present invention may have a schema image writing function.

For example, the image processing apparatus uses data of a three-dimensional medical image (for example, see FIG. 5) to create a schema image corresponding to a predetermined input by an operator. The schema image may be any two-dimensional image such as a line drawing, a monochrome image, or a color image. Predetermined input by the operator includes various inputs such as input by touching an icon on the screen, voice input, or predetermined gesture input via a motion sensor.

As an example of creating a two-dimensional schema image corresponding to three-dimensional medical image data, for example, a medical image displayed in an orientation (example) as shown in FIG. The image may be made two-dimensional as it is and the data may be written. At this time, contour extraction processing may be performed to create a line drawing. Any data format may be used, but for example, a PDF (Portable Document Format) format or any other image format such as GIF, PNG, or JPEG can be used. A doctor can write sketches, observations, and the like on the schema image created in this way, for example, using a touch pen or a finger.

A schema image created by the image processing device can be sent out from the device and stored in a predetermined storage area connected to the network (see FIG. 3). For example, it may be taken in as part of the electronic medical record.

(3-1) Concealment The image processing apparatus of this embodiment is a portable type such as a tablet terminal, and depending on the situation, can be taken out of the hospital for use. Such a configuration may be useful, for example, when simulating an operation while checking a three-dimensional medical image of a specific patient outside the hospital. However, when taken out of the hospital, it is necessary from the viewpoint of security that the internal information is kept confidential.

Therefore, the image processing apparatus of one embodiment of the present invention preferably has the following functions: (a) a function of recognizing the current position of the apparatus; (or if the device is not inside the hospital), and (c) if it is outside the hospital (or if it is not inside the hospital) A function that automatically hides predetermined information held in the device when

Information that is not displayed includes, for example, information (identification information) that includes at least information that can identify a patient. It should be noted that the target information may be encrypted or access to the information may be prohibited instead of simply displaying the information.

As a method for determining whether it is inside or outside the hospital, for example, it may be based on whether it is within the range of the wireless network system in the hospital.

In this way, according to the configuration in which predetermined information is automatically concealed when the device is taken out of the hospital, it contributes to prevention of leakage of patient information, etc., and is more preferable from the viewpoint of security.

It should be noted that the effect of anonymization as described above is not necessarily limited to being inside or outside the "hospital". Such anonymization may be performed in a specific area of interest (which may be any facility or location).

(3-2)
Concerning anonymization, the following functions may also be provided. FIG. 12 is a diagram schematically showing a state in which anonymization is being performed. On this screen, all information that can identify a patient is kept confidential. Also, an icon 441 is displayed on the screen.

By the way, it is conceivable that a doctor viewing a three-dimensional medical image may want to temporarily check patient information, for example, to check which patient the image belongs to.

Therefore, the medical image processing apparatus of this example first determines that the icon 441 has been pressed (FIGS. 12 and 13), and then displays patient information. The patient information may be data stored inside the device, or may be data obtained by accessing an external server (for example, a server within a hospital system).

More specifically, it is also preferable that the conditions under which such patient information can be displayed are limited to certain conditions. For example, this is the case when fingerprint authentication of the operator is performed on the device. Of course, it may be the case that the person is authenticated by another authentication method.

It is also preferable that communication with an external server is possible only under secure communication conditions such as VPN (Virtual Private Network), for example.

The information to be displayed may be, for example, one, two, or three or more of the patient's initials, date of birth, sex, age, address, surgery date, doctor in charge, and the like. A patient ID, an examination ID, and the like may be displayed.

When confirming patient information under the above circumstances, only the minimum necessary information about the patient (for example, content that can identify the patient and the date of surgery or the doctor) is displayed. One is preferred in one form.

The displayed patient information may be automatically hidden again after a certain period of time. Alternatively, after being displayed once, the display may be continued during the operation (for example, the display is continued until the end of work (logout)).

As described above, by being able to check the minimum patient information and/or surgical information as needed, it is possible to avoid confusion between the displayed 3D medical image and the patient actually undergoing surgery. It is possible to reduce the possibility of occurrence of problems such as

Regarding the above features, this specification discloses not only the device invention but also the method and program invention corresponding to the above content.

(4) Stereo image A medical image processing apparatus according to one aspect of the present invention may have a function of displaying a stereo image as described below.

As shown in FIG. 11, an image including a first image 431L and a second image 431R may be displayed so that the operator can view stereoscopically. The first image 431L and the second image 431R display the same subject with a predetermined parallax. They may also be referred to as left-eye images, right-eye images, and the like.

As shown in FIG. 11, an operation pad area 433 may be displayed in the screen including the images 431L and 431R. This operation pad area 433 is an area for changing the display angle of the subject. When the operator touches the area 433 with a finger and moves the finger, the medical image processing apparatus simultaneously changes the display angles of the subject (two images) accordingly. In other words, it is possible to change the orientation of the subject in conjunction with the movement of the finger.

Although the above operation assumes an operation on a touch panel, the input method is not limited to this, and various methods can be used. For example, one or more of the contactless input methods disclosed herein can be utilized.

As shown in FIG. 11, in a mode in which one operation pad area 433 is displayed together with the first image 431L and the second image 431R, the operator can change the orientation of the subject image by operating there. can be intuitively understood.

The subject image is not particularly limited, but may be a contrast-enhanced image of blood vessels.

As described above, according to the configuration in which the first image 431L and the second image 431R are displayed, the doctor can more accurately grasp the three-dimensional structure of the subject through stereoscopic vision.

The apparatus according to one aspect of the present invention described above has a function of displaying a first image and a second image having different parallaxes. More specifically, it has a function of displaying an operation pad area for operating those display angles. For example, one or a combination of gesture input, voice input, motion input, etc. may be used to change the display of these stereoscopic images. This specification also discloses inventions of methods and programs corresponding to the above contents.

(5) Flow of Image Creation As a method of creating a medical image, for example, the following method may be adopted. In this method, first, a fluoroscopic captured image of a patient is saved in a predetermined data server (eg, a DICOM server: a server that saves data received from modalities in a predetermined format).

Then, in the predetermined data server (or other computer), individual parts of the subject are automatically recognized and managed as separate and independent objects. As shown in the table exemplified in FIG. 14, the information may be managed by being divided into categories such as arteries, veins, bones, organs, etc. (further subdivided categories thereof). For each individual object, information to be selected by the voice recognition function (for example, voice input of "aorta" for the aorta) may be set.

Also, an automatic discrimination key (symbol, alphabet, number, combination thereof) may be set for convenience in selectively reading only necessary objects among individual objects.

Also, an offset value may be set for each individual object. In this example, the offset value (see "CombineOfs" in the table) is set to "+50" for the aorta, "+100" for the abdominal artery, "+400" for the vein, etc. .

The offset value is used in one example as follows. For example, assume that the central CT value of the artery is 350 HU and the central CT value of the vein is 200 HU. In this way, blood vessels, such as arteries and veins (and portal veins), have different levels of contrast enhancement. Therefore, in order to align them, the offset value is used In the above example, the offset value of the vein is set to +150 HU to virtually increase the contrast effect, so that the artery and vein are treated with the same threshold setting. becomes possible. Of course, offset values may be set not only for arteries and veins but also for other blood vessels such as portal veins.

Also, the offset value of the parenchymal system instead of the vascular system may be used as follows. Here, as an example, it is assumed that the offset value of a solid organ (for example, liver) is set to a large value such as +700 HU. By setting the solid organ to such a large value, there is an advantage that the shape of the organ is easily kept as described below.

That is, for example, when the display of the obliteration of arteries is not necessary, an operation is assumed to increase the threshold value so that the obliteration is not displayed. , the shape will collapse. In order to prevent this (that is, to change the display mode of the blood vessels while maintaining the original shape of the solid organ), the solid organ may be displayed with a large offset value such as +700 HU.

(data conversion)
When blood vessels, bones, organs, etc. are managed as separate objects (set in a table as an example) in a predetermined data server (for example, a DICOM server), there are also the following advantages. In other words, when composing medical images on a tablet terminal, etc., there is no need to manually enter information about individual objects (text for speech recognition, offset values for image synthesis, etc.), making medical images easy to use. Images can be created.

In this embodiment, basically, it is necessary to create a medical image dedicated to the medical image processing apparatus. At that time, it is preferable to have the setting table as described above in that the time and effort for preparation can be saved even a little. However, depending on the technique (what type of surgery is to be performed, etc.), it may be desirable to prepare several tables. The reason for this is that it is preferable that the A blood vessel (or A organ) is clearly visible in one surgical procedure, while the B blood vessel (or B organ) is more clearly visible than the A blood vessel (or A organ) in another surgical procedure. is preferable. In other words, it is preferable in one embodiment that several tables each having an offset value set according to each technique are registered.

Also, in this case, the system or device may be configured as follows. That is, it is configured to (i) display a plurality of techniques, (ii) accept that the operator has selected one of them, and (iii) call up a table corresponding to it (display it as necessary). may have been

The surgical procedure may be displayed, for example, in a user interface for setting injection conditions, according to a mode (site selection) for selecting which part of the body is to be treated. For example, it may be configured such that, corresponding to a selected predetermined region (head, chest, abdomen, etc.), the surgical procedure corresponding to that region is displayed (see (i) above).

(5) Combined use with physical switches, etc. The above-described embodiments utilize voice input and motion input (input corresponding to human motion). In one form of the invention, certain physical switches may also be used in conjunction with these inputs.

Specifically, a switch (for example, a foot switch) that detects physical contact can be used. A “foot switch” is an example of a switch that is placed on the floor and used, and has a switch housing in which a sensor and a substrate are placed, and a pressing portion that is stepped on by a foot or the like. The pressing part is not limited, but may be a part configured to be movable so as to be pressed down when stepped on. The detection signal from the foot switch may be supplied to the outside via a cable (wired), or may be supplied to the outside wirelessly.

The foot switch may be electrically connected to the medical image processing apparatus of the present invention (eg, control unit 350, see FIG. 2), but is not limited to this. A configuration in which the foot switch is connected to another device (in one example, provided as part of the liquid injector) may be employed.

FIG. 15 shows an example of the arrangement of foot switches. In this example, the drug injection device includes an injection head 475 arranged near the imaging device 470, a first control unit (power supply unit) 478 connected thereto, and a console (second control unit) connected thereto. ) 476. The foot switch 477d is connected to the power supply unit as an example.

(Usage example of physical switch)
In the apparatus of one embodiment of the present invention, a plurality of reception conditions may be set for voice operation. A foot switch may be used as one of the acceptance conditions.

First, the following settings may be made (one or more) as a plurality of acceptance conditions for voice operation.
1) Accepts voice input only when there is motion input 2) Accepts voice input only when the footswitch is ON 3) Only when there is motion input and the footswitch is ON Accept voice input 4) Always accept voice input

Note that the above input conditions may be registered in the voice operation command table of the device. The input of 1) is, in other words, a configuration in which voice input is not accepted when there is no motion input.

Regarding voice input, it is also preferable that the input method is as follows. That is, when a combination of directions and angles such as "left 45 degrees, up 30 degrees" is input, command input is accepted. This is because the probability of misrecognition may increase if the information is not combined information. A detailed description will be given below.

When voice input is ON and only words such as "above" and "before" can be recognized, in some cases, words in the conversation during the treatment may be recognized, and the intention may be lost. It is also assumed that an input that does not Therefore, a configuration may be adopted in which a command is accepted only when a combination of a plurality of words is recognized.

The voice input method for preventing erroneous recognition is not necessarily limited to the combination information of direction and angle as described above. As a specific example, consider moving an image forward. In this case, when the word "direction" (one example) for identifying the operation mode for movement and the word "forward" (one example) for the desired direction of movement are recognized, a command is issued. accept. From the viewpoint of erroneous recognition prevention, an input such as "direction" + "forward" may be accepted, but an input such as "forward" + "direction" may not be accepted (that is, the order of combination is determined). ing). In addition, regarding the command to enlarge and display an image, instead of simply recognizing the word "enlarge", the command may be accepted only when a combination such as "3D" + "enlarge" is recognized. . With such a configuration, it is possible to prevent unintended voice input, and the usability is improved.

As one aspect of voice input, in addition to or in addition to the above aspects, a configuration may be used in which voice input is accepted only for a certain period of time (a configuration in which voice input is not accepted outside of that certain period of time).

Regarding the input of the footswitch, the footswitch may be such that it is ON only while the pressing portion is stepped on. As the processing for the footswitch input, the following processing may be performed only when the footswitch is pressed for a certain period of time or longer (so-called long press). According to this, it is possible to prevent unnecessary processing from being performed by unintentionally pressing the foot switch. The above-described input may be made with respect to a process in which such processing is a concern. Also, the above operation may be performed when a switch (physical switch) other than the foot switch is turned on.

Regarding voice input, the following functions may be provided. Here, in a typical image processing apparatus according to one aspect of the present invention, input is performed by voice input only in a predetermined mode suitable for voice input, and input is performed by another input method in other modes. shall be In such a case, when a predetermined input is made (for example, by inputting "Voice all ON" by voice), even those (at least some of them) that are not originally input by voice input by default will be voiced. A function that enables input may be provided. However, such extension of voice input is just an example, and may be automatically terminated after a certain period of time has elapsed and returned to the original state (timeout function). Although not limited, it may be a preset timeout period of about 1 minute, 3 minutes, or 5 minutes.

Although not limited, the input for designating the angle may always be configured to respond only to voice. In addition, a table of each word of voice input is prepared, and for each word, in what kind of situation the input is permitted (for example, when the foot switch is turned on for the word input of "aaa") Otherwise, the word input such as "bbb" is always accepted) may be set.

(Home position setting function)
By the way, basically, the image processing apparatus of the present invention may be configured so as to always default to display a medical image viewed from a fixed direction regardless of the part. One example is a configuration in which an image viewed from the front of the body is always displayed by default, regardless of the site and/or surgical procedure.

However, depending on the type of site and/or surgical procedure, it is also preferable in one embodiment to have a configuration in which an image viewed from a preset angle is displayed by default. For example, in thoracoscopic surgery, the lateral recumbent position is the basic position. Therefore, in the case of thoracoscopic surgery, the orientation of the lateral recumbent position may be set as the home position and displayed by default.

In other words, in one embodiment of the present invention, not only the front view but also separate display angles suitable for each region and/or type of surgery are preset and displayed by default. may be

The home position selection may be manually or automatically set depending on the surgical procedure, site and/or tumor location. As a specific example, whether the lesion is in the left or right lung determines whether the patient should be placed in the left or right lateral decubitus position. A configuration that automatically recognizes the position of a lesion (tumor) and automatically determines one of them accordingly is also useful.

(Example of screen display)
The screen displayed when the voice input is ON may be as shown in FIG. 16, for example. Although this screen displays some images in addition to the display example shown in FIG. 6, the images in FIG. It will be easily understood by a business person.

As shown in FIG. 16, there may be a voice recognition (voice input) ON display 397b on the screen so that the user can understand that voice recognition (voice input) is possible. Similarly, for motion input, there may be a motion input ON display 397a. It is also preferable to have a display unit 398 that displays the recognized voice as characters. This makes it possible to visually confirm what kind of voice input has been made.

Further, a display unit 399 may be provided to indicate whether or not the recognized voice has been accepted as a command. If the request is not accepted, a display such as "REJECTED" may be displayed so that the operator can visually confirm that the request was not accepted.

Although the embodiments of the present invention have been described above with reference to the drawings, the present invention can be modified in various ways without departing from the scope of the invention, and is not limited to the above examples.

A plurality of technical features have been described above, but the respective technical features disclosed above can be used in combination as appropriate, except in cases where they contradict each other. also disclose Also, for convenience of explanation, even where some technical features are described as one embodiment, one or more of those features may be omitted. It should be understood that a person skilled in the art can grasp the technical contents described without distinguishing between inventions of products, inventions of methods, or inventions of computer programs as any invention.

(Appendix)
This specification discloses the following inventions (note that the symbols in parentheses do not limit the invention in any way):
1. a display (361);
a controller (350) connected to the display;
an audio input device (370);
a motion sensor (380);
A medical image processing apparatus (301) comprising
The control unit (350)
a: an image display unit for displaying a three-dimensional medical image on the display;
b: a mode selection unit that recognizes voice input using the voice recognition device (170) and switches modes related to display of the three-dimensional medical image accordingly;
c: a display processing unit that recognizes an operator's motion input via the motion sensor and changes the display of the three-dimensional medical image accordingly;
having
Medical image processing equipment. Alternatively, a device or system having at least a control unit having the features as described above.

2. As the mode for displaying a three-dimensional medical image,
Zoom mode to enlarge and reduce the image,
pan mode to translate the image,
rotation mode to rotate the image,
A medical image processing apparatus as described above, comprising at least one of

3. In the zoom mode, moving the operator's hand in a first direction magnifies the three-dimensional medical image, and moving it in the opposite second direction reduces the three-dimensional medical image.
A medical image processing apparatus as described above.

4. The medical image processing apparatus described above, wherein in the zoom mode, the three-dimensional image is enlarged or reduced without being rotated.

5. In the rotation mode, the three-dimensional medical image rotates according to the operator's hand movement.
A medical image processing apparatus as described above.

6. The medical image processing apparatus described above, wherein in the rotation mode, the three-dimensional image is not enlarged or reduced but is enlarged or reduced.

7. A housing (301a) is provided,
The portable medical image processing apparatus described above, wherein at least the display and the control unit are integrally incorporated in the housing.

8. A medical image processing apparatus as described above, wherein the motion sensor comprises one or more cameras.

9. The medical image processing apparatus described above, wherein the three-dimensional medical image includes at least liver and blood vessel image data.

10. The medical image processing apparatus as described above, wherein the audio input device is a microphone.

11. to the computer,
a: processing for displaying a three-dimensional medical image on a display;
b: a process of recognizing an input voice using a voice recognition device (170) and switching the mode related to the display of the three-dimensional medical image accordingly;
c: a process of recognizing an operator's motion input via a motion sensor and changing the display of the three-dimensional medical image accordingly;
A medical image processing program that allows you to perform

12. As the mode for displaying a three-dimensional medical image,
Zoom mode to enlarge and reduce the image,
pan mode to translate the image,
rotation mode to rotate the image,
A program as described above, having at least one of

13. In said zoom mode, the computer is instructed such that movement of the operator's hand in a first direction magnifies the three-dimensional medical image and movement in the opposite second direction reduces the three-dimensional medical image. A program as described above that causes processing to occur.

14. The program as described above, causing the computer to perform processing such that, in the zoom mode, the 3D image is enlarged or reduced without being rotated.

15. The program according to the above, causing the computer to perform processing so that, in the rotation mode, the three-dimensional medical image rotates according to the hand movement of the operator.

16. The program as described above, causing the computer to perform processing such that, in the rotation mode, the three-dimensional image is not enlarged or reduced, but is enlarged or reduced.

17. causing the computer to display a three-dimensional medical image on the display;
a computer recognizing input speech using a speech recognition device (170) and switching modes for displaying the three-dimensional medical image accordingly;
a computer recognizing an operator's motion input via a motion sensor and modifying the display of the three-dimensional medical image accordingly;
A method of operating a medical imaging device, comprising:

18. As the mode for displaying a three-dimensional medical image,
Zoom mode to enlarge and reduce the image,
pan mode to translate the image,
rotation mode to rotate the image,
The method as described above, having at least one of

19. In the zoom mode, moving the operator's hand in a first direction enlarges the three-dimensional medical image, and moving it in the opposite second direction reduces the three-dimensional medical image. , a method of operation as described above.

20. A method of operation as described above, wherein in said zoom mode the 3D image is enlarged or reduced without being rotated.

21. A method of operation as described above, wherein in the rotation mode, the three-dimensional medical image is rotated in response to the operator's hand movement.

1 Imaging device 10 Liquid injection device 71, 371 Liver 73, 75, 375 Blood vessel 301 Medical image processing device 301a Housing 350 Control unit 351 Voice recognition unit 353 Gesture recognition unit 355a Image display unit 355b Operation determination unit 355c Display processing unit 359 Storage Unit 360 Touch panel type display 361 Display 363 Touch panel 365 Input device 367 Communication unit 368 Interface 369 Slot 370 Microphone 380 Motion sensor

Claims (12)

a display;
a controller connected to the display;
an audio input device;
a motion sensor;
A medical image processing apparatus comprising
The control unit
a: an image display unit for displaying a three-dimensional medical image on the display;
b: a mode selection unit that recognizes voice input using the voice recognition device and switches a mode related to display of the three-dimensional medical image accordingly;
c: a display processing unit that recognizes an operator's motion input via the motion sensor and changes the display of the three-dimensional medical image accordingly;
has a
Medical image processing equipment. As the mode for displaying a three-dimensional medical image,
Zoom mode to enlarge and reduce the image,
pan mode to translate the image,
rotation mode to rotate the image,
The medical image processing apparatus according to claim 1, comprising at least one of: In the zoom mode, moving the operator's hand in a first direction magnifies the three-dimensional medical image, and moving it in the opposite second direction reduces the three-dimensional medical image.
The medical image processing apparatus according to claim 2. 4. The medical image processing apparatus according to claim 2, wherein in said zoom mode, enlargement or reduction is performed without rotating the three-dimensional image. In the rotation mode, the three-dimensional medical image rotates according to the operator's hand movement.
The medical image processing apparatus according to claim 2. 6. The medical image processing apparatus according to claim 5, wherein, in said rotation mode, enlargement or reduction is performed without enlarging or reducing the three-dimensional image. with a housing,
7. The medical image processing apparatus according to any one of claims 1 to 6, which is a portable medical image processing apparatus, wherein at least the display and the control unit are integrally incorporated in the housing. The medical image processing apparatus according to any one of claims 1 to 7, wherein said motion sensor comprises one or more cameras. The medical image processing apparatus according to any one of claims 1 to 8, wherein said three-dimensional medical image includes at least liver and blood vessel image data. The medical image processing apparatus according to any one of claims 1 to 9, wherein said audio input device is a microphone. to the computer,
a: processing for displaying a three-dimensional medical image on a display;
b: a process of recognizing an input voice using a voice recognition device and switching the mode related to the display of the three-dimensional medical image accordingly;
c: a process of recognizing an operator's motion input via a motion sensor and changing the display of the three-dimensional medical image accordingly;
A medical image processing program that allows you to perform causing the computer to display a three-dimensional medical image on the display;
a computer recognizing an input voice using a voice recognition device and switching a mode for displaying the three-dimensional medical image accordingly;
a computer recognizing an operator's motion input via a motion sensor and modifying the display of the three-dimensional medical image accordingly;
A method of operating a medical imaging device, comprising:

Images