JP7326308B2 - MEDICAL IMAGE PROCESSING APPARATUS, OPERATION METHOD OF MEDICAL IMAGE PROCESSING APPARATUS, ENDOSCOPE SYSTEM, PROCESSOR DEVICE, DIAGNOSTIC SUPPORT DEVICE, AND PROGRAM - Google Patents

Description

The present invention relates to a medical image processing apparatus and method, an endoscope system, a processor, a diagnosis support apparatus, and a program, and in particular, image processing that provides information for supporting diagnosis by processing time-series medical images. Regarding technology.

In the medical field, examinations using endoscope systems are performed. In recent years, there has been known a system that recognizes a lesion contained in a medical image by using image analysis technology. As disclosed in Patent Document 1 and Patent Document 2, in an endoscope system, a body cavity is photographed using a camera at the tip of a scope, and a processor device performs recognition processing on the obtained observation image. and notify the user of the recognition result. The user can take appropriate measures according to the reported recognition result.

For example, if a lesion is being imaged, the processor device recognizes the area of the lesion from the image by performing image analysis, and recognizes whether it is cancerous or non-cancerous, Notify the user of the recognition result. The user can refer to the reported information to observe the lesion in more detail, and if it can be determined that the lesion is cancerous, can take measures such as excision of the lesion. A system with such a function supports decision-making such as diagnostic findings by a doctor or the like.

Japanese Patent No. 6150554 WO2017/002184 Japanese Patent Publication No. 7-27481

In the diagnostic support function such as the endoscope system described above, misrecognizing a non-cancerous lesion as cancerous, or misrecognizing a cancerous lesion as non-cancerous, resulting in an incorrect recognition result is a problem. However, during endoscopic examination, the lesion may be hidden or the observed image may be blurred due to manipulation of the scope, fetal movement, or debris, making it difficult to accurately recognize the lesion by image analysis. Sometimes.

In such a case, for example, as described in Patent Literature 3, image recognition is performed by applying a technology of an abnormality display processing method that switches from normal image display to display of an abnormality message when an abnormality is detected. It is conceivable to notify that the image is unsuitable for

However, operation of the endoscope is complicated, and the user needs to operate the scope and check the observation image in parallel. Notification of the recognition results by the diagnosis support function makes the user's identification work more efficient, while for example, the image recognition results are notified even when the user who is observing the image has no intention of distinguishing lesions. This, in turn, impairs the user's attention, hinders observation, and may lead to overlooking an attention area such as another lesion area.

This problem is not limited to endoscope systems, but is common to systems such as ultrasonic diagnostic equipment that process medical images captured in time series to provide support.

The present invention has been made in view of such circumstances, and is a medical image that can assist diagnosis without hindering the user's observation of the image and without impairing the user's attention required during observation. An object of the present invention is to provide a processing device and method, an endoscope system, a diagnostic support device, and a program.

A medical image processing apparatus according to an aspect of the present disclosure includes an image acquisition unit that acquires a plurality of time-series images including a subject image, and whether or not the image obtained from the image acquisition unit is an image that is unsuitable for recognition. a motion estimation unit that performs motion estimation from two or more images obtained from the image acquisition unit; and a motion information obtained from the motion estimation unit. Notification information is controlled based on the determination unit, the classification unit that recognizes and classifies images obtained from the image acquisition unit, and the behavior information obtained from the behavior determination unit and the classification results obtained from the classification unit. and a notification control unit for

According to the medical image processing apparatus according to this aspect, the behavior of the user is determined from the acquired image, and the classification result by image recognition is reported or not supported in accordance with the intention of the behavior. be able to. In addition, according to this aspect, it is possible to avoid notification of classification results for images that are unsuitable for recognition. can provide information with high classification results.

The plurality of time-series images may be a moving image, or a group of images captured at specific time intervals such as continuous shooting or interval shooting. In addition, the time intervals of time-series photography do not necessarily have to be constant.

The medical image processing apparatus may be configured as a single device, or may be configured by combining a plurality of devices. For example, a medical image processing device may be implemented using one or more computers. "Apparatus" includes the concepts of "system" and "module".

The “unsuitable image” may be, for example, an image in which the subject is blurred, an image in which the subject is blurred, an image in which the subject is covered with water, or an image in which the subject has residue.

In the medical image processing apparatus according to another aspect of the present disclosure, the propriety determination unit may include a recognition unit that recognizes whether the image obtained from the image acquisition unit is an image unsuitable for recognition. .

The term "recognition" includes concepts such as identification, discrimination, inference, estimation, detection, classification, and region extraction. "Recognizer" includes concepts such as recognizer, classifier, discriminator, detector, classifier, and recognition model. A "recognition model" may be, for example, a trained model that has acquired recognition performance through machine learning.

The medical image processing apparatus according to still another aspect of the present disclosure may be configured such that the motion estimation unit and the motion determination unit do not perform processing on an image that the propriety determination unit determines to be an inappropriate image.

The medical image processing apparatus according to still another aspect of the present disclosure may be configured so that the classifying unit does not perform the classification process on the image that the propriety determining unit determines to be an inappropriate image.

According to this aspect, it is possible to avoid providing inaccurate classification results.

In the medical image processing apparatus according to still another aspect of the present disclosure, the motion information may include a motion vector, and the action determination unit may determine the user's action based on the magnitude of the motion vector.

In the medical image processing apparatus according to still another aspect of the present disclosure, the gesture determination unit may be configured to determine gestures using a database that defines correspondence relationships between motion information and user's gestures.

A medical image processing apparatus according to still another aspect of the present disclosure may be configured to include a storage unit that stores a database.

In the medical image processing apparatus according to still another aspect of the present disclosure, the action may include the action of discrimination.

In the medical image processing apparatus according to still another aspect of the present disclosure, the notification control unit is configured to not notify the classification result of the classification unit for images for which the determination result of the behavior determination unit does not correspond to the behavior of discrimination. you can

In the medical image processing apparatus according to still another aspect of the present disclosure, the classification unit may be configured to recognize the lesion area from the image obtained from the image acquisition unit and classify the lesion into classes.

In the medical image processing apparatus according to still another aspect of the present disclosure, the classifier may be configured using a convolutional neural network.

The medical image processing apparatus according to still another aspect of the present disclosure may be configured to include a notification unit that notifies the classification result of the classification unit based on the control of the notification control unit.

In the medical image processing apparatus according to still another aspect of the present disclosure, each of the plurality of time-series images may be an endoscopic image captured using an electronic endoscope.

A medical image processing method according to still another aspect of the present disclosure includes an image acquisition step of acquiring a plurality of time-series images including a subject image; a motion estimation step of estimating motion from two or more images obtained from the image acquisition step; and determining the user's behavior based on the motion information obtained from the motion estimation step. a classification step of recognizing and classifying the image obtained from the image acquisition step; based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step, the notification information and a notification control step for controlling the

In the medical image processing method of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined. In this case, elements of the processing units and functional units as means for carrying out the processing and operations specified in the medical image processing apparatus can be grasped as the elements of the processes (steps) of the corresponding processing and operations. Also, the medical image processing method of this aspect can be understood as a method of operating a medical image processing apparatus.

An endoscope system according to still another aspect of the present disclosure is an endoscope system that includes an electronic endoscope that captures images of the inside of a body cavity and a processor device that processes image signals obtained from the electronic endoscope. The processor device includes an image acquisition unit that acquires a plurality of time-series images including a subject image captured using an electronic endoscope, and an image acquired from the image acquisition unit that is an image unsuitable for recognition. a motion estimation unit that performs motion estimation from two or more images obtained from the image acquisition unit; and motion information obtained from the motion estimation unit. Based on the behavior determination unit that judges, the classification unit that recognizes and classifies images obtained from the image acquisition unit, and the behavior information obtained from the behavior determination unit and the classification results obtained from the classification unit. and a notification control unit that controls information.

In the endoscope system of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined.

A processor device according to still another aspect of the present disclosure is a processor device that processes an image signal obtained from an electronic endoscope, and includes a plurality of time-series images including subject images captured using the electronic endoscope. an image acquisition unit that acquires an image; a propriety determination unit that determines whether the image obtained from the image acquisition unit is an image unsuitable for recognition; A motion estimation unit that performs estimation, a motion determination unit that determines the user's actions based on the motion information obtained from the motion estimation unit, and a classification unit that recognizes and classifies images obtained from the image acquisition unit. and a notification control unit for controlling notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit.

In the processor device of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined.

A diagnosis support device according to still another aspect of the present disclosure includes an image acquisition unit that acquires a plurality of time-series images including a subject image, a motion estimation unit that performs motion estimation from two or more images obtained from the image acquisition unit; and a motion information obtained from the motion estimation unit. A behavior determination unit, a classification unit that recognizes images obtained from the image acquisition unit and performs classification processing, and notification information based on the behavior information obtained from the behavior determination unit and the classification results obtained from the classification unit. A notification control unit for controlling, and a notification unit for notifying information including the classification result of the classification unit based on the control of the notification control unit.

In the diagnosis support device of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined. The display section that displays the classification result of the classification section can be one form of the notification section.

A program according to still another aspect of the present disclosure provides a computer with an image acquisition step of acquiring a plurality of time-series images including a subject image, and whether the image obtained from the image acquisition step is an image unsuitable for recognition. a motion estimation step of estimating motion from two or more images obtained from the image acquisition step; and determining the user's behavior based on the motion information obtained from the motion estimation step. a classification step of recognizing and classifying the image obtained from the image acquisition step; based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step, the notification information and a notification control step for controlling the

In the program of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined. In that case, the elements of the processing unit and functional unit as means for carrying out the processing and operation specified in the medical image processing apparatus can be grasped as program elements that implement the steps or functions of the corresponding processing and operation. .

According to the present invention, the behavior of the user is determined from time-series images, and notification information is controlled based on the obtained behavior information. Therefore, it is possible to realize support that matches the behavior of the user. According to the present invention, useful support information such as image classification results can be provided without hindering the user's observation of images and without impairing the user's attention required for observation.

FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a front view showing the distal end face of the rigid distal end portion of the electronic endoscope. FIG. 3 is a block diagram showing the control system of the endoscope system. FIG. 4 is a block diagram showing functions of the medical image processing apparatus according to the first embodiment of the present invention. FIG. 5 is an example of an image showing no lesion. FIG. 6 is an example of an image in which a lesion is blurred due to defocus. FIG. 7 is an example of an image in which a lesion is blurred due to motion blur. FIG. 8 is an example of a group of three images obtained in time series. FIG. 9 is an example of an image when the motion of the image is small. FIG. 10 is an example of an image when the motion of the image is large. FIG. 11 is an explanatory diagram showing an example of a database that defines the correspondence between motion information and user's actions. FIG. 12 is a diagram showing an example of notification of classification results obtained from the classification unit. FIG. 13 is a diagram showing an example of notification when the classification result is set to non-notification. FIG. 14 is a diagram showing an example of notification that presents information indicating that the classification process has not started. 15 is a flowchart illustrating an example of the operation of the medical image processing apparatus according to the first embodiment; FIG. FIG. 16 is a block diagram showing a configuration example of a medical information management system according to the second embodiment.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

<<Configuration example of endoscope system>>
FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. The endoscope system 10 includes an electronic endoscope 12 , a light source device 14 and a processor device 16 . A display device 18 and an input device 19 are connected to the processor device 16 . The electronic endoscope 12 may be called a "scope", an "electronic scope", or simply an "endoscope".

The electronic endoscope 12 of this example is a flexible endoscope. The electronic endoscope 12 includes an insertion section 20 , an operation section 30 and a universal cord 40 . The insertion portion 20 is a portion that is inserted into the body cavity of the person to be examined. The insertion section 20 includes a distal rigid section 22, a bending section 24, and a flexible section 26 in order from the distal end toward the proximal side. An illumination optical system, an objective optical system, an imaging element, and the like are arranged inside the distal end rigid portion 22 . The bending portion 24 has a structure that smoothly bends in four directions of up, down, left, and right from a reference position according to the operation of the angle knob 31 . The proximal side of the flexible portion 26 is called the proximal end portion of the insertion portion 20 .

The operation portion 30 is provided at the proximal end portion of the insertion portion 20 . The operation unit 30 includes various operation members operated by the operator. For example, the operation unit 30 includes an angle knob 31 used for bending operation of the bending portion 24, an air/water supply button 32 for performing an air/water supply operation, and a suction button 33 for performing a suction operation. is provided. An operator (an example of a user) can bend the bending portion 24 and freely change the direction of the distal end rigid portion 22 by operating the angle knob 31 . The operation unit 30 also includes a mode switching switch 34 used for switching observation modes, a zoom operation unit 35, and a still image photographing instruction unit (not shown) for instructing photographing of a still image of the site to be observed. is provided.

Further, a treatment instrument introduction port 36 is provided in the operation section 30 . The treatment instrument introduction port 36 is an opening for inserting a treatment instrument (not shown) into a treatment instrument insertion passage (not shown) passing through the insertion portion 20 . Examples of treatment tools include biopsy forceps, catheters, high-frequency snares, and the like. Treatment tools also include guide tubes, trocar tubes, sliding tubes, and the like. The treatment instrument introduction port 36 may be called a forceps port.

The universal cord 40 is a cord for connecting the electronic endoscope 12 to the light source device 14 and processor device 16 . A cable extending from the insertion portion 20 and a light guide are inserted through the universal cord 40 . The cables extending from the insertion portion 20 include a communication cable used for signal transmission and a power supply cable used for power supply. A connector 42 is provided at one end of the universal cord 40 .

The connector 42 is a composite type connector including a video connector 42A and a light guide connector 42B. One end of a cable is arranged in the video connector 42A. The video connector 42A is detachably connected to the processor device 16. As shown in FIG. One end of a light guide is arranged in the light guide connector 42B. The light guide connector 42B is detachably connected to the light source device 14 . A water supply connector 42C is provided to the light guide connector 42B, and the water supply tank 44 is connected via the water supply connector 42C.

The processor device 16 is electrically connected to the light source device 14 via the connector 42 . The processor device 16 centrally controls the operation of the endoscope system 10 including the light source device 14 . The processor unit 16 supplies power to the electronic endoscope 12 via a cable inserted through the universal cord 40, and controls the driving of the imaging device. The processor unit 16 also receives an imaging signal transmitted from the electronic endoscope 12 via a cable, performs various signal processing on the received imaging signal, and converts it into image data. The image data converted by the processor device 16 is displayed on the display device 18 as an endoscope-captured image (observation image).

FIG. 2 is a front view showing the distal end face 22A of the rigid distal end portion 22 of the electronic endoscope 12. As shown in FIG. An illumination window 50 , an observation window 52 , a forceps outlet 54 , and an air/water supply nozzle 56 are provided on the distal end surface 22</b>A of the distal end rigid portion 22 . Although not shown in FIG. 2, an emission end 122 of the light guide 120 that guides the light from the light source device 14 is arranged behind the illumination window 50 (see FIG. 3). Illumination light is emitted from the illumination window 50 to the observation area. Two illumination windows 50 are arranged at symmetrical positions with the observation window 52 interposed therebetween.

The observation window 52 is a window for taking in the reflected light from the observed area and capturing an image of the observed area. Although not shown in FIG. 2, behind the observation window 52, an objective optical system 60 and an imaging device 62 for taking in image light of the region to be observed in the body cavity are arranged (see FIG. 3). .

The forceps outlet 54 is connected to a forceps channel (not shown) arranged in the insertion section 20 and communicates with the treatment instrument introduction port 36 (see FIG. 1) provided in the operation section 30 . The treatment instrument inserted from the treatment instrument introduction port 36 is taken out from the forceps outlet 54 into the body cavity.

The air/water nozzle 56 jets cleansing water or air toward the observation window 52 and/or into the body cavity in response to the operation of the air/water button 32 (see FIG. 1) provided on the operation unit 30. . The cleaning water and air are supplied from an air/water supply device incorporated in the light source device 14 .

FIG. 3 is a block diagram showing the control system of the endoscope system 10. As shown in FIG. The distal end rigid portion 22 of the electronic endoscope 12 includes an objective optical system 60, an imaging element 62, an analog front end (AFE) circuit 64, a timing generator (TG) 65, a CPU (Central Processing Unit) 66 is provided.

The objective optical system 60 is configured using a zoom lens. A light guiding prism (not shown) is arranged between the objective optical system 60 and the imaging element 62 . The imaging element 62 may be a CMOS (complementary metal oxide semiconductor) type imaging element or a CCD (Charged Coupled Device) type imaging element. Here, an example using a CMOS type image sensor will be described.

Although not shown, the imaging device 62 is a solid-state imaging device of a single-plate color imaging system having a color filter composed of a plurality of color segments. The color filters may be, for example, Bayer array primary color filters including red (R), green (G), and blue (B).

A large number of pixels are arranged in a matrix on the imaging surface of the imaging device 62, and each pixel is provided with a photosensor. Light incident on the imaging surface of the imaging device 62 is accumulated as electric charge in the photosensor of each pixel. The amount of signal charge accumulated in the photosensor of each pixel is sequentially read out as a pixel signal by scanning in the vertical and horizontal directions by a vertical scanning circuit and a horizontal scanning circuit (both not shown), and is read out at a predetermined frame rate. output.

A timing generator 65 generates driving pulses for the imaging element 62 and synchronization pulses for the analog front-end circuit 64 under the control of the CPU 66 . Driving pulses for the imaging device 62 include a vertical scanning pulse, a horizontal scanning pulse, a reset pulse, and the like.

The imaging element 62 is driven by a driving pulse input from the timing generator 65, photoelectrically converts an optical image formed on the imaging surface through the objective optical system 60, and outputs the resultant as an imaging signal.

Analog front end circuitry 64 includes correlated double sampling (CDS) circuitry, automatic gain control (AGC) circuitry, and an A/D converter. The CDS circuit performs correlated double sampling processing on the imaging signal output from the imaging device 62 to remove reset noise and amplifier noise generated in the imaging device 62 .

The AGC circuit amplifies the imaging signal from which noise has been removed by the CDS circuit with a gain (amplification factor) designated by the CPU 66 . The A/D converter converts the imaging signal amplified by the AGC circuit into a digital signal having a predetermined number of bits and outputs the digital signal. The imaging signal digitized and output by the analog front-end circuit 64 is input to the processor unit 16 through the signal line.

Note that the imaging element 62 , the analog front-end circuit 64 and the timing generator 65 can be configured as a monolithic integrated circuit, and each of these circuit elements is included in one imaging chip 68 . The imaging chip 68 mounted on the electronic endoscope 12 of this example is a so-called "CMOS sensor chip" and is mounted on a support substrate (not shown).

The processor device 16 includes a CPU 70 , a ROM (read-only memory) 72 , a RAM (Random Access Memory) 74 , a digital signal processor (DSP) 76 , and a display control circuit 78 .

The CPU 70 controls each part in the processor device 16 and controls the entire endoscope system 10 in an integrated manner. Various programs and control data for controlling the operation of the processor device 16 are stored in the ROM 72 . The RAM 74 temporarily stores programs and data executed by the CPU 70 .

Under the control of the CPU 70, the digital signal processing circuit 76 performs various signal processing such as color interpolation, color separation, color balance adjustment, gamma correction, and image enhancement on the imaging signal input from the analog front end circuit 64. to generate image data. The digital signal processing circuit 76 also performs image recognition processing. The digital signal processing circuit 76 functions as an image processing section. The digital signal processing circuit 76 also has the function of an image recognition section that performs image recognition processing.

Image data output from the digital signal processing circuit 76 is input to the display control circuit 78 . The display control circuit 78 converts the image data input from the digital signal processing circuit 76 into a signal format compatible with the display device 18 and displays it on the screen of the display device 18 .

The display device 18 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. The display device 18 can display various types of information such as various setting information necessary for the processing of the processor device 16 or information indicating processing results.

The display device 18 and the input device 19 function as a user interface. The input device 19 may be, for example, a keyboard, mouse, touch panel, operation buttons, voice input device, or an appropriate combination thereof. A user may use the input device 19 to enter various instructions and/or information. The processor device 16 can execute various processes according to instructions and/or information input from the input device 19 .

The light source device 14 generates light for illuminating the body cavity through a light guide 120 inserted into the electronic endoscope 12 . The light source device 14 includes a first light source 100 , a first light source drive circuit 101 , a second light source 102 , a second light source drive circuit 103 , a CPU 104 and a multiplexer 105 . The CPU 104 communicates with the CPU 70 of the processor device 16 and controls the first light source driving circuit 101 and the second light source driving circuit 103 .

The first light source drive circuit 101 causes the first light source 100 to emit light according to instructions from the CPU 104 . The second light source drive circuit 103 causes the second light source 102 to emit light in accordance with instructions from the CPU 104 .

The first light source 100 is, for example, a laser diode that emits a blue laser with a wavelength of 445 nm. The first light source 100 is pulse-driven by a first light source drive circuit 101 to control the amount of light emitted. The second light source 102 is, for example, a laser diode that emits a blue laser with a wavelength of 405 nm. The second light source 102 is pulse-driven by a second light source drive circuit 103 to control the amount of light emitted. Blue light with a wavelength of 405 nm is used for special light observation.

The multiplexing unit 105 multiplexes the light emitted from the first light source 100 and the light emitted from the second light source 102 and outputs the combined light to the incident end 121 of the light guide 120 .

A phosphor 124 is provided between the exit end 122 of the light guide 120 and the illumination window 50 of the electronic endoscope 12 . The blue laser light that has passed through the light guide 120 irradiates the phosphor 124 to excite the phosphor 124 , and part of the laser light passes through the phosphor 124 and is emitted from the illumination window 50 as blue light.

The phosphor 124 is excited by blue laser light, and emits light (yellow in color) in a wide range of wavelengths from the boundary between blue and green to red. This yellow light and the blue light that passes through the phosphor 124 are mixed to form white light, which illuminates the subject through the illumination window 50 . The blue light that passes through phosphor 124 includes part of the blue light that is emitted from phosphor 124 .

As described above, the phosphor 124 emits yellow light and transmits blue light with a wavelength of 445 nm when irradiated with a blue laser beam with a wavelength of 445 nm, but is irradiated with a blue laser beam with a wavelength of 405 nm. In some cases, it has the property of penetrating most of it.

That is, by controlling the mixing ratio of the blue laser light having a wavelength of 445 nm and the blue laser light having a wavelength of 405 nm, the ratio of the blue light passing through the phosphor 124 and the yellow light emitted from the phosphor 124 can be controlled. is possible.

A color image of the subject is reproduced by receiving reflected light from the subject illuminated with white light using the imaging element 62 .

When observing the inside of a body cavity using the endoscope system 10 configured as described above, the electronic endoscope 12, the light source device 14, the processor device 16, and the display device 18 are all powered on. Then, the insertion portion 20 of the electronic endoscope 12 is inserted into the body cavity, and an image of the body cavity captured by the imaging device 62 is displayed on the screen of the display device 18 while illuminating the body cavity with illumination light from the light source device 14 . will be observed in

The endoscope system 10 of this example has a white light observation mode and a narrowband light observation mode. The white light observation mode is a mode in which an observation image with natural colors is displayed on the display device 18 using a captured image obtained by imaging an observation target using white light as illumination light. An image obtained by imaging an observation target in the white light observation mode is called a "white light observation image". The illumination light may also be called "observation light".

The narrow-band light observation mode uses an image signal obtained by imaging the observation target with narrow-band light of a specific wavelength band as illumination light, for example, emphasizing the visualization of blood vessels in a specific depth region of the observation target. In this mode, an image is generated and an image suitable for observing blood vessels is displayed on the display device 18 . An image obtained by imaging an observation target in the narrow-band light observation mode is called a "narrow-band light observation image".

The endoscope system 10 may have a plurality of types of narrow-band light observation modes in which the types of wavelength bands of narrow-band light to be used or their combinations are different.

<<Overview of Diagnosis Support Function in Endoscope System 10>>
In endoscopy, the user, the doctor, operates the scope to detect lesions in screening, and in detail, to check the condition of the detected lesions. is one of the important issues. For example, when the discrimination of a certain lesion is finished, the user may forget the presence of surrounding lesions. The user seamlessly transitions from screening work to discrimination work, and from discrimination work to screening work, and diagnostic support systems are required to operate in accordance with such user operations.

As a means for switching between the screening work and the discrimination work, timing at which the type of light source is switched can be considered, but this cannot be done when the work is changed while the same light source is used.

The medical image processing apparatus according to the embodiment of the present invention operates to determine a user's behavior from a plurality of images acquired in time series, and to provide support in a form that matches the intention of the behavior. A plurality of images acquired in time series are called "time series images". A time-series image may be, for example, a moving image. Also, each of the plurality of images acquired in time series may be frame images of a moving image acquired at regular time intervals.

<<Configuration of Medical Image Processing Apparatus According to First Embodiment>>
Processor device 16 is an example of a medical image processing device according to an embodiment of the present invention. FIG. 4 is a block diagram showing functions of the medical image processing apparatus according to the first embodiment. The medical image processing apparatus 160 shown in FIG. , provided.

The functions of the image acquisition unit 162, the availability determination unit 164, the motion estimation unit 166, the gesture determination unit 168, and the classification unit 170 can be realized by the digital signal processing circuit 76 described with reference to FIG. 3, the CPU 70, or a combination thereof. Also, the notification control unit 172 may include the display control circuit 78 described with reference to FIG.

The image acquisition unit 162 shown in FIG. 4 is an interface that acquires time-series images IM1, IM2, IM3, . The image acquisition unit 162 may be, for example, a connector terminal to which the video connector 42A is connected, or may be a signal input terminal of the digital signal processing circuit 76 . The image acquisition unit 162 may be a communication network terminal provided in the processor device 16, a media interface terminal for external storage media, a connection terminal for external equipment, or an appropriate combination thereof.

The medical image processing apparatus 160 automatically classifies lesions from images acquired in time series via the electronic endoscope 12, and notifies the user of the classification results to assist the user's diagnosis. However, images acquired during endoscopy may include images that are difficult to classify, as illustrated in FIGS.

FIG. 5 is an example of an image showing no lesion. FIG. 6 is an example of an image in which a lesion 182 is blurred due to defocus. FIG. 7 is an example of an image in which the lesion 182 is blurred due to motion blur. FIG. 7 is an example of a "blurred image".

The propriety determination unit 164 determines whether or not the image acquired from the image acquisition unit 162 is an image unsuitable for recognition. The availability determination unit 164 includes a recognition unit 164A. The recognition unit 164A recognizes whether the input image is suitable for image recognition or not. Here, "appropriate for image recognition" means that the image is suitable for recognition processing such as classification of lesions, which is the main purpose of recognition. “Unsuitable for image recognition” means that the image is unsuitable for recognition, such as classification of lesions, which is the main purpose. Examples of images unsuitable for recognition include, for example, a blurred image, a blurred image, an image in which the subject is covered with water, and an image in which the subject has residue. An image unsuitable for recognition is called an "inappropriate image".

164 A of recognition parts can be comprised using a convolutional neural network (CNN:Convolutional Neural Network), for example. The recognition unit 164A is configured using a first trained model learned by machine learning so as to perform two classification tasks, for example, whether an image is suitable for recognition or an image is unsuitable for recognition. . The recognition unit 164A can calculate the feature amount of each frame image of the moving image or the thinned-out frame images at regular intervals, and use the calculated feature amount to determine whether the image is unsuitable for recognition. .

An image unsuitable for recognition is an image that is difficult to be classified by the classification processing of the classification section 170 . An image unsuitable for recognition is also an image unsuitable for motion estimation processing in the motion estimation unit 166 .

Therefore, when the possibility determination unit 164 determines that the classification is difficult, the processing by the motion estimation unit 166, the motion determination unit 168, and the classification unit 170 is skipped, and the processing by the notification control unit 172 is performed.

The motion estimation unit 166 detects motion information of a group of images acquired in time series. Examples of methods for estimating image motion include optical flow, block matching, template matching, and the like. The method of estimating motion is not limited to the methods exemplified here, and various algorithms can be applied.

Also, the number of images used when estimating motion is not limited to two (the current image and the previous past image). FIG. 8 shows an example of a group of three images obtained in time series. FIG. 8 shows frame images of a moving image captured at regular time intervals. By performing image analysis on each of the three frame images shown in FIG. 8, it is possible to classify each image.

However, for example, if the image IM(t) at the current time t contains a bubble 184 or the like and the image is covered with the bubble 184 or the like, then the current image IM(t) and the previous past image IM It is difficult to accurately estimate the motion with only two images (t-1). Therefore, it is preferable to improve the accuracy of motion estimation by selecting two or more frame images including a past image or a future image from the current image IM(t).

In addition, the image state information recognized by the propriety determination unit 164 may be used for motion estimation by the motion estimation unit 166 . For example, when the input image is blurred, the template itself for template matching may be converted into a blurred image. Further, for example, when the input image is blurred, a process of removing the blurring component in the target image for motion estimation may be performed.

The gesture determination unit 168 determines the gesture of the user according to the motion information detected by the motion estimation unit 166 . 9 and 10 show examples of determination of user actions based on motion estimation results. FIG. 9 shows an example of an image when the motion of the image is small. The arrow A displayed in the observed image in FIG. 9 represents the motion vector of the area of the lesion 182 between the image IM(t) and the image IM(t+1) of the next frame.

FIG. 10 shows an example of an image when the motion of the image is large. The arrow B displayed in the observed image in FIG. 10 represents the motion vector of the area of the lesion 182 between the image IM(t) and the image IM(t+1) of the next frame. For example, as shown in FIG. 9, when the motion vector is small, it can be determined that the position of the scope has moved during user discrimination. On the other hand, as shown in FIG. 10, when the motion vector is large, it can be determined that the user has no intention of discrimination.

The behavior determination unit 168 determines the behavior of the user by comparing the magnitude of the motion vector with a threshold value. The magnitude of the motion vector reflects the amount of movement of the image. In other words, it is possible to determine whether or not the user's behavior is intended for discrimination from the magnitude of the motion vector. Note that the motion vector may be acquired as the motion of the entire image, or may be acquired as the motion of only a given region of interest.

In addition, the behavior determination unit 168 determines the behavior of the user not only based on the amount of movement of the image, but also in comparison with a predefined database or a database that is updated in chronological order. You may

FIG. 11 is an example of a database that defines the correspondence between motion information and user's actions. The medical image processing apparatus 160 may include a storage unit 169 that stores a database 167. FIG. The storage unit 169 may be a storage device exemplified by a semiconductor memory, a hard disk drive, a solid state drive, an optical disk, or an appropriate combination thereof. The storage unit 169 may be an internal storage device built into the medical image processing apparatus 160 or an external storage device connected to the medical image processing apparatus 160 .

The motion determining unit 168 can determine the intention of the user's motion by comparing the motion information obtained from the motion estimating unit 166 with the database 167 . For example, when the motion of the image indicated by the motion information has stopped, or when the motion is very small, it is determined that discrimination is in progress. Further, when the motion of the image indicated by the motion information is uniform motion or rotational motion, it is determined that discrimination is in progress. If the motion of the image indicated by the motion information is uniformly accelerated motion, it is determined that screening is in progress. The determination result of the behavior determination unit 168 is sent to the notification control unit 172 (see FIG. 4). The determination result of the behavior determination unit 168 corresponds to behavior information that estimates the user's behavior intention.

The classification unit 170 shown in FIG. 4 performs classification processing into a specific class for images that have been determined to be classifiable by the propriety determination unit 164 . Classification methods include, for example, the following classification classes.

<Specific examples of classification classes>
[Example 1] Two classifications can be adopted: neoplastic and non-neoplastic.

[Example 2] Endoscopic findings classification, specifically NICE classification, JNET classification, or the like can be adopted.

[Example 3] Classification by disease type, for example, classification into hyperplastic polyp, adenoma, intramucosal carcinoma, highly invasive carcinoma, inflammatory polyp, etc., can be adopted.

"NICE" is an abbreviation for NBI (Narrow band imaging) International Colorectal Endoscopic. "JNET" is an abbreviation for "the Japan NBI Expert Team."

The NICE classification is a classification by non-magnified NBI, and is classified into Type 1, Type 2 and Type 3 for each of the three items of lesion color (Color), microvessel pattern (Vessels), and surface pattern (Surface pattern). be. Type 1 is a hyperplastic lesion, Type 2 is an adenoma to intramucosal carcinoma, and Type 3 is a diagnostic index for SM (submucosa) deep invasive cancer. The JNET classification is a classification of NBI magnified endoscopy findings for colon tumors. The JNET classification is classified into Type1, Type2A, Type2B, and Type3 for each item of "vessel pattern" and "surface pattern."

The classification unit 170 may simply recognize two classifications of “cancerous” and “non-cancerous” instead of detailed classification such as the NICE classification, or in combination with this.

For example, a convolutional neural network (CNN) is used for the classification processing of the classification unit 170 shown in FIG. The classification unit 170 is configured using a second trained model learned by machine learning so as to perform the image classification task of classifying images into specific classes as exemplified in [Example 1] to [Example 3]. be able to.

When the determination result obtained from the propriety determination unit 164 is "an image suitable for recognition", the classification process by the classification unit 170 is executed. The classification unit 170 extracts feature amounts from images and classifies the images. The classification unit 170 may perform detection of a region of interest (for example, a lesion region), detection of a lesion region, and/or segmentation based on the calculated feature amount. Further, the classification unit 170 may perform classification processing using the feature amount calculated by the recognition unit 164A.

The notification control unit 172 controls the operation of the notification unit 174 based on the determination result obtained from the propriety determination unit 164, the determination result obtained from the behavior determination unit 168, and the classification result obtained from the classification unit 170. . The notification unit 174 may be configured including the display device 18 (see FIGS. 1 and 2). Also, the notification unit 174 may be a display device different from the display device 18 . The notification control section 172 can control the content of information output to the notification section 174 .

FIG. 12 is a diagram showing an example of notification of classification results obtained from the classification unit 170. As shown in FIG. In the example of FIG. 12, a polyp, which is a lesion 182, is detected in the observation image, and "result: neoplasm" (tumor) is displayed on the right side of the display area of the endoscopic image as notification information indicating the classification result. character information is displayed. The notification information indicating the classification result is preferably displayed on the same screen together with the endoscope-captured image.

On the other hand, even if the behavior determination unit 168 determines that the user has no intention of discrimination, reporting the automatic classification result or the like may impair the user's attention, hinder observation, or cause another lesion to be overlooked. are likely to be connected. Therefore, when the action determination unit 168 determines that the user does not intend to discriminate, the notification control unit 172 performs notification control so as to perform a display as shown in FIG. 13 or 14 .

FIG. 13 shows an example in which classification results are not reported. In the example shown in FIG. 13, the information "result: -" is displayed on the right side of the display area of the endoscopically captured image as information clearly indicating that the classification result is not reported.

FIG. 14 shows an example of notification in which information indicating that the classification process has not started is presented instead of character information indicating the classification result. In the example shown in FIG. 14, the information "classification off" is displayed to the right of the display area of the endoscopic image as information indicating that the classification process is not activated, and the characters "on" are disabled. It is displayed or grayed out. Instead of the notification form as shown in FIG. 13, a notification form as shown in FIG. 14 may be employed. Further, a notification form that is a combination of the notification form shown in FIG. 13 and the notification form shown in FIG. 14 may be adopted.

<<Example of Operation of Medical Image Processing Apparatus According to First Embodiment>>
A medical image processing method using the medical image processing apparatus 160 will be described. FIG. 15 is a flow chart showing an example of the operation of the medical image processing apparatus 160 according to the first embodiment. The operation of the medical imaging device may be understood as the way the medical imaging device operates, or it may be understood as the way the processor device operates.

In step S<b>11 , the medical image processing apparatus 160 receives the current image via the image acquisition section 162 . The image acquired by the image acquiring unit 162 is a medical image including a subject image captured using the electronic endoscope 12, and is one image of time-series images sequentially captured in time series. For example, the image is a one-frame image forming a moving image. Step S11 is an example of the "image acquisition step" in the present disclosure.

In step S12, the propriety determination unit 164 performs processing for recognizing the state of the input current image. Specifically, the propriety determination unit 164 performs processing for recognizing whether or not the image is suitable for classification processing.

In step S14, the propriety determination unit 164 determines whether or not classification is possible. Steps S12 and S14 are an example of the "possibility determination step" in the present disclosure. In the judgment process of step S14, when the possibility judgment unit 164 judges that classification is possible, the process proceeds to steps S16 and S20.

In step S16, the motion estimator 166 estimates motion of images using two or more images including past or future images. Step S16 is an example of the "motion estimation step" in the present disclosure.

After step S<b>16 , in step S<b>18 , the behavior determining unit 168 determines the user's behavior based on the motion information that is the estimation result of the motion estimating unit 166 . Step S18 is an example of a "behavior determination step" in the present disclosure.

Further, in parallel with the processes of steps S16 and S18, in step S20, the classification unit 170 performs a process of recognizing a lesion area from within the image and classifying it into a predetermined class. Step S20 is an example of the "classification step" in the present disclosure.

After steps S18 and S20, the process proceeds to step S22. If the determination result of step S14 is "No", that is, if it is determined that the classification is not possible, the processing from step S16 to step S20 is omitted, and the process proceeds to step S22.

In step S<b>22 , the notification control unit 172 sets notification information to be output to the notification unit 174 . The notification control unit 172 sets notification information to notify the classification result of the classification unit 170 when the determination result of the behavior determination unit 168 corresponds to the behavior of “discrimination” (see FIG. 12). On the other hand, the notification control unit 172 sets the notification information so that the classification result of the classification unit 170 is not notified when the determination result of the behavior determination unit 168 does not correspond to the behavior of “discrimination” (Fig. 13 and FIG. 14). Step S22 is an example of the "notification control step" in the present disclosure.

After step S22, the flow chart of FIG. 15 ends. The processing of the flowchart shown in FIG. 15 is repeated for each image acquired in time series.

Note that the processing of the flowchart shown in FIG. 15 is not limited to being performed for each image acquired in time series, and may be performed only for some of the images acquired in time series.

<<Advantages of the First Embodiment>>
The configuration of the first embodiment described above has the following advantages.

(1) According to the medical image processing apparatus 160, the intention of the user's gesture is determined from a plurality of images obtained in time series, and the classification result is reported for the image determined to have the intention of discrimination, while the intention of discrimination It is possible to set the classification result to non-notification for an image determined to have no image. As a result, excessive notification at unnecessary timing can be suppressed, and appropriate information can be provided at the timing when support is required.

(2) According to the medical image processing apparatus 160, the user's attention required for observation can be sustained without hindering the user's observation of the image. According to the medical image processing apparatus 160, it is possible to suppress the oversight of a region of interest such as a lesion region.

(3) According to the medical image processing device 160, when an image that is unsuitable for recognition is temporarily included in the images obtained in time series, motion estimation processing and behavior determination for the unsuitable image are performed. and the classification process are omitted, and these processes are not performed. Therefore, it is possible to avoid generating a classification result (erroneous recognition result) with low truthfulness from an image unsuitable for recognition and providing it to the user. In addition, it is possible to avoid notification of classification results for images that are unsuitable for recognition, and to provide accurate classification result information among time-series image groups.

(4) According to the medical image processing apparatus 160, it is possible to notify the user of the classification result in real time while displaying moving images obtained by continuous imaging during examination.

(5) According to the medical image processing apparatus 160, it is possible to provide useful diagnostic support for users such as doctors.

<<Second Embodiment: Example of Application to Medical Information Management System>>
The medical image processing apparatus according to the present invention is not limited to being applied to the processor device 16 of the endoscope system 10 illustrated in FIG. 1, and various applications are possible. For example, the medical image processing apparatus can be applied to a medical information management system that manages various medical information including endoscopic images.

FIG. 16 is a block diagram showing a configuration example of a medical information management system. The medical information management system 200 includes an image acquisition terminal 202 , an image storage server 204 , an information management device 210 , a display device 218 and an input device 219 . Each of the image capture terminal 202 , the image storage server 204 and the information management device 210 is connected to an electric communication line 230 . The term "connection" is not limited to wired connections, but also includes the concept of wireless connections.

Telecommunications line 230 may be a local area network or a wide area network. The electric communication line 230 is configured by an appropriate combination of wired and wireless.

The processor unit 16 of the endoscope system 10 is connected to the telecommunications line 230 . Medical images generated by processor device 16 are captured by at least one of image capture terminal 202 , image storage server 204 , and information management device 210 via telecommunications line 230 . For example, medical images generated by processor unit 16 are sent to image capture terminal 202 . Image capture terminal 202 receives medical images from processor unit 16 .

Image capture terminal 202 sends medical images received from processor unit 16 to image storage server 204 . The image capture terminal 202 may also transmit medical images received from the processor device 16 to the information management device 210 .

The image storage server 204 serves as a storage device that stores databases of various medical images. A cloud storage may be used instead of the image storage server 204 . In addition to the medical images, the image storage server 204 may also store image analysis results such as regions of interest (regions of interest) included in the medical images, presence or absence of a target of interest, and image classification results.

Although one endoscope system 10 is shown in FIG. 16 , a plurality of endoscope systems can be connected to the electrical communication line 230 . In addition, the electric communication line 230 may be connected to other medical imaging devices such as an ultrasonic diagnostic device as well as the endoscope system. An ultrasound image obtained from an ultrasound diagnostic apparatus is an example of a "medical image." The medical imaging device may be, for example, one or a combination of an X-ray imager, a CT imager, a magnetic resonance imaging (MRI) imager, a nuclear medicine diagnostic device, or a fundus camera.

The information management device 210 is realized by computer hardware and software, for example. A display device 218 and an input device 219 are connected to the information management device 210 . The information management device 210 may include some or all of the functions of the medical image processing device 160 shown in FIG. For example, the information management device 210 includes the functions of the image acquisition unit 162, availability determination unit 164, motion estimation unit 166, gesture determination unit 168, classification unit 170, and notification control unit 172 shown in FIG.

The display device 218 shown in FIG. 16 can function as the notification unit 174 . The functions of the information management device 210 can be realized by one or more computers, and can also be realized by cloud computing. The information management device 210 may include the functions of the image storage server 204 . Also, the image capture terminal 202 can function as the image acquisition unit 162 . A configuration in which the image capture terminal 202 is omitted is also possible, in which case the image storage server 204 and/or the information management device 210 are configured to capture medical images from the processor device 16 .

According to the medical information management system 200 according to the present embodiment, not only medical images obtained in real time from the electronic endoscope 12 but also moving images stored in the image storage server 204 can be reproduced as shown in FIG. can perform the processing of the flowchart described in .

The information management device 210 may be installed, for example, in an operating room, an examination room, or a conference room in a hospital, or may be installed in an out-of-hospital medical institution, research institution, or the like. The information management device 210 may be a workstation that supports examination, treatment, diagnosis, etc., or may be a business support device that supports medical work. The work support device may have a function of accumulating clinical information, assisting in the creation of diagnostic documents, assisting in the creation of reports, and the like. The information management device 210 is an example of a “medical image processing device” in the present disclosure. The medical information management system 200 is an example of a “diagnosis support device” in the present disclosure.

<<Hardware configuration of each processing unit and control unit>>
A processing unit that executes various processes such as the image acquisition unit 162, the availability determination unit 164, the motion estimation unit 166, the gesture determination unit 168, the classification unit 170, and the notification control unit 172 of the medical image processing apparatus 160 described in FIG. The hardware structure of the (processing unit) is various processors as shown below.

The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and an FPGA (Field Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc. Specially designed to execute specific processing. A dedicated electric circuit, which is a processor having circuitry, and the like are included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be configured by a plurality of FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU. Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with a single processor, first, as represented by a computer such as a client or a server, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

<<Modification 1>>
The electronic endoscope is not limited to a flexible endoscope, and may be a rigid endoscope or a capsule endoscope. Further, an apparatus that generates time-series medical images including subject images is not limited to an electronic endoscope, and may be, for example, an ultrasonic diagnostic apparatus.

<<Modification 2>>
The medical image processing apparatus of the present disclosure can be used as a diagnosis support apparatus that supports examination, treatment, diagnosis, or the like by a doctor or the like. The term "diagnostic aid" includes the concepts of diagnostic aid and/or therapeutic aid.

《About the observation light of the endoscope system》
As the observation light, light in various wavelength bands such as white light, light in one or more specific wavelength bands, or a combination thereof is selected according to the purpose of observation. White light is light in a white wavelength band or light in multiple wavelength bands. A "specific wavelength band" is a band narrower than the white wavelength band. Specific examples for specific wavelength bands are shown below.

<First example>
A first example of a specific wavelength band is, for example, the blue band or the green band in the visible range. The wavelength band of this first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example is within the wavelength band of 390 nm or more and 450 nm or less or a wavelength of 530 nm or more and 550 nm or less. It has a peak wavelength within the band.

<Second example>
A second example of a specific wavelength band is, for example, the visible red band. The wavelength band of this second example includes a wavelength band of 585 nm or more and 615 nm or less or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example is within the wavelength band of 585 nm or more and 615 nm or less or a wavelength of 610 nm or more and 730 nm or less. It has a peak wavelength within the band.

<Third example>
A third example of the specific wavelength band includes a wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the third example has a peak wavelength in the wavelength band in which oxygenated hemoglobin and reduced hemoglobin have different absorption coefficients. have The wavelength band of this third example includes a wavelength band of 400 ± 10 nm, a wavelength band of 440 ± 10 nm, a wavelength band of 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is the above 400 ± 10 nm , 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

<Fourth example>
A fourth example of the specific wavelength band is a wavelength band of excitation light that is used for observing fluorescence emitted by a fluorescent substance in vivo (fluorescence observation) and that excites this fluorescent substance, for example, from 390 nm to 470 nm.

<Fifth example>
A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of this fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength of 905 nm or more and 970 nm or less. It has a peak wavelength within the band.

《Switching observation light》
A laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof can be used as the light source. The type and wavelength of the light source, the presence or absence of a filter, etc. are preferably configured according to the type of subject and the purpose of observation. and/or preferably switch. When switching the wavelength, for example, by rotating a disc-shaped filter (rotary color filter) that is placed in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of the irradiated light is switched. good too.

The image pickup device used in the electronic endoscope is not limited to a color image pickup device in which a color filter is arranged for each pixel, and may be a monochrome image pickup device. When a monochrome imaging device is used, it is possible to sequentially switch the wavelength of illumination light and perform frame sequential (color sequential) imaging. For example, the wavelength of emitted illumination light may be sequentially switched between violet, blue, green, and red, or broadband light (white light) may be emitted and emitted by a rotary color filter (red, green, blue, etc.). The wavelength of illumination light may be switched. Alternatively, one or a plurality of narrow band lights may be applied and the wavelength of the illumination light emitted by the rotary color filter may be switched. The narrowband light may be infrared light having two or more wavelengths different from each other.

<<Generation example of special light image>>
The processor device 16 may generate a special light image having information of a specific wavelength band based on a normal light image obtained by imaging using white light. Note that the generation here includes the concept of “acquisition”. In this case, the processor device 16 functions as a special light image acquisition unit. The processor unit 16 converts signals in specific wavelength bands into red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y), which are normally included in an optical image. ) can be obtained by performing an operation based on the color information.

《Generation example of feature quantity image》
The processor device 16 produces, as medical images, a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band, and a special light image obtained by irradiating light in a specific wavelength band. may be used to generate the feature image. A feature image is a form of medical image.

《About the program that realizes the functions of a medical image processing device on a computer》
A program that causes a computer to realize the functions of the medical image processing apparatus described in the above embodiments is recorded on a computer-readable medium that is a non-temporary information storage medium that is an optical disk, magnetic disk, semiconductor memory, or other tangible object, and this It is possible to provide the program through an information storage medium. Instead of storing the program in a tangible non-temporary information storage medium and providing it, it is also possible to provide the program signal as a download service using an electric communication line such as the Internet.

It is also possible to provide a part or all of the functions of the medical image processing apparatus described in the above embodiment as an application server, and provide a service of providing processing functions through an electric communication line.

<<Combination of Embodiments and Modifications>>
The components described in the above embodiments and the components described in the modified examples can be used in combination as appropriate, and some of the components can be replaced.

<<Note>>
In addition to the embodiments and modifications described above, the present specification includes disclosure of the invention described below.

(Appendix 1)
The medical image processing apparatus includes a medical image analysis processing unit and a medical image analysis result acquisition unit. A medical image processing apparatus that detects an area and a medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit.

The medical image analysis processing section may include an image recognition section.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of an object of interest based on the pixel feature amount of the medical image, and the medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit Medical image processing. Device.

(Appendix 3)
The medical image analysis result acquisition unit acquires the analysis result of the medical image from a recording device that records the analysis result. , or both.

(Appendix 4)
A medical image processing apparatus in which the medical image is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band.

(Appendix 5)
A medical image processing apparatus in which a medical image is an image obtained by irradiating light of a specific wavelength band, and the specific wavelength band is a narrower band than the wavelength band of white.

(Appendix 6)
A medical image processing device in which the specific wavelength band is a visible blue or green band.

(Appendix 7)
The specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)
A medical image processing device in which the specific wavelength band is the visible red band.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which oxyhemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. Medical imaging equipment.

(Appendix 11)
The specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light in the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ± A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or from 600 nm to 750 nm.

(Appendix 12)
A medical image is an in-vivo image of the inside of a living body, and the in-vivo image is a medical image processing apparatus having information on fluorescence emitted by fluorescent substances in the living body.

(Appendix 13)
Fluorescence is a medical image processing apparatus obtained by irradiating the living body with excitation light having a peak of 390 nm or more and 470 nm or less.

(Appendix 14)
A medical image processing apparatus in which the medical image is an in vivo image of the inside of a living body, and the specific wavelength band is the wavelength band of infrared light.

(Appendix 15)
The specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. processing equipment.

(Appendix 16)
The medical image acquisition unit acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating light of a plurality of wavelength bands as light of a white band or light of a white band. A medical image processing apparatus comprising an optical image acquisition unit, wherein the medical image is a special optical image.

(Appendix 17)
A medical image processing apparatus in which a signal in a specific wavelength band is obtained by calculation based on RGB or CMY color information normally included in an optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light of a plurality of wavelength bands as white band light or light of a white band, and a special light image obtained by irradiating light of a specific wavelength band, A medical image processing apparatus comprising a feature amount image generating unit for generating a feature amount image, wherein the medical image is the feature amount image.

(Appendix 19)
The medical image processing apparatus according to any one of appendices 1 to 18, and an endoscope that acquires an image by irradiating at least one of light in a white wavelength band and light in a specific wavelength band , an endoscopic device comprising:

(Appendix 20)
A diagnosis support apparatus comprising the medical image processing apparatus according to any one of appendices 1 to 18.

(Appendix 21)
A medical service support apparatus comprising the medical image processing apparatus according to any one of appendices 1 to 18.

[others]
In the embodiments of the present invention described above, it is possible to appropriately change, add, or delete constituent elements without departing from the gist of the present invention. The present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention by those having ordinary knowledge in equivalent and related fields.

10 endoscope system 12 electronic endoscope 14 light source device 16 processor device 18 display device 19 input device 20 insertion section 22 distal end rigid section 22A distal end surface 24 bending section 26 flexible section 30 operating section 31 angle knob 32 air/water supply button 33 Suction button 34 Mode changeover switch 35 Zoom operation unit 36 Treatment instrument introduction port 40 Universal cord 42 Connector 42A Video connector 42B Light guide connector 42C Water supply connector 44 Water supply tank 50 Illumination window 52 Observation window 54 Forceps outlet 56 Air/water supply nozzle 60 Objective Optical system 62 Image sensor 64 Analog front end circuit 65 Timing generator 66 CPU
68 imaging chip 70 CPU
72 ROMs
74 RAM
76 digital signal processing circuit 78 display control circuit 100 first light source 101 first light source drive circuit 102 second light source 103 second light source drive circuit 105 combiner 120 light guide 121 incident end 122 exit end 124 phosphor 160 medical image processing apparatus 162 Image acquisition unit 164 Availability determination unit 164A Recognition unit 166 Motion estimation unit 167 Database 168 Action determination unit 169 Storage unit 170 Classification unit 172 Notification control unit 174 Notification unit 182 Lesion 184 Bubble 200 Medical information management system 202 Image capture terminal 204 Image Storage server 210 Information management device 218 Display device 219 Input device 230 Electric communication lines IM1, IM2, IM3 Image IM(t) Current image IM(t−1) Past image IM(t+1) Next frame images S11 to S22 Medical Processing steps in the image processing device

Claims (19)

an image acquisition unit that acquires a plurality of time-series frame images including a subject image;
a determination unit for determining whether the frame image obtained from the image acquisition unit is an image unsuitable for recognition;
a motion estimation unit that performs motion estimation from two or more frame images obtained from the image acquisition unit;
a motion determination unit that determines a user's motion based on the motion information obtained from the motion estimation unit;
a classification unit that recognizes the frame image obtained from the image acquisition unit and performs classification processing;
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit;
with
The behavior includes the behavior of identification,
the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are executed in parallel,
The notification control unit sets the classification result of the classification unit as non-notification for frame images for which the determination result of the behavior determination unit does not correspond to the behavior for discrimination,
With respect to the frame image determined by the propriety determination unit to be the inappropriate image, the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are not performed ,
When the classification process is not performed, the notification control unit displays information notifying that the classification process is not started.
Medical imaging equipment. 2. The medical image processing apparatus according to claim 1, wherein the inappropriate image is an image in which the subject is blurred. 2. The medical image processing apparatus according to claim 1, wherein the inappropriate image is an image in which a subject is blurred. 2. The medical image processing apparatus according to claim 1, wherein the inappropriate image is an image in which the subject is covered with water. 2. The medical image processing apparatus according to claim 1, wherein the inappropriate image is an image in which a subject has residue. The medical image processing according to any one of claims 1 to 5, wherein the propriety determination unit includes a recognition unit that recognizes whether the frame image obtained from the image acquisition unit is an image unsuitable for recognition. Device. the motion information includes motion vectors;
The medical image processing apparatus according to any one of claims 1 to 6 , wherein the behavior determination unit determines the behavior of the user based on the magnitude of the motion vector. The medical image processing apparatus according to any one of claims 1 to 7 , wherein the behavior determining unit determines the behavior using a database that defines a correspondence relationship between the movement information and user's behavior. 9. The medical image processing apparatus according to claim 8 , further comprising a storage unit for storing said database. The medical image processing apparatus according to any one of claims 1 to 9 , wherein the classification unit recognizes a lesion area from the frame image obtained from the image acquisition unit, and classifies the lesion into classes. The medical image processing apparatus according to any one of claims 1 to 10 , wherein the classifier is configured using a convolutional neural network. The medical image processing apparatus according to any one of claims 1 to 11 , further comprising a notification unit that notifies the classification result of the classification unit under the control of the notification control unit. 13. The medical image processing apparatus according to any one of claims 1 to 12 , wherein each of the plurality of time-series frame images is an endoscopic image captured using an electronic endoscope. A method of operating a medical imaging device comprising one or more processors, comprising:
an image acquisition step in which the one or more processors acquire a plurality of time-series frame images including a subject image;
a propriety determination step in which the one or more processors determine whether or not the frame image obtained from the image acquisition step is an image unsuitable for recognition;
a motion estimation step in which the one or more processors perform motion estimation from two or more frame images obtained from the image acquisition step;
a gesture determination step in which the one or more processors determine a user's gesture based on the motion information obtained from the motion estimation step;
a classification step in which the one or more processors recognize and classify the frame images obtained from the image acquisition step;
a notification control step in which the one or more processors control notification information based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step;
including
The behavior includes the behavior of identification,
the processing in the motion estimation step and the gesture determination step and the classification processing in the classification step are performed in parallel,
The notification control step sets the classification result of the classification step as non-notification for frame images for which the determination result of the behavior determination step does not correspond to the behavior of the discrimination,
With respect to the frame image determined to be the inappropriate image in the propriety determination step, the processing in the motion estimation step and the movement determination step and the classification processing in the classification step are not performed ,
When the classification process is not performed, the notification control step displays information notifying that the classification process is not activated.
A method of operating a medical imaging device . an electronic endoscope for imaging inside body cavities;
and a processor for processing image signals obtained from the electronic endoscope,
The processor device
an image acquisition unit configured to acquire a plurality of time-series frame images including a subject image captured using the electronic endoscope;
a determination unit for determining whether the frame image obtained from the image acquisition unit is an image unsuitable for recognition;
a motion estimation unit that estimates motion from two or more frame images obtained from the image acquisition unit; a behavior determination unit that determines a user's behavior based on the motion information obtained from the motion estimation unit; a classification unit that recognizes frame images obtained from the image acquisition unit and performs classification processing;
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit;
with
The behavior includes the behavior of identification,
the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are executed in parallel,
The notification control unit sets the classification result of the classification unit as non-notification for frame images for which the determination result of the behavior determination unit does not correspond to the behavior for discrimination,
With respect to the frame image determined by the propriety determination unit to be the inappropriate image, the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are not performed ,
When the classification process is not performed, the notification control unit displays information notifying that the classification process is not started.
endoscope system. A processor device for processing an image signal obtained from an electronic endoscope,
an image acquisition unit configured to acquire a plurality of time-series frame images including a subject image captured using the electronic endoscope;
a determination unit for determining whether the frame image obtained from the image acquisition unit is an image unsuitable for recognition;
a motion estimation unit that performs motion estimation from two or more frame images obtained from the image acquisition unit;
a motion determination unit that determines a user's motion based on the motion information obtained from the motion estimation unit;
a classification unit that recognizes the frame image obtained from the image acquisition unit and performs classification processing;
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit;
with
The behavior includes the behavior of identification,
the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are executed in parallel,
The notification control unit sets the classification result of the classification unit as non-notification for frame images for which the determination result of the behavior determination unit does not correspond to the behavior for discrimination,
With respect to the frame image determined by the propriety determination unit to be the inappropriate image, the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are not performed ,
When the classification process is not performed, the notification control unit displays information notifying that the classification process is not started.
processor unit. an image acquisition unit that acquires a plurality of time-series frame images including a subject image;
a determination unit for determining whether the frame image obtained from the image acquisition unit is an image unsuitable for recognition;
a motion estimation unit that performs motion estimation from two or more frame images obtained from the image acquisition unit;
a motion determination unit that determines a user's motion based on the motion information obtained from the motion estimation unit;
a classification unit that recognizes the frame image obtained from the image acquisition unit and performs classification processing;
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit;
a notification unit that notifies information including the classification result of the classification unit under the control of the notification control unit;
with
The behavior includes the behavior of identification,
the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are executed in parallel,
The notification control unit sets the classification result of the classification unit as non-notification for frame images for which the determination result of the behavior determination unit does not correspond to the behavior for discrimination,
With respect to the frame image determined by the propriety determination unit to be the inappropriate image, the processing in the motion estimation unit and the movement determination unit and the classification processing in the classification unit are not performed ,
When the classification process is not performed, the notification control unit displays information notifying that the classification process is not started.
Diagnostic support device. to the computer,
an image acquisition step of acquiring a plurality of time-series frame images including a subject image;
a propriety determination step of determining whether or not the frame image obtained from the image acquisition step is an image unsuitable for recognition;
A motion estimation step of estimating motion from two or more frame images obtained from the image acquisition step;
a motion determination step of determining a user's motion based on the motion information obtained from the motion estimation step;
a classification step of recognizing and classifying the frame image obtained from the image acquisition step;
a notification control step of controlling notification information based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step;
and
The behavior includes the behavior of identification,
the processing in the motion estimation step and the gesture determination step, and the classification processing in the classification step are performed in parallel,
The notification control step sets the classification result of the classification step as non-notification for frame images for which the determination result of the behavior determination step does not correspond to the behavior of the discrimination,
With respect to the frame image determined to be the inappropriate image in the propriety determination step, the processing in the motion estimation step and the movement determination step and the classification processing in the classification step are not performed ,
When the classification process is not performed, the notification control step displays information notifying that the classification process is not started.
program. A non-transitory computer-readable storage medium, wherein when instructions stored on the storage medium are read by a computer,
an image acquisition step of acquiring a plurality of time-series frame images including a subject image;
a propriety determination step of determining whether or not the frame image obtained from the image acquisition step is an image unsuitable for recognition;
A motion estimation step of estimating motion from two or more frame images obtained from the image acquisition step;
a motion determination step of determining a user's motion based on the motion information obtained from the motion estimation step;
a classification step of recognizing and classifying the frame image obtained from the image acquisition step;
a notification control step of controlling notification information based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step;
on the computer, and
The behavior includes the behavior of identification,
the processing in the motion estimation step and the gesture determination step, and the classification processing in the classification step are performed in parallel,
The notification control step sets the classification result of the classification step as non-notification for frame images for which the determination result of the behavior determination step does not correspond to the behavior of the discrimination,
With respect to the frame image determined to be the inappropriate image in the propriety determination step, the processing in the motion estimation step and the movement determination step and the classification processing in the classification step are not performed ,
When the classification process is not performed, the notification control step displays information notifying that the classification process is not started.
storage medium.

Images