JPWO2020054543A1 - Medical image processing equipment and methods, endoscopic systems, processor equipment, diagnostic support equipment and programs - Google Patents

Abstract

Medical image processing devices and methods, endoscopic systems, diagnostic support devices, and programs that can support diagnosis without disturbing the user's observation of images and without impairing the user's attention required for observation. I will provide a. The medical image processing device (160) has an image acquisition unit (162) that acquires a plurality of time-series images including a subject image, and a determination as to whether or not the obtained image is an image unsuitable for recognition. A unit (164), a motion estimation unit (166) that estimates motion from two or more images, a behavior determination unit (168) that determines a user's behavior based on motion information, and a classification process that recognizes images. A notification control unit (172) that controls notification information based on the behavior information obtained from the behavior determination unit (168) and the classification result obtained from the classification unit (170). To be equipped.

Description

The present invention relates to medical image processing devices and methods, endoscopic systems, processor devices, diagnostic support devices and programs, and in particular, image processing that provides information that supports diagnosis by processing time-series medical images. Regarding technology.

In the medical field, examinations using an endoscopic system are performed. In recent years, a system for recognizing a lesion contained in a medical image by using an image analysis technique has been known. As shown in Patent Document 1 and Patent Document 2, in an endoscopic system, a body cavity is photographed using a camera at the tip of a scope, and a processor device performs recognition processing on the obtained observation image. It may be done and the recognition result may be notified to the user. The user can take appropriate measures according to the notified recognition result.

For example, if the lesion has been imaged, the processor device recognizes the area of the lesion from the image by performing image analysis and recognizes whether it is cancerous or non-cancerous. Notify the user of the recognition result. The user can observe the lesion in more detail by referring to the notified information, and when it is determined that the lesion is cancerous, the user can take measures such as excising the lesion. A system having such a function supports decision-making such as diagnostic findings by doctors and the like.

Japanese Patent No. 6150554 International Publication No. 2017/002184 Special Fair 7-27481

In the diagnostic support function such as the endoscopic system described above, a non-cancerous lesion is mistakenly recognized as cancerous, or a cancerous lesion is mistakenly recognized as non-cancerous, resulting in an erroneous recognition result. Is a problem to inform the user. However, in endoscopy, the lesion site may be hidden or the observed image may be blurred due to the operation of the scope, fetal movement, or residue, and it is difficult to accurately recognize the lesion by image analysis. In some cases.

In such a case, for example, as described in Patent Document 3, image recognition is applied by applying a technique of an abnormality display processing method that switches from a normal image display to an abnormality message display when an abnormality is detected. It is conceivable to notify that the image is unsuitable for.

However, the operation of the endoscope is complicated, and the user needs to operate the scope and check the observed image in parallel. The notification of the recognition result by the diagnosis support function streamlines the user's discrimination work, and at the same time, the image recognition result is notified even when the user who is observing the image does not intend to discriminate the lesion, for example. On the contrary, it may impair the user's attention, lead to obstruction of observation, and may lead to oversight of a region of interest such as another lesion region.

This problem is not limited to endoscopic systems, but is a common problem for systems that process and support medical images taken in time series, such as ultrasonic diagnostic equipment.

The present invention has been made in view of such circumstances, and is a medical image capable of supporting diagnosis without hindering the user's observation of the image and without impairing the user's attention required for the observation. It is an object of the present invention to provide processing devices and methods, endoscopic systems, diagnostic support devices, and programs.

The medical image processing apparatus according to one aspect of the present disclosure includes an image acquisition unit that acquires a plurality of time-series images including a subject image, and whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition. Based on the motion estimation unit that estimates motion from two or more images obtained from the image acquisition unit and the motion estimation unit, and the motion information obtained from the motion estimation unit, the action of determining the user's behavior. The notification information is controlled based on the judgment unit, the classification unit that recognizes the image obtained from the image acquisition unit and performs the classification process, the behavior information obtained from the behavior judgment unit, and the classification result obtained from the classification unit. The notification control unit is provided.

According to the medical image processing device according to this aspect, the user's behavior is determined from the acquired image, and the classification result is notified or non-notified by image recognition in a form according to the intention of the behavior. be able to. In addition, according to this aspect, it is possible to take measures such as avoiding notification of the classification result for images that are unsuitable for recognition, and it is possible to suppress the provision of erroneous information, and the accuracy in the time-series image group. It is possible to provide information on high classification results.

The plurality of images in the time series may be moving images, or may be a group of images taken at specific time intervals such as continuous shooting or interval shooting. In addition, the time interval of shooting in chronological order does not necessarily have to be constant.

The medical image processing device may be configured as a single device, or may be configured by combining a plurality of devices. For example, a medical image processing apparatus can be realized by using one or a plurality of computers. "Device" includes the concepts of "system" and "module".

The "inappropriate image" may be, for example, an image in which the subject is blurred, an image in which the subject is blurred, an image in which the subject is covered with water, or an image in which the subject has a residue.

In the medical image processing apparatus according to another aspect of the present disclosure, the pass / fail determination unit may include a recognition unit that recognizes whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition. ..

The term "recognition" includes concepts such as identification, discrimination, reasoning, estimation, detection, classification, and region extraction. The "recognition unit" includes concepts such as a recognizer, a classifier, a discriminator, a detector, a classifier, and a recognition model. The "recognition model" may be, for example, a trained model whose recognition performance has been acquired by machine learning.

In the medical image processing apparatus according to still another aspect of the present disclosure, the motion estimation unit and the behavior determination unit may not perform the processing on the image determined by the admissibility determination unit to be an inappropriate image.

In the medical image processing apparatus according to still another aspect of the present disclosure, the classification process of the image determined by the pass / fail determination unit to be an inappropriate image may not be performed by the classification unit.

According to such an embodiment, it is possible to avoid providing an inaccurate classification result.

In the medical image processing apparatus according to still another aspect of the present disclosure, the motion information may include a motion vector, and the motion determination unit may be configured to determine the user's motion based on the magnitude of the motion vector.

In the medical image processing apparatus according to still another aspect of the present disclosure, the behavior determination unit may be configured to determine the behavior using a database in which the correspondence between the motion information and the user's behavior is defined.

The medical image processing apparatus according to still another aspect of the present disclosure may be configured to include a storage unit for storing a database.

In the medical image processing apparatus according to still another aspect of the present disclosure, the action may be configured to include a differential action.

In the medical image processing apparatus according to still another aspect of the present disclosure, the notification control unit has a configuration in which the classification result of the classification unit is not notified for an image whose judgment result of the behavior determination unit does not correspond to the behavior of discrimination. It's okay.

In the medical image processing apparatus according to still another aspect of the present disclosure, the classification unit may be configured to recognize the region of the lesion from the image obtained from the image acquisition unit and classify the lesion.

In the medical image processing apparatus according to still another aspect of the present disclosure, the classification unit may be configured by using a convolutional neural network.

The medical image processing apparatus according to still another aspect of the present disclosure may be configured to include a notification unit that notifies the classification result of the classification unit based on the control of the notification control unit.

In the medical image processing apparatus according to still another aspect of the present disclosure, each of the plurality of images in the time series may be an endoscopic image taken by using an electronic endoscope.

The medical image processing method according to still another aspect of the present disclosure includes an image acquisition step of acquiring a plurality of time-series images including a subject image, and whether the image obtained from the image acquisition step is an image unsuitable for recognition. The user's behavior is determined based on the motion estimation process that determines whether or not the motion is valid, the motion estimation process that estimates the motion from two or more images obtained from the image acquisition process, and the motion information obtained from the motion estimation process. Notification information based on the behavior determination process, the classification process that recognizes the image obtained from the image acquisition process and performs the classification process, the behavior information obtained from the behavior determination process, and the classification result obtained from the classification process. The notification control step for controlling the above is included.

In the medical image processing method of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined. In that case, the elements of the processing unit and the functional unit as means for carrying out the processing and operation specified in the medical image processing apparatus can be grasped as the elements of the corresponding processing and operation processes (steps). Further, the medical image processing method of this aspect can be understood as an operation method of the medical image processing device.

The endoscopic system according to still another aspect of the present disclosure is an endoscopic system including an electronic endoscope that images the inside of a body cavity and a processor device that processes an image signal obtained from the electronic endoscope. The processor device is an image acquisition unit that acquires a plurality of time-series images including a subject image taken by using an electronic endoscope, and an image obtained from the image acquisition unit is an image that is unsuitable for recognition. Based on the motion estimation unit that determines whether or not it is possible, the motion estimation unit that estimates motion from two or more images obtained from the image acquisition unit, and the motion information obtained from the motion estimation unit, the user's actions are taken. Notification based on the behavior judgment unit to be determined, the classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing, the behavior information obtained from the behavior judgment unit, and the classification result obtained from the classification unit. It includes a notification control unit that controls information.

In the endoscope system of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined.

The processor device according to still another aspect of the present disclosure is a processor device that processes an image signal obtained from an electronic endoscope, and is a plurality of time series including a subject image taken by the electronic endoscope. An image acquisition unit that acquires an image, a pass / fail determination unit that determines whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition, and a motion from two or more images obtained from the image acquisition unit. A motion estimation unit that performs estimation, a behavior determination unit that determines the user's behavior based on the motion information obtained from the motion estimation unit, and a classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing. And a notification control unit that controls the notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit.

In the processor device of this embodiment, the same items as those specified in each aspect of the medical image processing device described above can be appropriately combined.

The diagnostic support device according to still another aspect of the present disclosure includes an image acquisition unit that acquires a plurality of time-series images including a subject image, and whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition. The user's behavior is determined based on the motion estimation unit that estimates the motion from two or more images obtained from the image acquisition unit and the motion information obtained from the motion estimation unit. Notification information is provided based on the behavior determination unit, the classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing, the behavior information obtained from the behavior determination unit, and the classification result obtained from the classification unit. A notification control unit for controlling and a notification unit for notifying information including the classification result of the classification unit based on the control of the notification control unit are provided.

In the diagnostic support device of this aspect, the same items as those specified in each aspect of the medical image processing device described above can be appropriately combined. The display unit that displays the classification result of the classification unit can be a form of the notification unit.

The program according to still another aspect of the present disclosure includes an image acquisition step of acquiring a plurality of time-series images including a subject image on a computer, and whether the image obtained from the image acquisition step is an image unsuitable for recognition. The user's behavior is determined based on the motion estimation process that determines whether or not the motion is valid, the motion estimation process that estimates the motion from two or more images obtained from the image acquisition process, and the motion information obtained from the motion estimation process. Notification information based on the behavior determination process, the classification process that recognizes the image obtained from the image acquisition process and performs the classification process, the behavior information obtained from the behavior determination process, and the classification result obtained from the classification process. The notification control process for controlling the above is executed.

In the program of this aspect, the same items as those specified in each aspect of the medical image processing apparatus described above can be appropriately combined. In that case, the elements of the processing unit and the functional unit as means for carrying out the processing and operation specified in the medical image processing apparatus can be grasped as program elements that realize the processes or functions of the corresponding processing and operation. ..

According to the present invention, since the user's behavior is determined from the time-series images and the notification information is controlled based on the obtained behavior information, it is possible to realize support according to the user's behavior. According to the present invention, it is possible to provide useful support information such as an image classification result without hindering the user's observation of the image and without impairing the user's attention required for the observation.

FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a front view showing the tip surface of the tip rigid portion of the electronic endoscope. FIG. 3 is a block diagram showing a control system of the endoscopic system. FIG. 4 is a block diagram showing a function of the medical image processing apparatus according to the first embodiment of the present invention. FIG. 5 is an example of an image in which no lesion is shown. FIG. 6 is an example of an image in which the lesion is out of focus due to out of focus. FIG. 7 is an example of an image in which the lesion is blurred due to motion blur. FIG. 8 is an example of three image groups obtained in chronological order. FIG. 9 is an image example when the movement of the image is small. FIG. 10 is an image example when the movement of the image is large. FIG. 11 is an explanatory diagram showing an example of a database that defines the correspondence between motion information and user behavior. FIG. 12 is a diagram showing an example of notification of the classification result obtained from the classification unit. FIG. 13 is a diagram showing an example of notification when the classification result is not notified. FIG. 14 is a diagram showing an example of notification that presents information for notifying that the classification process has not been activated. FIG. 15 is a flowchart showing an example of the operation of the medical image processing apparatus according to the first embodiment. FIG. 16 is a block diagram showing a configuration example of the medical information management system according to the second embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<< Configuration example of endoscopic system >>
FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. The endoscope system 10 includes an electronic endoscope 12, a light source device 14, and a processor device 16. A display device 18 and an input device 19 are connected to the processor device 16. The electronic endoscope 12 may be referred to as a "scope", an "electronic scope", or simply an "endoscope".

The electronic endoscope 12 of this example is a flexible endoscope. The electronic endoscope 12 includes an insertion unit 20, an operation unit 30, and a universal cord 40. The insertion portion 20 is a portion to be inserted into the body cavity of the subject to be inspected. The insertion portion 20 includes a tip hard portion 22, a curved portion 24, and a soft portion 26 in order from the tip side toward the hand side. An illumination optical system, an objective optical system, an image sensor, and the like are arranged inside the hard tip portion 22. The curved portion 24 has a structure that smoothly bends in four directions of up, down, left, and right from the reference position according to the operation of the angle knob 31. The hand side of the flexible portion 26 is referred to as the base end portion of the insertion portion 20.

The operation unit 30 is provided at the base end portion of the insertion unit 20. The operation unit 30 includes various operation members operated by the operator. For example, the operation unit 30 includes an angle knob 31 used for the bending operation of the bending unit 24, an air supply water supply button 32 for performing an air supply / water supply operation, and a suction button 33 for performing a suction operation. It is provided. An operator (an example of a user) can bend the curved portion 24 and freely change the direction of the hard tip portion 22 by operating the angle knob 31. The operation unit 30 also includes a mode changeover switch 34 used for the observation mode switching operation, a zoom operation unit 35, and a still image shooting instruction unit (not shown) for giving a still image shooting instruction of the observed portion. It is provided.

Further, the operation unit 30 is provided with a treatment tool introduction port 36. The treatment tool introduction port 36 is an opening for inserting a treatment tool (not shown) into the treatment tool insertion passage (not shown) that is inserted through the insertion portion 20. Treatment tools may include, for example, biopsy forceps, catheters, high frequency snares, and the like. The treatment tool also includes a guide tube, a tracal tube, a sliding tube and the like. The treatment tool introduction port 36 may be called a forceps port.

The universal cord 40 is a cord for connecting the electronic endoscope 12 to the light source device 14 and the processor device 16. A cable and a light guide extending from the insertion portion 20 are inserted into the universal cord 40. The cable extending from the insertion portion 20 includes a communication cable used for signal transmission and a power supply cable used for power supply. A connector 42 is provided at one end of the universal cord 40.

The connector 42 is a composite type connector including a video connector 42A and a light guide connector 42B. One end of the cable is arranged on the video connector 42A. The video connector 42A is detachably connected to the processor device 16. One end of the light guide is arranged on the light guide connector 42B. The light guide connector 42B is detachably connected to the light source device 14. Further, the light guide connector 42B is provided with a water supply connector 42C, and the water supply tank 44 is connected via the water supply connector 42C.

The processor device 16 is electrically connected to the light source device 14 via the connector 42. The processor device 16 comprehensively controls the operation of the endoscope system 10 including the light source device 14. The processor device 16 supplies power to the electronic endoscope 12 via a cable inserted in the universal cord 40, and controls the drive of the image sensor. Further, the processor device 16 receives an image pickup signal transmitted from the electronic endoscope 12 via a cable, and performs various signal processing on the received image pickup signal to convert it into image data. The image data converted by the processor device 16 is displayed on the display device 18 as an endoscopically captured image (observation image).

FIG. 2 is a front view showing the tip surface 22A of the tip rigid portion 22 of the electronic endoscope 12. The tip surface 22A of the tip rigid portion 22 is provided with an illumination window 50, an observation window 52, a forceps outlet 54, and an air supply / water supply nozzle 56. Although not shown in FIG. 2, an emission end 122 of the light guide 120 that guides the light from the light source device 14 is arranged behind the illumination window 50 (see FIG. 3). Illumination light is emitted from the illumination window 50 to the observed area. Two illumination windows 50 are arranged symmetrically with respect to the observation window 52.

The observation window 52 is a window for taking in the reflected light from the observed area and taking an image of the observed area. Although not shown in FIG. 2, an objective optical system 60 and an image sensor 62 for capturing the image light of the observed region in the body cavity are arranged behind the observation window 52 (see FIG. 3). ..

The forceps outlet 54 is connected to a forceps channel (not shown) arranged in the insertion portion 20 and communicates with a treatment tool introduction port 36 (see FIG. 1) provided in the operation portion 30. The treatment tool inserted from the treatment tool introduction port 36 is taken out into the body cavity from the forceps outlet 54.

The air supply water supply nozzle 56 injects wash water or air toward the observation window 52 and / or the body cavity in response to the operation of the air supply water supply button 32 (see FIG. 1) provided in the operation unit 30. .. The washing water and air are supplied from the air supply / water supply device built in the light source device 14.

FIG. 3 is a block diagram showing a control system of the endoscope system 10. The hard tip portion 22 of the electronic endoscope 12 includes an objective optical system 60, an image sensor 62, an analog front end (AFE) circuit 64, a timing generator (TG) 65, and a CPU. (Central Processing Unit) 66 and.

The objective optical system 60 is configured by using a zoom lens. A prism for guiding light (not shown) is arranged between the objective optical system 60 and the image sensor 62. The image pickup device 62 may be a CMOS (complementary metal oxide semiconductor) type image pickup device or a CCD (Charged Coupled Device) type image pickup device. Here, an example of using a CMOS type image sensor will be described.

Although not shown, the image sensor 62 is a solid-state image sensor of a single plate color image pickup system including a color filter composed of a plurality of color segments. The color filter may be, for example, a Bayer-arranged primary color filter including red (R), green (G), and blue (B).

A large number of pixels are arranged in a matrix on the image pickup surface of the image pickup device 62, and each pixel is provided with a photo sensor. The light incident on the image pickup surface of the image pickup element 62 is stored as an electric charge in the photosensor of each pixel. The amount of signal charge accumulated in the photosensor of each pixel is sequentially read out as a pixel signal by scanning in the vertical and horizontal directions by a vertical scanning circuit and a horizontal scanning circuit (both not shown), and at a predetermined frame rate. It is output.

The timing generator 65 generates a drive pulse of the image sensor 62 and a synchronous pulse to the analog front end circuit 64 based on the control of the CPU 66. The drive pulse of the image pickup device 62 includes a vertical scan pulse, a horizontal scan pulse, a reset pulse, and the like.

The image sensor 62 is driven by a drive pulse input from the timing generator 65, photoelectrically converts an optical image imaged on the image pickup surface via the objective optical system 60, and outputs the image sensor 62 as an image pickup signal.

The analog front-end circuit 64 includes a correlated double sampling (CDS) circuit, an automatic gain control (AGC) circuit, and an A / D converter. The CDS circuit performs a correlation double sampling process on the image pickup signal output from the image pickup element 62, and removes the reset noise and the amplifier noise generated in the image pickup element 62.

The AGC circuit amplifies the image pickup signal whose noise has been removed by the CDS circuit with a gain (amplification rate) specified by the CPU 66. The A / D converter converts the imaging signal amplified by the AGC circuit into a digital signal having a predetermined number of bits and outputs the signal. The image pickup signal digitized and output by the analog front-end circuit 64 is input to the processor device 16 through the signal line.

The image sensor 62, the analog front end circuit 64, and the timing generator 65 can be configured as a monolithic integrated circuit, and each of these circuit elements is included in one image pickup chip 68. The imaging chip 68 mounted on the electronic endoscope 12 of this example is a so-called “CMOS sensor chip” and is mounted on a support substrate (not shown).

The processor device 16 includes a CPU 70, a ROM (read-only memory) 72, a RAM (Random Access Memory) 74, a digital signal processor (DSP) 76, and a display control circuit 78.

The CPU 70 controls each part in the processor device 16 and comprehensively controls the entire endoscope system 10. Various programs and control data for controlling the operation of the processor device 16 are stored in the ROM 72. The program and data executed by the CPU 70 are temporarily stored in the RAM 74.

Based on the control of the CPU 70, the digital signal processing circuit 76 performs various signal processing such as color interpolation, color separation, color balance adjustment, gamma correction, and image enhancement processing on the image pickup signal input from the analog front end circuit 64. To generate image data. Further, the digital signal processing circuit 76 performs image recognition processing. The digital signal processing circuit 76 functions as an image processing unit. Further, the digital signal processing circuit 76 includes a function of an image recognition unit that performs image recognition processing.

The image data output from the digital signal processing circuit 76 is input to the display control circuit 78. The display control circuit 78 converts the image data input from the digital signal processing circuit 76 into a signal format corresponding to the display device 18 and displays it on the screen of the display device 18.

The display device 18 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. The display device 18 can display various setting information necessary for the processing of the processor device 16 or various information such as information indicating the processing result.

The display device 18 and the input device 19 function as a user interface. The input device 19 may be, for example, a keyboard, a mouse, a touch panel, operation buttons, a voice input device, or an appropriate combination thereof. The user can input various instructions and / or information using the input device 19. The processor device 16 can execute various processes according to the instruction and / or information input from the input device 19.

The light source device 14 generates light that illuminates the inside of the body cavity through the light guide 120 inserted in the electronic endoscope 12. The light source device 14 includes a first light source 100, a first light source drive circuit 101, a second light source 102, a second light source drive circuit 103, a CPU 104, and a combiner 105. The CPU 104 communicates with the CPU 70 of the processor device 16 to control the first light source drive circuit 101 and the second light source drive circuit 103.

The first light source drive circuit 101 causes the first light source 100 to emit light according to the instruction from the CPU 104. The second light source drive circuit 103 causes the second light source 102 to emit light according to the instruction from the CPU 104.

The first light source 100 is, for example, a laser diode that emits a blue laser having a wavelength of 445 nm. The first light source 100 is pulse-driven by the first light source drive circuit 101 to control the amount of light emitted. The second light source 102 is, for example, a laser diode that emits a blue laser having a wavelength of 405 nm. The second light source 102 is pulse-driven by the second light source drive circuit 103 to control the amount of light emitted. Blue light with a wavelength of 405 nm is used for special light observation.

The combined wave unit 105 combines the emitted lights of the first light source 100 and the second light source 102 and emits them to the incident end 121 of the light guide 120.

A phosphor 124 is provided between the emission end 122 of the light guide 120 and the illumination window 50 of the electronic endoscope 12. The blue laser light that has passed through the light guide 120 irradiates the phosphor 124 with an excited state, and a part of the blue laser light passes through the phosphor 124 and is emitted from the illumination window 50 as blue light.

The phosphor 124 is excited by blue laser light and emits a wide range of light (yellow as a color) from the wavelength range per boundary between blue and green to the red wavelength range in terms of the wavelength band of light. The yellow light and the blue light transmitted through the phosphor 124 are mixed to form white light, which illuminates the subject through the illumination window 50. The blue light transmitted through the phosphor 124 includes a part of the blue light emitted by the phosphor 124.

As described above, the phosphor 124 emits yellow light when irradiated with a blue laser light having a wavelength of 445 nm and transmits blue light having a wavelength of 445 nm, but is irradiated with a blue laser light having a wavelength of 405 nm. In some cases, it has the property of transmitting most of it.

That is, by controlling the mixing ratio of the blue laser light having a wavelength of 445 nm and the blue laser light having a wavelength of 405 nm, the ratio of the blue light transmitted through the phosphor 124 and the yellow light emitted by the phosphor 124 is controlled. Is possible.

A color image of the subject is reproduced by receiving the reflected light from the subject illuminated by the white light by using the image sensor 62.

When observing the inside of the body cavity using the endoscope system 10 configured as described above, the power of the electronic endoscope 12, the light source device 14, the processor device 16, and the display device 18 is turned on. Then, the insertion portion 20 of the electronic endoscope 12 is inserted into the body cavity, and the image in the body cavity captured by the image pickup element 62 is displayed on the screen of the display device 18 while illuminating the inside of the body cavity with the illumination light from the light source device 14. Will be observed at.

The endoscope system 10 of this example has a white light observation mode and a narrow band light observation mode. The white light observation mode is a mode in which an observation image having a natural hue is displayed on the display device 18 by using a photographed image obtained by imaging an observation target using white light as illumination light. An image obtained by imaging an observation target in the white light observation mode is called a "white light observation image". Illumination light may be paraphrased as "observation light".

In the narrow-band light observation mode, for example, visualization of blood vessels in a specific depth region of an observation target is emphasized by using an image signal obtained by imaging an observation target using narrow-band light of a specific wavelength band as illumination light. This mode generates an image and displays an image suitable for observing blood vessels on the display device 18. An image obtained by imaging an observation target in the narrow band light observation mode is called a "narrow band light observation image".

The endoscope system 10 may have a plurality of types of narrow-band light observation modes in which the types of wavelength bands of narrow-band light used or combinations thereof are different.

<< Overview of diagnostic support function in endoscope system 10 >>
In endoscopy, a screening process in which a doctor who is a user operates a scope to detect a lesion and a differential process in which the state of the detected lesion is confirmed in detail are performed to prevent oversight of the lesion. Is one of the important issues. For example, when the differentiation of a certain lesion is completed, the user may forget the existence of the surrounding lesion. The user seamlessly implements the transition from the screening work to the discrimination work and the transition from the discrimination work to the screening work, and the diagnostic support system is also required to operate according to such user's operation.

As a means for switching between the screening work and the discrimination work, the timing at which the type of the light source is switched can be considered, but it cannot be dealt with when the work is shifting with the same light source.

The medical image processing apparatus according to the embodiment of the present invention operates so as to determine a user's action from a plurality of images acquired in a time series and provide support in a form according to the intention of the action. A plurality of images acquired in time series are called "time series images". The time-series image may be, for example, a moving image. Further, each of the plurality of images acquired in time series may be a frame image of a moving image obtained at regular time intervals.

<< Configuration of Medical Image Processing Device According to First Embodiment >>
The processor device 16 is an example of the medical image processing device according to the embodiment of the present invention. FIG. 4 is a block diagram showing the functions of the medical image processing apparatus according to the first embodiment. The medical image processing device 160 shown in FIG. 4 includes an image acquisition unit 162, a pass / fail determination unit 164, a motion estimation unit 166, an action determination unit 168, a classification unit 170, a notification control unit 172, and a notification unit 174. , Equipped with.

The functions of the image acquisition unit 162, the pass / fail determination unit 164, the motion estimation unit 166, the action determination unit 168, and the classification unit 170 can be realized by the digital signal processing circuit 76 described in FIG. 3, the CPU 70, or a combination thereof. Further, the notification control unit 172 may include the display control circuit 78 described with reference to FIG.

The image acquisition unit 162 shown in FIG. 4 is an interface for acquiring time-series images IM1, IM2, IM3 ... Taken by using the electronic endoscope 12. The image acquisition unit 162 may be, for example, a connector terminal to which the video connector 42A is connected, or a signal input terminal of the digital signal processing circuit 76. Further, the image acquisition unit 162 may be a communication network terminal provided in the processor device 16, a media interface terminal for external storage media, a terminal for connecting an external device, or an appropriate combination thereof.

The medical image processing device 160 automatically classifies lesions from images acquired in time series via the electronic endoscope 12, and notifies the classification result to support the diagnosis by the user. However, the images obtained in the endoscopy may include images that are difficult to classify, as illustrated in FIGS. 5-7.

FIG. 5 is an example of an image in which no lesion is shown. FIG. 6 is an example of an image in which the lesion 182 is out of focus due to out of focus. FIG. 7 is an example of an image in which the lesion 182 is blurred due to motion blur. FIG. 7 is an example of a “blurred image”.

The approval / disapproval determination unit 164 determines whether or not the image acquired from the image acquisition unit 162 is an image unsuitable for recognition. The pass / fail determination unit 164 includes a recognition unit 164A. The recognition unit 164A recognizes whether the input image is appropriate or inappropriate for image recognition. Here, "suitable for image recognition" means that the image is suitable for recognition processing for classifying lesions, which is the main purpose of recognition. "Inappropriate for image recognition" means that the image is unsuitable for recognition such as classification of lesions, which is the main purpose. Examples of images that are unsuitable for recognition include blurred images, blurred images, images in which the subject is covered with water, and images in which the subject has residues. An image that is unsuitable for recognition is called an "inappropriate image".

The recognition unit 164A can be configured by using, for example, a convolutional neural network (CNN). The recognition unit 164A is configured by using the first trained model learned by machine learning so as to perform two classification tasks, for example, an image suitable for recognition and an image unsuitable for recognition. .. The recognition unit 164A can calculate the feature amount of each frame image of the moving image or the frame image at regular intervals thinned out, and can determine whether or not the image is unsuitable for recognition by using the calculated feature amount. ..

An image that is unsuitable for recognition is an image that is difficult to classify by the classification process of the classification unit 170. Further, the image unsuitable for recognition is an image that is not suitable for the motion estimation process in the motion estimation unit 166.

Therefore, when the possibility determination unit 164 determines that the classification is difficult, the process proceeds to the notification control unit 172 without performing the processing by the motion estimation unit 166, the action determination unit 168, and the classification unit 170.

The motion estimation unit 166 detects the motion information of the image group acquired in time series. Examples of the method of estimating the motion of the image include optical flow, block matching, template matching, and the like. The method for estimating the motion is not limited to the method illustrated here, and various algorithms can be applied.

Further, the number of images used when estimating the movement is not limited to two (the current image and the previous image). FIG. 8 shows an example of three image groups obtained in chronological order. FIG. 8 shows a frame image of a moving image taken at regular time intervals. By performing image analysis on each of the three frame images shown in FIG. 8, it is possible to classify each image.

However, for example, when the bubble 184 or the like is reflected in the image IM (t) at the current time t and the image is covered by the bubble 184 or the like, the current image IM (t) and the previous past image IM are It is difficult to estimate an accurate movement only with the two images (t-1). Therefore, it is preferable to improve the motion estimation accuracy by selecting two or more frame images including the past image or the future image when viewed from the current image IM (t).

Further, the state information of the image recognized by the pass / fail determination unit 164 may be used for the motion estimation by the motion estimation unit 166. For example, when the input image is blurred, the template matching template itself may be converted into a blurred image. Further, for example, when the input image is blurred, a process of removing the blur component in the target image for motion estimation may be performed.

The behavior determination unit 168 determines the user's behavior according to the motion information detected by the motion estimation unit 166. 9 and 10 show an example of determining the user's behavior based on the motion estimation result. FIG. 9 shows an example of an image when the movement of the image is small. The arrow A displayed in the observation image in FIG. 9 represents the motion vector of the region of the lesion 182 between the images of the image IM (t) and the image IM (t + 1) of the next frame.

FIG. 10 shows an example of an image when the movement of the image is large. The arrow B displayed in the observation image in FIG. 10 represents the motion vector of the region of the lesion 182 between the images of the image IM (t) and the image IM (t + 1) of the next frame. For example, as shown in FIG. 9, when the motion vector is small, it can be determined that the position of the scope has moved during the discrimination of the user. On the other hand, as shown in FIG. 10, when the motion vector is large, it can be determined that the user has no intention of discrimination.

The behavior determination unit 168 determines the user's behavior by comparing the magnitude of the motion vector with the threshold value. The magnitude of the motion vector reflects the amount of movement of the image. That is, it is possible to determine whether or not the user's behavior is intended for discrimination from the magnitude of the motion vector. The motion vector may be acquired as the motion of the entire image, or may be acquired as the motion of only a certain region of interest.

Further, the behavior determination unit 168 is not limited to the mode of determining the user's behavior based on the amount of movement of the image, and determines the user's behavior by comparing it with a predefined database or a database updated in time series. You may.

FIG. 11 is an example of a database that defines the correspondence between motion information and user behavior. The medical image processing apparatus 160 may include a storage unit 169 that stores a database 167. The storage unit 169 may be a storage device exemplified by a semiconductor memory, a hard disk drive, a solid state drive, an optical disk, or the like, or may be an appropriate combination thereof. The storage unit 169 may be an internal storage device built in the medical image processing device 160, or an external storage device connected to the medical image processing device 160.

The behavior determination unit 168 can determine the intention of the user's behavior by collating with the database 167 based on the motion information obtained from the motion estimation unit 166. For example, if the movement of the image indicated by the movement information is stopped, or if the movement is very small, it is determined that the image is being discriminated. Further, when the movement of the image indicated by the movement information is a constant velocity motion or a rotational motion, it is determined that the image is being discriminated. If the movement of the image indicated by the movement information is a constant acceleration motion, it is determined that screening is in progress. The determination result of the action determination unit 168 is sent to the notification control unit 172 (see FIG. 4). The determination result of the action determination unit 168 corresponds to the action information in which the user's action intention is estimated.

The classification unit 170 shown in FIG. 4 performs classification processing into a specific class for images determined to be classable by the pass / fail determination unit 164. Examples of the classification method include the following classification classes.

<Specific example of classification class>
[Example 1] Two classifications, neoplastic or non-neoplastic, can be adopted.

[Example 2] Endoscopic findings classification, specifically NICE classification, JNET classification, or the like can be adopted.

[Example 3] Classification by disease type, for example, a mode of classifying into hyperplastic polyps, adenomas, intramucosal cancers, highly invasive cancers, inflammatory polyps and the like can be adopted.

"NICE" is an abbreviation for NBI (Narrow band imaging) International Colorectal Endoscopic. "JNET" is an abbreviation for "the Japan NBI Expert Team".

The NICE classification is a non-expanded NBI classification, and each of the three items of lesion color tone (Color), microvascular pattern (Vessels), and surface pattern (Surface pattern) is classified into Type1, Type2, and Type3. NS. Type 1 is a diagnostic index for hyperplastic lesions, Type 2 is a diagnostic index for adenoma to intramucosal cancer, and Type 3 is a diagnostic index for SM (submucosa) deep infiltration cancer. The JNET classification is a classification of NBI magnifying endoscopic findings for colorectal tumors. The JNET classification is classified into Type1, Type2A, Type2B, and Type3 for each item of "vessel pattern" and "surface pattern".

The classification unit 170 may recognize two classifications, which are simply "cancerous" or "non-cancerous", in place of or in combination with detailed classification such as NICE classification.

For example, a convolutional neural network (CNN) is used for the classification process of the classification unit 170 shown in FIG. The classification unit 170 is configured by using a second trained model learned by machine learning so as to perform an image classification task of classifying into a specific class as illustrated in [Example 1] to [Example 3]. be able to.

When the determination result obtained from the pass / fail determination unit 164 is an "image suitable for recognition", the classification process by the classification unit 170 is executed. The classification unit 170 extracts the feature amount from the image and classifies the image. The classification unit 170 may perform detection of a region of interest (for example, a region of a lesion), detection of a region of a lesion, and / or segmentation based on the calculated feature amount. Further, the classification unit 170 may perform the classification process using the feature amount calculated by the recognition unit 164A.

The notification control unit 172 controls the operation of the notification unit 174 based on the determination result obtained from the pass / fail determination unit 164, the determination result obtained from the action determination unit 168, and the classification result obtained from the classification unit 170. .. The notification unit 174 may be configured to include a display device 18 (see FIGS. 1 and 2). Further, the notification unit 174 may be a display device different from the display device 18. The notification control unit 172 can control the information content output to the notification unit 174.

FIG. 12 is a diagram showing an example of notification of the classification result obtained from the classification unit 170. In the example of FIG. 12, a polyp, which is a lesion 182, is detected in the observation image, and “result: neoplasm” (tumor) as notification information indicating the classification result is on the right side of the display area of the endoscopically photographed image. Character information is displayed. It is preferable that the notification information indicating the classification result is displayed on the same screen together with the endoscopically captured image.

On the other hand, even if the behavior determination unit 168 determines that the user has no intention of discrimination, if the automatic classification result is notified, the user's attention will be impaired, observation will be hindered, and another lesion will be overlooked. There is a risk of being connected. Therefore, when the action determination unit 168 determines that the user has no intention of discrimination, the notification control unit 172 implements notification control so as to display as shown in FIGS. 13 or 14.

FIG. 13 is an example in which the classification result is not notified. In the example shown in FIG. 13, the information "result: −" is displayed on the right side of the display area of the endoscopically captured image as information clearly indicating that the classification result is not notified.

FIG. 14 is a notification example in which information for notifying that the classification process has not been activated is presented instead of the character information indicating the classification result. In the example shown in FIG. 14, the information "classification off" is displayed on the right side of the display area of the endoscopically captured image as information clearly indicating that the classification process has not been activated, and the character "on" is not. It is displayed or grayed out. Instead of the notification form as shown in FIG. 13, the notification form as shown in FIG. 14 may be adopted. Further, a notification form in which the notification form shown in FIG. 13 and the notification form shown in FIG. 14 may be combined may be adopted.

<< Example of operation of the medical image processing apparatus according to the first embodiment >>
A medical image processing method using the medical image processing device 160 will be described. FIG. 15 is a flowchart showing an example of the operation of the medical image processing device 160 according to the first embodiment. The operation of the medical image processing device may be understood as an operating method of the medical image processing device, or may be understood as an operating method of the processor device.

In step S11, the medical image processing device 160 receives the current image via the image acquisition unit 162. The image acquired by the image acquisition unit 162 is a medical image including a subject image taken by using the electronic endoscope 12, and is one image of a time-series image sequentially taken in time series. For example, the image is a one-frame image constituting a moving image. Step S11 is an example of the “image acquisition step” in the present disclosure.

In step S12, the pass / fail determination unit 164 performs a process of recognizing the state of the input current image. Specifically, the pass / fail determination unit 164 performs a process of recognizing whether or not the image is suitable for the classification process.

In step S14, the approval / disapproval determination unit 164 determines whether or not classification is possible. Step S12 and step S14 are examples of the "possibility determination step" in the present disclosure. When the approval / disapproval determination unit 164 determines in the determination process of step S14 that it can be classified, the process proceeds to step S16 and step S20.

In step S16, the motion estimation unit 166 estimates the motion of an image using two or more images including past or future images. Step S16 is an example of the “motion estimation process” in the present disclosure.

After step S16, in step S18, the action determination unit 168 determines the user's action based on the motion information which is the estimation result of the motion estimation unit 166. Step S18 is an example of the “behavior determination step” in the present disclosure.

Further, in parallel with the processes of steps S16 and S18, in step S20, the classification unit 170 performs a process of recognizing the lesion region from the image and classifying the lesion into a predetermined class. Step S20 is an example of the "classification step" in the present disclosure.

After step S18 and step S20, the process proceeds to step S22. Further, when the determination result in step S14 is "No determination", that is, when it is determined that the classification is impossible, the process of step S20 is omitted from step S16, and the process proceeds to step S22.

In step S22, the notification control unit 172 sets the notification information to be output to the notification unit 174. When the determination result of the action determination unit 168 corresponds to the action of "discrimination", the notification control unit 172 sets the notification information so as to notify the classification result of the classification unit 170 (see FIG. 12). On the other hand, when the determination result of the action determination unit 168 does not correspond to the action of "discrimination", the notification control unit 172 sets the notification information so that the classification result of the classification unit 170 is not notified (FIG. 13 and FIG. 14). Step S22 is an example of the “notification control step” in the present disclosure.

After step S22, the flowchart of FIG. 15 ends. The processing of the flowchart shown in FIG. 15 is repeated for each image acquired in time series.

The processing of the flowchart shown in FIG. 15 is not limited to the mode of performing each image acquired in the time series, and may be performed only for a part of the images acquired in the time series.

<< Advantages of the First Embodiment >>
According to the configuration of the first embodiment described above, there are the following advantages.

(1) According to the medical image processing device 160, the intention of the user's action is determined from a plurality of images obtained in time series, and the classification result is notified for the image determined to have the intention of discrimination, while the intention of discrimination is given. It is possible to make the classification result non-notification for the image determined to be absent. As a result, excessive notification at an unnecessary timing can be suppressed, and appropriate information can be provided at a timing when support is required.

(2) According to the medical image processing device 160, it is possible to maintain the user's attention required for observation without hindering the user's observation of the image. According to the medical image processing apparatus 160, it is possible to suppress oversight of a region of interest such as a lesion region.

(3) According to the medical image processing apparatus 160, when an image unsuitable for recognition is temporarily included in the images obtained in time series, motion estimation processing and behavior determination for the unsuitable image are performed. Processing and classification processing are omitted, and these processing is not performed. Therefore, it is possible to prevent a classification result (erroneous recognition result) having low truthfulness from being generated from an image unsuitable for recognition and being provided to the user. In addition, it is possible to avoid notification of the classification result for images that are unsuitable for recognition, and it is possible to provide accurate classification result information in the time-series image group.

(4) According to the medical image processing device 160, the classification result can be notified to the user in real time during the display of the moving image obtained by the continuous shooting during the examination.

(5) According to the medical image processing apparatus 160, it is possible to provide useful diagnostic support for users such as doctors.

<< Second Embodiment: Application example to medical information management system >>
The medical image processing device according to the present invention is not limited to the form applied to the processor device 16 of the endoscope system 10 illustrated in FIG. 1, and various applications are possible. For example, the medical image processing device can be applied to a medical information management system that manages various medical information including endoscopic images.

FIG. 16 is a block diagram showing a configuration example of a medical information management system. The medical information management system 200 includes an image acquisition terminal 202, an image storage server 204, an information management device 210, a display device 218, and an input device 219. Each of the image acquisition terminal 202, the image storage server 204, and the information management device 210 is connected to the telecommunication line 230. The term "connection" is not limited to wired connection, but also includes the concept of wireless connection.

The telecommunication line 230 may be a local area network or a wide area network. The telecommunication line 230 is composed of an appropriate combination of wired and wireless.

The processor device 16 of the endoscope system 10 is connected to the telecommunication line 230. The medical image generated by the processor device 16 is taken into at least one of the image acquisition terminal 202, the image storage server 204, and the information management device 210 via the telecommunication line 230. For example, the medical image generated by the processor device 16 is sent to the image acquisition terminal 202. The image acquisition terminal 202 receives a medical image from the processor device 16.

The image acquisition terminal 202 sends the medical image received from the processor device 16 to the image storage server 204. Further, the image acquisition terminal 202 may transmit the medical image received from the processor device 16 to the information management device 210.

The image storage server 204 serves as a storage device for storing a database of various medical images. Instead of the image storage server 204, cloud storage may be used. In addition to the medical image, the image storage server 204 may store image analysis results such as a region of interest (region of interest) included in the medical image, the presence or absence of an object of interest, and the result of image classification.

Although one endoscope system 10 is shown in FIG. 16, a plurality of endoscope systems can be connected to the telecommunication line 230. Further, the telecommunication line 230 may be connected not only to an endoscopic system but also to another medical imaging device such as an ultrasonic diagnostic device. The ultrasonic image obtained from the ultrasonic diagnostic apparatus is an example of a "medical image". The medical imaging device may be, for example, one or a combination of an X-ray imaging device, a CT imaging device, an MRI (magnetic resonance imaging) imaging device, a nuclear medicine diagnostic device, or a fundus camera.

The information management device 210 is realized by, for example, computer hardware and software. A display device 218 and an input device 219 are connected to the information management device 210. The information management device 210 may include a part or all of the functions of the medical image processing device 160 shown in FIG. For example, the information management device 210 includes the functions of the image acquisition unit 162, the pass / fail determination unit 164, the motion estimation unit 166, the action determination unit 168, the classification unit 170, and the notification control unit 172 shown in FIG.

The display device 218 shown in FIG. 16 can function as a notification unit 174. The function of the information management device 210 can be realized by one or a plurality of computers, and can also be realized by cloud computing. The information management device 210 may include the function of the image storage server 204. Further, the image acquisition terminal 202 can function as an image acquisition unit 162. It is also possible to omit the image acquisition terminal 202, in which case the image storage server 204 and / or the information management device 210 is configured to acquire medical images from the processor device 16.

According to the medical information management system 200 according to the present embodiment, not only the medical image obtained from the electronic endoscope 12 in real time, but also when playing back a moving image or the like stored in the image storage server 204, FIG. The flowchart described in the above can be processed.

The information management device 210 may be installed in, for example, an operating room, an examination room, a conference room, or the like in a hospital, or may be installed in a medical institution, a research institution, or the like of an out-of-hospital facility. The information management device 210 may be a workstation that supports medical examination, treatment, diagnosis, or the like, or may be a business support device that supports medical work. The business support device may have a function of accumulating clinical information, supporting the creation of diagnostic documents, supporting the creation of reports, and the like. The information management device 210 is an example of the “medical image processing device” in the present disclosure. The medical information management system 200 is an example of the "diagnosis support device" in the present disclosure.

<< Hardware configuration of each processing unit and control unit >>
A processing unit that executes various processes such as an image acquisition unit 162, a pass / fail determination unit 164, a motion estimation unit 166, an action determination unit 168, a classification unit 170, and a notification control unit 172 of the medical image processing unit 160 described with reference to FIG. The hardware structure of the (processing unit) is various processors as shown below.

The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes programs and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and an FPGA (Field). Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as Programmable Gate Array), and ASIC (Application Specific Integrated Circuit), which are specially designed to execute specific processing. A dedicated electric circuit or the like, which is a processor having a circuit configuration, is included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or a server. There is a form in which a processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

<< Modification 1 >>
The electronic endoscope is not limited to a flexible endoscope, and may be a rigid endoscope or a capsule endoscope. Further, the device for generating a time-series medical image including a subject image is not limited to an electronic endoscope, and may be, for example, an ultrasonic diagnostic device.

<< Modification 2 >>
The medical image processing device of the present disclosure can be used as a diagnostic support device that supports medical examination, treatment, diagnosis, etc. by a doctor or the like. The term "diagnostic support" includes the concept of diagnostic support and / or treatment support.

<< About the observation light of the endoscopic system >>
As the observation light, white light, light in one or a plurality of specific wavelength bands, or light in various wavelength bands according to the observation purpose such as a combination thereof is selected. White light is light in a white wavelength band or light in a plurality of wavelength bands. The "specific wavelength band" is a band narrower than the white wavelength band. Specific examples relating to a specific wavelength band are shown below.

<First example>
A first example of a particular wavelength band is, for example, the blue or green band in the visible range. The wavelength band of the first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example is in the wavelength band of 390 nm or more and 450 nm or less or a wavelength of 530 nm or more and 550 nm or less. It has a peak wavelength in the band.

<Second example>
A second example of a particular wavelength band is, for example, the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nm or more and 615 nm or less or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example has a wavelength of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. It has a peak wavelength in the band.

<Third example>
The third example of a specific wavelength band includes a wavelength band in which the absorption coefficient of hemoglobin oxide and reduced hemoglobin are different, and the light of the third example has a peak wavelength in a wavelength band in which the absorption coefficient of hemoglobin oxide and reduced hemoglobin are different. Has. The wavelength band of the third example includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, a wavelength band of 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is 400 ± 10 nm. It has a peak wavelength in the wavelength band of 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less.

<4th example>
The fourth example of the specific wavelength band is used for observing the fluorescence emitted by the fluorescent substance in the living body (fluorescence observation), and the wavelength band of the excitation light that excites the fluorescent substance, for example, 390 nm to 470 nm.

<Example 5>
A fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example is within the wavelength band of 790 nm or more and 820 nm or less or a wavelength of 905 nm or more and 970 nm or less. It has a peak wavelength in the band.

<< Switching of observation light >>
As the type of light source, a laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof may be adopted. The type and wavelength of the light source, the presence or absence of a filter, etc. are preferably configured according to the type of subject, the purpose of observation, etc., and when observing, the wavelength of the illumination light is combined according to the type of subject, the purpose of observation, etc. And / or switching is preferred. When switching the wavelength, for example, by rotating a disk-shaped filter (rotary color filter) arranged in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of the emitted light is switched. May be good.

The image sensor used in the electronic endoscope is not limited to the color image sensor in which the color filter is arranged for each pixel, and may be a monochrome image sensor. When a monochrome image sensor is used, the wavelength of the illumination light can be sequentially switched to perform surface-sequential (color-sequential) imaging. For example, the wavelength of the emitted illumination light may be sequentially switched between purple, blue, green, and red, or a wide band light (white light) is irradiated and emitted by a rotary color filter (red, green, blue, etc.). The wavelength of the illumination light to be used may be switched. Further, the wavelength of the illumination light emitted by the rotary color filter by irradiating one or a plurality of narrow band lights may be switched. The narrow band light may be infrared light having two or more wavelengths having different wavelengths.

<< Example of generating a special light image >>
The processor device 16 may generate a special optical image having information in a specific wavelength band based on a normal optical image obtained by imaging with white light. The generation referred to here includes the concept of "acquisition". In this case, the processor device 16 functions as a special optical image acquisition unit. The processor device 16 transmits signals in a specific wavelength band to red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y), which are usually contained in an optical image. ) Can be obtained by performing an operation based on the color information.

<< Example of feature image generation >>
The processor device 16 is a medical image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band, and a special optical image obtained by irradiating light in a specific wavelength band. An operation based on at least one of the above can be used to generate a feature image. A feature image is a form of medical image.

<< About the program that makes the computer realize the function of the medical image processing device >>
A program for realizing the function of the medical image processing apparatus described in the above-described embodiment on a computer is recorded on an optical disk, a magnetic disk, or a computer-readable medium such as a semiconductor memory or other tangible non-temporary information storage medium. It is possible to provide a program through an information storage medium. Further, instead of the mode in which the program is stored and provided in such a tangible non-temporary information storage medium, it is also possible to provide the program signal as a download service by using a telecommunication line such as the Internet.

It is also possible to provide a part or all of the functions of the medical image processing apparatus described in the above-described embodiment as an application server, and provide a service that provides the processing function through a telecommunication line.

<< Combination of Embodiments and Modifications >>
The components described in the above-described embodiment and the components described in the modified examples can be used in combination as appropriate, or some components can be replaced.

<< Additional notes >>
The present specification includes the disclosure of the invention described below in addition to the above-described embodiments and modifications.

(Appendix 1)
The medical image processing apparatus includes a medical image analysis processing unit and a medical image analysis result acquisition unit, and the medical image analysis processing unit is a notable area based on the feature amount of pixels of a medical image. The medical image analysis result acquisition unit is a medical image processing device that detects an area and acquires the analysis result of the medical image analysis processing unit.

The medical image analysis processing unit may include an image recognition unit.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of the pixel of the medical image, and the medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit. Device.

(Appendix 3)
The medical image analysis result acquisition unit acquires from a recording device that records the analysis result of the medical image, and the analysis result is either a notable area included in the medical image or the presence or absence of a notable object. A medical image processor that is, or both.

(Appendix 4)
A medical image is a medical image processing apparatus that is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band.

(Appendix 5)
A medical image is an image obtained by irradiating light in a specific wavelength band, and the specific wavelength band is a narrower band than the white wavelength band.

(Appendix 6)
A medical image processing device in which a specific wavelength band is a blue or green band in the visible range.

(Appendix 7)
A medical treatment in which a specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and light in a specific wavelength band has a peak wavelength in a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)
A medical image processing device in which a specific wavelength band is a red band in the visible range.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which the absorption coefficient differs between the oxidized hemoglobin and the reduced hemoglobin. Medical image processing device.

(Appendix 11)
The specific wavelength band includes 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less, and the light in the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ±. A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or 600 nm or more and 750 nm or less.

(Appendix 12)
A medical image is an in-vivo image that captures the inside of a living body, and an in-vivo image is a medical image processing device that has information on fluorescence emitted by a fluorescent substance in the living body.

(Appendix 13)
Fluorescence is a medical image processing device obtained by irradiating a living body with excitation light having a peak of 390 nm or more and 470 nm or less.

(Appendix 14)
A medical image is an in-vivo image of the inside of a living body, and a specific wavelength band is a wavelength band of infrared light.

(Appendix 15)
A medical image in which a specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and light in a specific wavelength band has a peak wavelength in a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. Processing equipment.

(Appendix 16)
The medical image acquisition unit acquires a special optical image having information in a specific wavelength band based on a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band. A medical image processing device equipped with an optical image acquisition unit and a medical image is a special optical image.

(Appendix 17)
A medical image processing device that obtains a signal in a specific wavelength band by calculation based on RGB or CMY color information included in a normal optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band and a special light image obtained by irradiating light in a specific wavelength band. A medical image processing device including a feature amount image generation unit that generates a feature amount image, and the medical image is a feature amount image.

(Appendix 19)
The medical image processing apparatus according to any one of Supplementary note 1 to Supplementary note 18, and an endoscope that acquires an image by irradiating at least one of light in a white wavelength band or light in a specific wavelength band. An endoscopic device comprising.

(Appendix 20)
A diagnostic support device including the medical image processing device according to any one of Supplementary Note 1 to Supplementary Note 18.

(Appendix 21)
A medical business support device including the medical image processing device according to any one of Supplementary Note 1 to Supplementary Note 18.

[others]
In the embodiment of the present invention described above, the constituent requirements can be appropriately changed, added, or deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made by a person having ordinary knowledge in an equivalent related field within the technical idea of the present invention.

10 Endoscope system 12 Electronic endoscope 14 Light source device 16 Processor device 18 Display device 19 Input device 20 Insertion part 22 Tip hard part 22A Tip surface 24 Curved part 26 Flexible part 30 Operation part 31 Angle knob 32 Air supply / water supply button 33 Suction button 34 Mode selector switch 35 Zoom operation unit 36 Treatment tool introduction port 40 Universal cord 42 Connector 42A Video connector 42B Light guide connector 42C Water supply connector 44 Water supply tank 50 Lighting window 52 Observation window 54 Force outlet 56 Air supply water supply nozzle 60 Objective Optical system 62 Imaging element 64 Analog front-end circuit 65 Timing generator 66 CPU
68 Imaging chip 70 CPU
72 ROM
74 RAM
76 Digital signal processing circuit 78 Display control circuit 100 1st light source 101 1st light source drive circuit 102 2nd light source 103 2nd light source drive circuit 105 Converging part 120 Light guide 121 Incident end 122 Exit end 124 Phosphorant 160 Medical image processing device 162 Image acquisition unit 164 Possibility judgment unit 164A Recognition unit 166 Motion estimation unit 167 Database 168 Action judgment unit 169 Storage unit 170 Classification unit 172 Notification control unit 174 Notification unit 182 Disease 184 Bubble 200 Medical information management system 202 Image acquisition terminal 204 Image Storage server 210 Information management device 218 Display device 219 Input device 230 Telecommunications line IM1, IM2, IM3 Image IM (t) Current image IM (t-1) Past image IM (t + 1) Image of next frame S11-S22 Medical Processing steps in the image processing device

Claims (23)

An image acquisition unit that acquires multiple time-series images including a subject image,
A feasibility determination unit for determining whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition, and a feasibility determination unit.
A motion estimation unit that estimates motion from two or more images obtained from the image acquisition unit, and a motion estimation unit.
Based on the motion information obtained from the motion estimation unit, the behavior determination unit that determines the user's behavior and the behavior determination unit
A classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing,
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit.
Medical image processing device including. The medical image processing apparatus according to claim 1, wherein the unsuitable image is an image in which the subject is blurred. The medical image processing apparatus according to claim 1, wherein the unsuitable image is an image in which the subject is blurred. The medical image processing apparatus according to claim 1, wherein the unsuitable image is an image in which the subject is covered with water. The medical image processing apparatus according to claim 1, wherein the unsuitable image is an image having a residue on the subject. The medical image processing apparatus according to any one of claims 1 to 5, wherein the pass / fail determination unit includes a recognition unit that recognizes whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition. .. The medical image processing apparatus according to any one of claims 1 to 6, wherein the motion estimation unit and the behavior determination unit do not perform processing on an image determined by the possibility determination unit to be an inappropriate image. .. The medical image processing apparatus according to any one of claims 1 to 7, wherein the classification process of the classification unit does not perform the classification process of the image determined by the pass / fail determination unit to be an unsuitable image. The motion information includes a motion vector.
The medical image processing apparatus according to any one of claims 1 to 8, wherein the action determination unit determines a user's action based on the magnitude of the motion vector. The medical image processing apparatus according to any one of claims 1 to 9, wherein the action determination unit determines the action using a database in which a correspondence relationship between the movement information and the user's action is defined. The medical image processing apparatus according to claim 10, further comprising a storage unit for storing the database. The medical image processing apparatus according to any one of claims 1 to 11, wherein the action includes a discrimination action. The medical image processing device according to claim 12, wherein the notification control unit does not notify the classification result of the classification unit for an image whose determination result of the behavior determination unit does not correspond to the behavior of the discrimination. The medical image processing apparatus according to any one of claims 1 to 13, wherein the classification unit recognizes a lesion region from an image obtained from the image acquisition unit and classifies the lesion. The medical image processing apparatus according to any one of claims 1 to 14, wherein the classification unit is configured by using a convolutional neural network. The medical image processing apparatus according to any one of claims 1 to 15, further comprising a notification unit that notifies the classification result of the classification unit based on the control of the notification control unit. The medical image processing apparatus according to any one of claims 1 to 16, wherein each of the plurality of images in the time series is an endoscopic image taken by using an electronic endoscope. An image acquisition process for acquiring multiple time-series images including a subject image,
A step of determining whether or not the image obtained from the image acquisition step is an image unsuitable for recognition, and a step of determining whether or not the image is unsuitable for recognition.
A motion estimation step of estimating motion from two or more images obtained from the image acquisition step, and a motion estimation step.
Based on the motion information obtained from the motion estimation process, the behavior determination process for determining the user's behavior and the behavior determination process
A classification step of recognizing an image obtained from the image acquisition step and performing a classification process,
A notification control step that controls notification information based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step, and
Medical image processing methods including. An electronic endoscope that photographs the inside of the body cavity,
A processor device that processes an image signal obtained from the electronic endoscope, and
Is an endoscopic system that includes
The processor device is
An image acquisition unit that acquires a plurality of time-series images including a subject image taken by using the electronic endoscope, and an image acquisition unit.
A feasibility determination unit for determining whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition, and a feasibility determination unit.
A motion estimation unit that estimates motion from two or more images obtained from the image acquisition unit, and a motion estimation unit.
Based on the motion information obtained from the motion estimation unit, the behavior determination unit that determines the user's behavior and the behavior determination unit
A classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing,
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit.
Endoscopic system with. A processor device that processes image signals obtained from an electronic endoscope.
An image acquisition unit that acquires a plurality of time-series images including a subject image taken by using the electronic endoscope, and an image acquisition unit.
A feasibility determination unit for determining whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition, and a feasibility determination unit.
A motion estimation unit that estimates motion from two or more images obtained from the image acquisition unit, and a motion estimation unit.
Based on the motion information obtained from the motion estimation unit, the behavior determination unit that determines the user's behavior and the behavior determination unit
A classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing,
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit.
A processor device comprising. An image acquisition unit that acquires multiple time-series images including a subject image,
A feasibility determination unit for determining whether or not the image obtained from the image acquisition unit is an image unsuitable for recognition, and a feasibility determination unit.
A motion estimation unit that estimates motion from two or more images obtained from the image acquisition unit, and a motion estimation unit.
Based on the motion information obtained from the motion estimation unit, the behavior determination unit that determines the user's behavior and the behavior determination unit
A classification unit that recognizes the image obtained from the image acquisition unit and performs classification processing,
A notification control unit that controls notification information based on the behavior information obtained from the behavior determination unit and the classification result obtained from the classification unit.
A notification unit that notifies information including the classification result of the classification unit based on the control of the notification control unit, and a notification unit.
Diagnostic support device equipped with. On the computer
An image acquisition process for acquiring multiple time-series images including a subject image,
A step of determining whether or not the image obtained from the image acquisition step is an image unsuitable for recognition, and a step of determining whether or not the image is unsuitable for recognition.
A motion estimation step of estimating motion from two or more images obtained from the image acquisition step, and a motion estimation step.
Based on the motion information obtained from the motion estimation process, the behavior determination process for determining the user's behavior and the behavior determination process
A classification step of recognizing an image obtained from the image acquisition step and performing a classification process,
A notification control step that controls notification information based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step, and
A program that executes. A non-temporary, computer-readable storage medium when the instructions stored in the storage medium are read by a computer.
An image acquisition process for acquiring multiple time-series images including a subject image,
A step of determining whether or not the image obtained from the image acquisition step is an image unsuitable for recognition, and a step of determining whether or not the image is unsuitable for recognition.
A motion estimation step of estimating motion from two or more images obtained from the image acquisition step, and a motion estimation step.
Based on the motion information obtained from the motion estimation process, the behavior determination process for determining the user's behavior and the behavior determination process
A classification step of recognizing an image obtained from the image acquisition step and performing a classification process,
A notification control step that controls notification information based on the behavior information obtained from the behavior determination step and the classification result obtained from the classification step, and
A storage medium that causes a computer to execute.

Images