JP6889282B2 - Medical image processing equipment and methods, endoscopic systems, processor equipment, diagnostic support equipment and programs - Google Patents

Description

The present invention relates to a medical image processing device and method, an endoscope system, a processor device, a diagnostic support device, and a program, and relates to an image processing technique for performing image recognition using a medical image obtained by photographing an observation target.

In the medical field, examinations using an endoscopic system are performed. In recent years, a system for recognizing a lesion contained in a medical image by using an image analysis technique has been known. In a general endoscopic system, a camera at the tip of a scope is used to photograph a body cavity, and a processor device receives the captured observation image and displays it on a monitor. At this time, the processor device may perform recognition processing of the received image and notify the user of the recognition result.

For example, when a lesion is photographed, the processor device recognizes whether it is cancerous or non-cancerous from image analysis, and notifies the user of the recognition result. A system having such a function supports decision-making such as diagnostic findings by doctors and the like.

Japanese Unexamined Patent Publication No. 2004-180923 Japanese Unexamined Patent Publication No. 2015-204960 Japanese Unexamined Patent Publication No. 2014-03488

However, for example, the manipulation of the scope, fetal movement, and the effects of at least one of the residues often result in blurring, blurring, or hiding of the lesion site in the residue. In addition, even when water is supplied to the observation site using a water supply device in order to secure a field of view, the image may be blurred or the lesion site may be covered with water due to the influence of water. Accurate recognition of lesions, etc. by image analysis from such "blurred images", "blurred images", "images containing residues", or "images with water on the subject". Is difficult and may give different recognition results. This problem is particularly problematic in a system that performs recognition processing for each image obtained in time series, such as performing recognition processing for each frame of a moving image obtained during inspection, and notifies the user of the recognition result in real time.

Further, not only the endoscopic system but also a system such as an ultrasonic diagnostic apparatus that performs recognition processing from medical images obtained in time series has the same problem.

The present invention has been made in view of such circumstances, and when an image unsuitable for image recognition is included in an image obtained in time series, the recognition result obtained from the image unsuitable for image recognition is given to the user. It is an object of the present invention to provide medical image processing devices and methods, endoscopic systems, processor devices, diagnostic support devices and programs that can be avoided from being provided.

In order to solve the problem, the following aspects of the invention are provided.

The medical image processing apparatus according to the first aspect is an image acquisition unit that acquires a time-series medical image including a subject image, an image recognition unit that performs image recognition from the medical image, and a recognition result that saves the recognition results of the image recognition unit. It includes a storage unit and a change processing unit that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.

According to the first aspect, the recognition result of an image unsuitable for image recognition, such as a blurred image, is changed to the recognition result of another image saved in the recognition result storage unit among the time-series medical images. .. The recognition result storage unit can store recognition results for images other than images that are unsuitable for image recognition. As a result, it is possible to avoid presenting an inaccurate recognition result and to show a valid recognition result in a time series.

The time-series medical image may be a moving image, or may be a still image taken at a specific time interval such as continuous shooting or interval shooting.

In the second aspect, the medical image processing apparatus of the first aspect includes a determination unit that determines whether or not to save the recognition result of the image recognition unit according to the recognition result of the image recognition unit, and according to the determination result of the determination unit. This is a medical image processing device in which the recognition result of the image recognition unit is stored in the recognition result storage unit.

The judgment unit determines that the recognition result of an image that is unsuitable for image recognition, such as a blurred image, is not saved in the recognition result storage unit (judgment that the image is not saved), and the image other than the image that is not suitable for image recognition. It is possible to determine that the recognition result is to be saved in the recognition result storage unit.

In the third aspect, in the medical image processing apparatus of the second aspect, the change processing unit stores the recognition result of the image recognition unit for the medical image given the determination that the determination unit does not save the recognition result in the recognition result storage unit. It is a medical image processing device that changes the recognition result.

In the fourth aspect, in the medical image processing apparatus of the third aspect, the image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition, and the determination unit includes the image recognition unit. It is a medical image processing device that determines that the recognition result of a medical image recognized as an unsuitable image is not stored when it is recognized as an image unsuitable for recognition.

Aspect 5 includes a task of recognizing a medical image acquired from an image acquisition unit as an image unsuitable for recognition in the medical image processing apparatus according to any one of aspects 1 to 3. The change processing unit is a medical image processing device that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit when the image recognition unit recognizes that the image is unsuitable for recognition.

Aspect 6 is a medical image processing device in which the recognition result of a medical image recognized by the image recognition unit as an image unsuitable for recognition in the medical image processing device of the fifth aspect is not stored in the recognition result storage unit. is there.

Aspect 7 is a medical image processing apparatus according to any one of aspects 4 to 6, wherein the unsuitable image is an image in which the subject is blurred.

Aspect 8 is a medical image processing apparatus according to any one of aspects 4 to 6, wherein the unsuitable image is an image in which the subject is blurred.

Aspect 9 is a medical image processing apparatus according to any one of aspects 4 to 6, wherein the unsuitable image is an image in which the subject is covered with water.

Aspect 10 is a medical image processing apparatus according to any one of aspects 4 to 6, wherein the unsuitable image is an image having a residue on the subject.

Aspect 11 is a medical image processing apparatus according to any one of aspects 1 to 10, wherein the image recognition performed by the image recognition unit includes a task of detecting a region of interest.

The region of interest may be, for example, a lesion region.

Aspect 12 is a medical image processing apparatus according to any one of aspects 1 to 11, wherein the image recognition performed by the image recognition unit includes image classification.

Aspect 13 is a medical image processing apparatus according to any one of aspects 1 to 12, wherein the image recognition unit recognizes whether or not the medical image acquired from the image acquisition unit is an image unsuitable for recognition. A first recognition unit including a recognition unit 1 and a second recognition unit that detects a region of interest from a medical image acquired from an image acquisition unit and processes at least one of the image classifications of the medical image. This is a medical image processing apparatus that does not perform processing in the second recognition unit for a medical image recognized as an image unsuitable for recognition.

Aspect 14 is a medical image processing apparatus according to any one of aspects 1 to 13, in which the image recognition unit performs image recognition for each medical image acquired in time series and outputs a recognition result for each medical image. It is a medical image processing device.

In the fifteenth aspect, in the medical image processing apparatus according to any one of the first to the fourth aspects, the recognition result of the medical image in which the recognition result of the image recognition unit is maintained without being processed by the change processing unit is obtained. It is a medical image processing device that is stored in the recognition result storage unit.

Aspect 16 is a medical image processing apparatus according to any one of aspects 1 to 15, in which the change processing unit changes the recognition result of the image recognition unit to the latest recognition result stored in the recognition result storage unit. It is a medical image processing device.

The recognition result saved in the recognition result storage unit is preferably updated to the latest one each time, and only the latest recognition result needs to be saved in the recognition result storage unit.

Aspect 17 includes a display unit for displaying the recognition result of the image recognition unit in the medical image processing apparatus according to any one of aspects 1 to 16, and the recognition result storage unit is provided according to the recognition result of the image recognition unit. This is a medical image processing device in which the stored recognition result is displayed on the display unit.

Aspect 18 indicates that in the medical image processing apparatus of Aspect 17, the display unit is a recognition result stored in the recognition result storage unit when displaying the recognition result stored in the recognition result storage unit. It is a medical image processing device that displays.

In the medical image processing apparatus of the aspect 18, the display indicating that the recognition result is stored in the recognition result storage unit has a display color as compared with the case where the recognition result of the image recognition unit is displayed as it is. It is a medical image processing apparatus including at least one display form of a different color display, a symbol display to which a specific symbol is added, a character string display to which a specific character string is added, and a blinking display.

In the 20th aspect, in the medical image processing apparatus according to any one of the 1st to 19th aspects, the acquisition of the medical image in which the recognition result of the image recognition unit is changed to the recognition result stored in the recognition result storage unit continues for a certain period of time. This is a medical image processing device that stops the recognition process of the image recognition unit in such a case.

Aspect 21 is a medical image processing apparatus according to any one of aspects 1 to 20, wherein the medical image is an endoscopic image obtained from an endoscope or an ultrasonic image obtained from an ultrasonic diagnostic apparatus. It is a processing device.

The medical image processing method according to aspect 22 is an image acquisition step for acquiring a time-series medical image including a subject image, an image recognition step for performing image recognition from the medical image, and a recognition result storage unit for recognizing the recognition result of the image recognition step. It includes a recognition result saving step to be saved in, and a change processing step to change the recognition result of the image recognition step to the recognition result saved in the recognition result saving unit according to the recognition result of the image recognition step.

In the medical image processing method of aspect 22, the same items as those specified in aspects 2 to 21 can be appropriately combined. In that case, the elements of the processing unit and the functional unit as means for carrying out the processing and operation specified in the medical image processing apparatus can be grasped as the elements of the corresponding processing and operation steps (processes). Further, the medical image processing method of aspect 22 may be understood as an operation method of the medical image processing device.

The endoscope system according to aspect 23 is an endoscope system including an electronic endoscope for photographing the inside of a body cavity and a processor device for processing an image signal obtained from the electronic endoscope. , An image acquisition unit that acquires a time-series medical image including a subject image taken with an electronic endoscope, an image recognition unit that performs image recognition from the medical image, and a recognition that saves the recognition results of the image recognition unit. It includes a result storage unit and a change processing unit that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.

In the endoscope system of aspect 23, the same items as those specified in aspects 2 to 21 can be appropriately combined.

The processor device according to aspect 24 is a processor device that processes an image signal obtained from an electronic endoscope, and is an image acquisition that acquires a time-series medical image including a subject image taken by the electronic endoscope. The recognition result of the image recognition unit, the image recognition unit that performs image recognition from the medical image, the recognition result storage unit that stores the recognition result of the image recognition unit, and the recognition result of the image recognition unit according to the recognition result of the image recognition unit. It includes a change processing unit that changes the recognition result stored in the storage unit.

In the processor device of the aspect 24, the same items as the items specified in the aspects 2 to 21 can be appropriately combined.

The diagnosis support device according to the 25th aspect is an image acquisition unit that acquires a time-series medical image including a subject image, an image recognition unit that performs image recognition from the medical image, and a recognition result storage that stores the recognition results of the image recognition unit. A change processing unit that changes the recognition result of the image recognition unit to a recognition result saved in the recognition result storage unit according to the recognition result of the unit and the image recognition unit, and a notification that notifies the recognition result of the image recognition unit. It has a part and.

In the diagnostic support device of the 25th aspect, the same items as those specified in the 2nd to 21st aspects can be appropriately combined. The display unit that displays the recognition result of the image recognition unit can be a form of the notification unit.

The program according to aspect 26 is a recognition result storage unit that stores the recognition results of an image acquisition step of acquiring a time-series medical image including a subject image, an image recognition step of performing image recognition from the medical image, and an image recognition step on a computer. The recognition result saving step to be saved in the image recognition step and the change processing step to change the recognition result of the image recognition step to the recognition result saved in the recognition result saving unit are executed according to the recognition result of the image recognition step.

In the program of aspect 26, the same items as those specified in aspects 2 to 21 can be appropriately combined. In that case, the elements of the processing unit and the functional unit as means for carrying out the processing and operation specified in the medical image processing device can be grasped as program elements that realize the corresponding processing and operation steps (functions). it can.

According to the present invention, when an image unsuitable for image recognition is included in an image obtained in time series, it is possible to avoid providing the recognition result obtained from the image unsuitable for image recognition to the user. It is possible.

FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a front view showing the tip surface of the tip rigid portion of the electronic endoscope. FIG. 3 is a block diagram showing a control system of the endoscopic system. FIG. 4 is a block diagram showing a function of the medical image processing apparatus according to the first embodiment of the present invention. FIG. 5 is a flowchart showing the operation of the medical image processing apparatus according to the first embodiment. FIG. 6 is a block diagram showing a function of the medical image processing apparatus according to the second embodiment of the present invention. FIG. 7 is a flowchart showing the operation of the medical image processing apparatus according to the second embodiment. FIG. 8 is a diagram showing an example of an endoscopic diagnosis support screen that displays a recognition result. FIG. 9 is a diagram showing another example of the endoscopic diagnosis support screen that displays the recognition result. FIG. 10 is a block diagram showing a configuration example of a medical information management system.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<< Configuration example of endoscopic system >>
FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. The endoscope system 10 includes an electronic endoscope 12, a light source device 14, and a processor device 16. A display device 18 and an input device 19 are connected to the processor device 16. The electronic endoscope 12 may be referred to as a "scope", an "electronic scope", or simply an "endoscope".

The electronic endoscope 12 of this example is a flexible endoscope. The electronic endoscope 12 includes an insertion unit 20, an operation unit 30, and a universal cord 40. The insertion portion 20 is a portion to be inserted into the body cavity of the subject to be inspected. The insertion portion 20 includes a tip hard portion 22, a curved portion 24, and a soft portion 26 in order from the tip side toward the hand side. An illumination optical system, an objective optical system, an image pickup device, and the like are arranged inside the hard tip portion 22. The curved portion 24 has a structure that smoothly bends in four directions of up, down, left, and right from the reference position according to the operation of the angle knob 31. The hand side of the flexible portion 26 is referred to as the base end portion of the insertion portion 20.

The operation unit 30 is provided at the base end portion of the insertion unit 20. The operation unit 30 includes various operation members operated by the operator. For example, the operation unit 30 includes an angle knob 31 used for the bending operation of the bending unit 24, an air supply water supply button 32 for performing an air supply / water supply operation, and a suction button 33 for performing a suction operation. It is provided. By operating the angle knob 31, the operator can bend the curved portion 24 and freely change the direction of the hard tip portion 22. The operation unit 30 also includes a mode changeover switch 34 used for the observation mode switching operation, a zoom operation unit 35, and a still image shooting instruction unit (not shown) for giving a still image shooting instruction of the observed portion. It is provided.

Further, the operation unit 30 is provided with a treatment tool introduction port 36. The treatment tool introduction port 36 is an opening for inserting a treatment tool (not shown) into the treatment tool insertion passage (not shown) that is inserted through the insertion portion 20. Treatment tools may include, for example, biopsy forceps, catheters, high frequency snares, and the like. The treatment tool also includes a guide tube, a tracal tube, a sliding tube and the like. The treatment tool introduction port 36 may be called a forceps port.

The universal cord 40 is a cord for connecting the electronic endoscope 12 to the light source device 14 and the processor device 16. A cable and a light guide extending from the insertion portion 20 are inserted into the universal cord 40. The cable extending from the insertion portion 20 includes a communication cable used for signal transmission and a power supply cable used for power supply. A connector 42 is provided at one end of the universal cord 40.

The connector 42 is a composite type connector including a video connector 42A and a light guide connector 42B. One end of the cable is arranged on the video connector 42A. The video connector 42A is detachably connected to the processor device 16. One end of the light guide is arranged on the light guide connector 42B. The light guide connector 42B is detachably connected to the light source device 14. Further, the light guide connector 42B is provided with a water supply connector 42C, and the water supply tank 44 is connected via the water supply connector 42C.

The processor device 16 is electrically connected to the light source device 14 via the connector 42. The processor device 16 comprehensively controls the operation of the endoscope system 10 including the light source device 14. The processor device 16 supplies power to the electronic endoscope 12 via a cable inserted in the universal cord 40, and controls the drive of the image sensor. Further, the processor device 16 receives an image pickup signal transmitted from the electronic endoscope 12 via a cable, and performs various signal processing on the received image pickup signal to convert it into image data. The image data converted by the processor device 16 is displayed on the display device 18 as an endoscopically captured image (observation image).

FIG. 2 is a front view showing the tip surface 22A of the tip rigid portion 22 of the electronic endoscope 12. The tip surface 22A of the tip rigid portion 22 is provided with an illumination window 50, an observation window 52, a forceps outlet 54, and an air supply / water supply nozzle 56. Although not shown in FIG. 2, an emission end 122 of the light guide 120 that guides the light from the light source device 14 is arranged behind the illumination window 50 (see FIG. 3). Illumination light is emitted from the illumination window 50 to the observed area. Two illumination windows 50 are arranged symmetrically with respect to the observation window 52.

The observation window 52 is a window for taking in the reflected light from the observed area and taking an image of the observed area. Although not shown in FIG. 2, an objective optical system 60 and an image sensor 62 for capturing the image light of the observed region in the body cavity are arranged behind the observation window 52 (see FIG. 3). ..

The forceps outlet 54 is connected to a forceps channel (not shown) arranged in the insertion portion 20 and communicates with a treatment tool introduction port 36 (see FIG. 1) provided in the operation portion 30. The treatment tool inserted from the treatment tool introduction port 36 is taken out into the body cavity from the forceps outlet 54.

The air supply water supply nozzle 56 injects wash water or air toward the observation window 52 and / or the body cavity in response to the operation of the air supply water supply button 32 (see FIG. 1) provided in the operation unit 30. .. The washing water and air are supplied from the air supply / water supply device built in the light source device 14.

FIG. 3 is a block diagram showing a control system of the endoscope system 10. The hard tip portion 22 of the electronic endoscope 12 includes an objective optical system 60, an image sensor 62, an analog front end (AFE) circuit 64, a timing generator (TG) 65, and a CPU. (Central Processing Unit) 66 and.

The objective optical system 60 is configured by using a zoom lens. A prism for guiding light (not shown) is arranged between the objective optical system 60 and the image sensor 62. The image pickup device 62 may be a CMOS (complementary metal oxide semiconductor) type image pickup device or a CCD (Charged Coupled Device) type image pickup device. Here, an example of using a CMOS type image sensor will be described.

Although not shown, the image sensor 62 is a solid-state image sensor of a single plate color image pickup system including a color filter composed of a plurality of color segments. The color filter may be, for example, a Bayer-arranged primary color filter including red (R), green (G), and blue (B).

A large number of pixels are arranged in a matrix on the image pickup surface of the image pickup device 62, and each pixel is provided with a photo sensor. The light incident on the image pickup surface of the image pickup element 62 is stored as an electric charge in the photosensor of each pixel. The amount of signal charge accumulated in the photosensor of each pixel is sequentially read out as a pixel signal by scanning in the vertical and horizontal directions by a vertical scanning circuit and a horizontal scanning circuit (both not shown), and at a predetermined frame rate. It is output.

The timing generator 65 generates a drive pulse of the image sensor 62 and a synchronous pulse to the analog front-end circuit 64 based on the control of the CPU 66. The drive pulse of the image pickup device 62 includes a vertical scanning pulse, a horizontal scanning pulse, a reset pulse, and the like.

The image sensor 62 is driven by a drive pulse input from the timing generator 65, photoelectrically converts an optical image imaged on the image pickup surface via the objective optical system 60, and outputs the image sensor 62 as an image pickup signal.

The analog front-end circuit 64 includes a correlated double sampling (CDS) circuit, an automatic gain control (AGC) circuit, and an A / D converter. The CDS circuit performs a correlation double sampling process on the image pickup signal output from the image pickup element 62, and removes the reset noise and the amplifier noise generated in the image pickup element 62.

The AGC circuit amplifies the image pickup signal whose noise has been removed by the CDS circuit with a gain (amplification rate) specified by the CPU 66. The A / D converter converts the imaging signal amplified by the AGC circuit into a digital signal having a predetermined number of bits and outputs the signal. The image pickup signal digitized and output by the analog front-end circuit 64 is input to the processor device 16 through the signal line.

The image sensor 62, the analog front end circuit 64, and the timing generator 65 can be configured as a monolithic integrated circuit, and each of these circuit elements is included in one image pickup chip 68. The image pickup chip 68 mounted on the electronic endoscope 12 of this example is a so-called "CMOS sensor chip", and is mounted on a support substrate (not shown).

The processor device 16 includes a CPU 70, a ROM (read-only memory) 72, a RAM (Random Access Memory) 74, a digital signal processor (DSP) 76, and a display control circuit 78.

The CPU 70 controls each part in the processor device 16 and comprehensively controls the entire endoscope system 10. Various programs and control data for controlling the operation of the processor device 16 are stored in the ROM 72. The program and data executed by the CPU 70 are temporarily stored in the RAM 74.

Based on the control of the CPU 70, the digital signal processing circuit 76 performs various signal processing such as color interpolation, color separation, color balance adjustment, gamma correction, and image enhancement processing on the image pickup signal input from the analog front end circuit 64. To generate image data. Further, the digital signal processing circuit 76 performs image recognition processing. The digital signal processing circuit 76 functions as an image processing unit. Further, the digital signal processing circuit 76 includes a function of an image recognition unit that performs image recognition processing.

The image data output from the digital signal processing circuit 76 is input to the display control circuit 78. The display control circuit 78 converts the image data input from the digital signal processing circuit 76 into a signal format corresponding to the display device 18 and displays it on the screen of the display device 18.

The display device 18 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. The display device 18 can display various setting information necessary for the processing of the processor device 16 or various information such as information indicating the processing result.

The display device 18 and the input device 19 function as a user interface. The input device 19 may be, for example, a keyboard, a mouse, a touch panel, operation buttons, a voice input device, or an appropriate combination thereof. The user can input various instructions and / or information using the input device 19. The processor device 16 can execute various processes according to the instruction and / or information input from the input device 19.

The light source device 14 generates light that illuminates the inside of the body cavity through the light guide 120 inserted in the electronic endoscope 12. The light source device 14 includes a first light source 100, a first light source drive circuit 101, a second light source 102, a second light source drive circuit 103, a CPU 104, and a combiner 105. The CPU 104 communicates with the CPU 70 of the processor device 16 to control the first light source drive circuit 101 and the second light source drive circuit 103.

The first light source drive circuit 101 causes the first light source 100 to emit light according to an instruction from the CPU 104. The second light source drive circuit 103 causes the second light source 102 to emit light according to the instruction from the CPU 104.

The first light source 100 is, for example, a laser diode that emits a blue laser having a wavelength of 445 nm. The first light source 100 is pulse-driven by the first light source drive circuit 101 to control the amount of light emitted. The second light source 102 is, for example, a laser diode that emits a blue laser having a wavelength of 405 nm. The second light source 102 is pulse-driven by the second light source drive circuit 103 to control the amount of light emitted. Blue light with a wavelength of 405 nm is used for special light observation.

The combined wave unit 105 combines the emitted lights of the first light source 100 and the second light source 102 and emits them to the incident end 121 of the light guide 120.

A phosphor 124 is provided between the emission end 122 of the light guide 120 and the illumination window 50 of the electronic endoscope 12. The blue laser light that has passed through the light guide 120 irradiates the phosphor 124 with an excited state, and a part of the blue laser light passes through the phosphor 124 and is emitted from the illumination window 50 as blue light.

The phosphor 124 is excited by blue laser light and emits a wide range of light (yellow as a color) from the wavelength range per boundary between blue and green to the red wavelength range in terms of the wavelength band of light. The yellow light and the blue light transmitted through the phosphor 124 are mixed to form white light, which illuminates the subject through the illumination window 50. The blue light transmitted through the phosphor 124 includes a part of the blue light emitted by the phosphor 124.

As described above, the phosphor 124 emits yellow light when irradiated with a blue laser light having a wavelength of 445 nm and transmits blue light having a wavelength of 445 nm, but is irradiated with a blue laser light having a wavelength of 405 nm. In some cases, it has the property of transmitting most of it.

That is, by controlling the mixing ratio of the blue laser light having a wavelength of 445 nm and the blue laser light having a wavelength of 405 nm, the ratio of the blue light transmitted through the phosphor 124 and the yellow light emitted by the phosphor 124 is controlled. Is possible.

A color image of the subject is reproduced by receiving the reflected light from the subject illuminated by the white light by using the image sensor 62.

When observing the inside of the body cavity using the endoscope system 10 configured as described above, the power of the electronic endoscope 12, the light source device 14, the processor device 16, and the display device 18 is turned on. Then, the insertion portion 20 of the electronic endoscope 12 is inserted into the body cavity, and the image in the body cavity captured by the image pickup element 62 is displayed on the screen of the display device 18 while illuminating the inside of the body cavity with the illumination light from the light source device 14. Will be observed at.

The endoscope system 10 of this example has a white light observation mode and a narrow band light observation mode. The white light observation mode is a mode in which an observation image having a natural color is displayed on the display device 18 by using a captured image obtained by imaging an observation target using white light as illumination light. An image obtained by imaging an observation target in the white light observation mode is called a "white light observation image". Illumination light may be paraphrased as "observation light".

In the narrow-band light observation mode, for example, visualization of blood vessels in a specific depth region of an observation target is emphasized by using an image signal obtained by imaging an observation target using narrow-band light of a specific wavelength band as illumination light. This mode generates an image and displays an image suitable for observing blood vessels on the display device 18. An image obtained by imaging an observation target in the narrow band light observation mode is called a "narrow band light observation image".

The endoscope system 10 may have a plurality of types of narrow-band light observation modes in which the types of wavelength bands of narrow-band light used or combinations thereof are different.

<< Overview of diagnostic support function in endoscope system 10 >>
The processor device 16 includes an image recognition unit that recognizes an image received from the electronic endoscope 12. For example, a convolutional neural network (CNN) is used for the image recognition process. The image recognition unit performs, for example, an image classification task of classifying lesions contained in an image into a plurality of categories according to NICE classification or JNET classification. "NICE" is an abbreviation for NBI (Narrow band imaging) International Colorectal Endoscopic. "JNET" is an abbreviation for "the Japan NBI Expert Team".

The NICE classification is a non-expanded NBI classification, and each of the three items of lesion color tone (Color), microvascular pattern (Vessels), and surface pattern (Surface pattern) is classified into Type 1, Type 2 and Type 3. To. Type 1 is a diagnostic index for hyperplastic lesions, Type 2 is a diagnostic index for adenoma to intramucosal cancer, and Type 3 is a diagnostic index for SM (submucosa) deep infiltration cancer. The JNET classification is a classification of NBI magnifying endoscopic findings for colorectal tumors. The JNET classification is classified into Type1, Type2A, Type2B, and Type3 for each item of "vessel pattern" and "surface pattern".

The image recognition unit may recognize two classifications, which are simply "cancerous" or "non-cancerous", in place of or in combination with detailed classification such as NICE classification.

In addition to recognition processing such as lesion classification, the image recognition unit also recognizes whether the input image is appropriate or inappropriate for image recognition. "Appropriate for image recognition" means that the image is suitable for the recognition process for classifying lesions, which is the main purpose of recognition. "Inappropriate for image recognition" means that the image is unsuitable for recognition such as classification of lesions, which is the main purpose. Examples of images that are unsuitable for recognition include blurred images, blurred images, images in which the subject is covered with water, and images with residues. An image that is unsuitable for recognition is called an "inappropriate image".

The image recognition unit recognizes two categories, for example, an image suitable for recognition or an image unsuitable for recognition, and when it recognizes that the image is suitable for recognition, the image recognition unit classifies the image as NICE classification. Execute the task. Alternatively, the image recognition unit may add a classification such as "an image unsuitable for recognition" to the recognition process of a classification task such as NICE classification.

When the recognition result is other than "an image unsuitable for recognition", the processor device 16 saves the recognition result. After that, the next image is received by the processor device 16, and image recognition is performed on the newly received image. When the recognition result of the newly received image is determined to be an "image unsuitable for recognition", the processor device 16 determines this recognition result (recognition result as "an image unsuitable for recognition"). Change to the saved recognition result.

The recognition result can be displayed on the display device 18. Here, when the recognition result is changed to the saved one (the previous recognition result that was saved), the recognition result should be displayed so that it can be recognized as the previous recognition result. Is preferable. As a specific example of "display that shows the previous recognition result", for example, a form in which an icon is displayed next to the information indicating the recognition result, or a form in which the display color of the information indicating the recognition result is changed, etc. There can be. The display device 18 is an example of a “display unit”. Further, the display device 18 is an example of a "notification unit" that notifies the recognition result of the image recognition unit.

<< Configuration of Medical Image Processing Device According to First Embodiment >>
The processor device 16 is an example of the medical image processing device according to the embodiment of the present invention. FIG. 4 is a block diagram showing a function of the medical image processing apparatus according to the first embodiment of the present invention. The medical image processing device 160 shown in FIG. 4 includes an image acquisition unit 162, an image recognition unit 164, a determination unit 166, a recognition result storage unit 168, a change processing unit 170, a display control unit 172, and a display unit 174. And.

The image acquisition unit 162 is an interface for acquiring time-series images IM1, IM2, IM3 ... Taken by using the electronic endoscope 12. The image acquisition unit 162 may be, for example, a connector terminal to which the video connector 42A is connected, or a signal input terminal of the digital signal processing circuit 76. Further, the image acquisition unit 162 may be a communication network terminal provided in the processor device 16, a media interface terminal for external storage media, a terminal for connecting an external device, or an appropriate combination thereof.

The image recognition unit 164 is a processing unit that performs image recognition from an image acquired via the image acquisition unit 162. The image recognition unit 164 includes a first recognition unit 164A and a second recognition unit 164B. The first recognition unit 164A recognizes whether the input image is appropriate or inappropriate for image recognition. The second recognition unit 164B performs image classification processing from the input image according to NIEC classification, JNET classification, or the like. The second recognition unit 164B may perform a process of detecting a region of interest such as a lesion region in place of or in combination with the image classification process. The recognition process performed by the first recognition unit 164A is referred to as "first recognition process". The recognition process performed by the second recognition unit 164B is referred to as "second recognition process". Each of the first recognition unit 164A and the second recognition unit 164B can be configured by using, for example, a convolutional neural network (CNN). The image recognition unit 164 is configured by using a trained model learned by machine learning.

When the recognition result obtained from the first recognition unit 164A is an "image suitable for recognition", the recognition process by the second recognition unit 164B is executed. The second recognition unit 164B extracts the feature amount from the image and performs at least one of the image classification, the detection of the region of interest, the segmentation, and the calculation of the similarity. For example, the second recognition unit 164B performs image classification according to a predetermined classification method such as NICE classification or JNET classification.

The recognition result storage unit 168 is a storage unit that stores the recognition result of the image recognition unit 164. The recognition result storage unit 168 may be the storage area of the RAM 74 shown in FIG. The recognition result storage unit 168 may be a storage area of a memory (not shown) other than the RAM 74. The recognition result stored in the recognition result storage unit 168 is sequentially updated with the latest information. Only the latest (most recent) recognition result needs to be saved in the recognition result storage unit 168.

The determination unit 166 determines whether or not to save the recognition result of the image recognition unit 164 according to the recognition result of the image recognition unit 164. The determination unit 166 of this example determines storage or non-preservation of the recognition result based on the recognition result from the first recognition unit 164A.

When the recognition result from the first recognition unit 164A is an "image suitable for recognition", the second recognition process by the second recognition unit 164B is executed. In this case, the determination unit 166 determines that the recognition result is to be saved, and the recognition result obtained from the second recognition unit 164B is saved in the recognition result storage unit 168. Further, the recognition result obtained from the second recognition unit 164B is displayed on the display unit 174 via the display control unit 172. The display control unit 172 corresponds to the display control circuit 78 shown in FIG. The display unit 174 may be a display device 18 (see FIGS. 1 and 2). Further, the display unit 174 may be a display device different from the display device 18.

On the other hand, when the recognition result from the first recognition unit 164A is an "image unsuitable for recognition", the second recognition process by the second recognition unit 164B is stopped. That is, the medical image recognized by the first recognition unit 164A as an image unsuitable for recognition is not processed by the second recognition unit 164B. In this case, the determination unit 166 determines that the recognition result is not saved (not saved).

The change processing unit 170 changes the recognition result of the image recognition unit 164 to the recognition result stored in the recognition result storage unit 168 according to the recognition result of the image recognition unit 164. The change processing unit 170 changes the recognition result of the image to which the determination unit 166 has given the determination that the recognition result is not stored to the recognition result stored in the recognition result storage unit 168.

The recognition result after the change by the change processing unit 170 is displayed on the display unit 174 via the display control unit 172.

The recognition result storage unit 168 stores the recognition result of the image in which the recognition result of the image recognition unit 164 is maintained as it is without receiving the change processing by the change processing unit 170. The change processing unit 170 changes the recognition result of the image recognition unit 164 to the latest (most recent) recognition result stored in the recognition result storage unit 168.

The function of the image recognition unit 164 can be realized by the digital signal processing circuit 76 described with reference to FIG. 2, the CPU 70, or a combination thereof. The functions of the determination unit 166 and the change processing unit 170 can be realized by the CPU 70.

<< Operation of the medical image processing device according to the first embodiment >>
FIG. 5 is a flowchart showing the operation of the medical image processing apparatus according to the first embodiment. The operation of the medical image processing device may be understood as a medical image processing method, an operating method of the medical image processing device, or an operating method of the processor device.

In step S12, the image acquisition unit 162 receives the input image. The input image here is a medical image including a subject image taken by using the electronic endoscope 12, and is one image of a time-series image sequentially taken in time series. For example, the input image is a one-frame image constituting a moving image. Step S12 is an example of the “image acquisition step”.

In step S14, the image recognition unit 164 performs the first recognition process on the input image.

In step S16, the CPU 70 determines whether or not the input image is an image suitable for the second recognition process based on the recognition result of the first recognition process. If the determination result in step S16 is "Yes determination", that is, if the input image is an image suitable for the second recognition process, the process proceeds to step S18. In step S18, the image recognition unit 164 performs the second recognition process. Step S14 and step S18 are examples of the "image recognition step".

In step S20, the CPU 70 obtains the recognition result from the image recognition unit 164.

In step S22, the CPU 70 stores the recognition result obtained in step S20 in the recognition result storage unit 168. Step S22 is an example of the “recognition result saving step”.

On the other hand, if the determination result in step S16 is "No determination", that is, if the input image is an image unsuitable for the second recognition process, the process proceeds to step S24.

In step S24, the CPU 70 acquires the recognition result stored in the recognition result storage unit 168. In step S26, the CPU 70 changes the recognition result of the first recognition process to the recognition result stored in the recognition result storage unit 168, and obtains the recognition result. Step S24 and step S26 are examples of "change processing steps".

The recognition result obtained in step S20 or step S26 is displayed on the display unit 174.

After step S22 or step S26, the flowchart of FIG. 5 ends. The processing of the flowchart of FIG. 5 is repeated for each input image acquired in time series.

According to such a configuration, when an image unsuitable for recognition is temporarily included in the images obtained in time series, the latest recognition result stored in the recognition result storage unit 168 is used. , Can present reasonable recognition results with high truthfulness.

As a result, it is possible to prevent an image having a low truthfulness (erroneous recognition result) from being generated from an image unsuitable for recognition and being provided to the user.

Further, even if the recognition result of an image unsuitable for recognition is changed, if an image suitable for recognition is obtained thereafter, the processes of steps S18 to S22 are performed, so that highly accurate recognition is possible.

<What to do if images that are unsuitable for recognition are acquired continuously for a certain period of time>
Although not shown in the flowchart of FIG. 5, the processor device 16 stores the recognition result of the image recognition unit 164 in the recognition result storage unit 168 when an image unsuitable for recognition is continuously acquired for a certain period of time. When the acquisition of the medical image changed to the recognition result is continued for a certain period of time, the recognition process of the image recognition unit 164 may be stopped. In this case, it is preferable to display a warning message or the like on the display unit 174 to call attention to the user. The display unit 174 is an example of the “notification unit”.

In the first embodiment shown in FIG. 4, an example in which the "first recognition process" and the "second recognition process" are carried out stepwise is shown, but instead of the configuration in which such a two-step recognition process is performed. Therefore, a configuration may be adopted in which the images are classified into a plurality of categories including the classification of "images unsuitable for recognition" in one recognition process.

<< Configuration of Medical Image Processing Device According to Second Embodiment >>
In the first embodiment shown in FIG. 4, an example in which the "first recognition process" and the "second recognition process" are carried out stepwise is shown, but instead of the configuration in which such a two-step recognition process is performed. Therefore, a configuration may be adopted in which the images are classified into a plurality of categories including the classification of "images unsuitable for recognition" in one recognition process.

FIG. 6 is a block diagram showing a function of the medical image processing apparatus according to the second embodiment of the present invention. Instead of the configuration shown in FIG. 4, the configuration shown in FIG. 6 can be adopted. In FIG. 6, elements having the same or similar configurations as those shown in FIG. 4 are designated by the same reference numerals. The difference from FIG. 4 will be described.

The image recognition unit 164 of the medical image processing device 160 shown in FIG. 6 is a processing unit that performs image recognition from an image acquired via the image acquisition unit 162. The image recognition unit 164 is a category classification in which the classification of "images unsuitable for recognition" is added to the recognition processing of the classification task according to the classification method such as NICE classification or JNET classification. The image recognition unit 164 performs image classification processing including classification of "images unsuitable for recognition" from the input image in one recognition process.

The determination unit 166 determines whether or not to save the recognition result based on the recognition result of the image recognition unit 164.

<< Operation of the medical image processing device according to the second embodiment >>
FIG. 7 is a flowchart showing the operation of the medical image processing apparatus according to the second embodiment. In FIG. 7, steps that are the same as or similar to the flowchart shown in FIG. 5 are given the same step numbers, and duplicate description will be omitted. The difference from FIG. 5 will be described.

The flowchart shown in FIG. 7 includes steps S15 and S17 instead of step S14, step S16 and step S18 in FIG.

In step S15, the image recognition unit 164 performs recognition processing on the input image. The recognition process performed here is, for example, an image classification process in which a category of "images unsuitable for recognition" is added to a classification category such as NICE classification or JNET classification.

In step S17, the CPU 70 determines whether or not the desired recognition has been achieved from the input image. The term "target recognition" as used herein means recognizing a category other than "images unsuitable for recognition". When the recognition result of "an image unsuitable for recognition" is obtained by the recognition process in step S15, the determination result in step S17 is "No determination". On the other hand, when the recognition process of step S15 obtains a recognition result of a category other than "an image unsuitable for recognition", the determination result of step S17 is "Yes determination".

If the determination result in step S17 is "Yes determination", the process proceeds to step S20.

In step S20, the CPU 70 obtains the recognition result of the image recognition unit 164.

In step S22, the CPU 70 stores the recognition result obtained in step S20 in the recognition result storage unit 168.

On the other hand, if the determination result in step S17 is "No determination", the process proceeds to step S24.

In step S24, the CPU 70 acquires the recognition result stored in the recognition result storage unit 168. In step S26, the CPU 70 changes the recognition result of the image recognition unit 164 to the recognition result stored in the recognition result storage unit 168, and obtains the recognition result.

The recognition result obtained in step S20 or step S26 is displayed on the display unit 174.

After step S22 or step S26, the flowchart of FIG. 7 ends. The processing of the flowchart of FIG. 7 is repeated for each input image acquired in time series.

<< Display example of recognition result >>
FIG. 8 is an example of the endoscopic diagnosis support screen displayed on the display unit 174. The window 300 illustrated in FIG. 8 includes an image display area 301 and a recognition result display area 310. FIG. 8 shows a display example when an image suitable for recognition is acquired. In the image display area 301, the image contents of the medical image 302 taken by the electronic endoscope 12 are displayed in real time. FIG. 8 shows an example in which the lesion region 303 is included in the image of the medical image 302. In addition, a plurality of lesion areas may be included in one image.

In the recognition result display area 310, for example, information indicating whether the recognition result is "cancerous" or "non-cancerous" is displayed. Further, in the recognition result display area 310, for example, the recognition result according to the NIEC classification is displayed.

FIG. 9 is another example of the endoscopic diagnosis support screen displayed on the display unit 174. FIG. 9 shows a display example when an image unsuitable for recognition is acquired. The medical image 302 displayed in the image display area 301 of the window 300 shown in FIG. 9 is, for example, a blurred image as an example of an image unsuitable for recognition. When such an unclear image is acquired, the recognition result obtained from the image recognition unit 164 is changed to the recognition result stored in the recognition result storage unit 168. In the recognition result display area 310, the changed recognition result, that is, the recognition result stored in the recognition result storage unit 168 is displayed.

At this time, next to the display of the recognition result, an icon 312 notifying the user that it is the previous recognition result is displayed.

When the image obtained from the electronic endoscope 12 returns to an image suitable for recognition, the screen of FIG. 9 returns to the screen of FIG.

<< Example of display indicating that the recognition result is stored in the recognition result storage unit 168 >>
The display indicating that the recognition result is stored in the recognition result storage unit 168 is a different color display in which the display color is different from the case where the recognition result of the image recognition unit 164 is displayed as it is, and a specific symbol is added. Any combination thereof may be used as long as it includes at least one display form of the symbol display, the character string display to which a specific character string is added, and the blinking display. The icon 312 shown in FIG. 9 is an example of symbol display. The different color display is not limited to the mode of changing the color of characters, but also includes the mode of changing the background color.

<< Advantages of Embodiments of the Present Invention >>
According to the configuration of the above-described embodiment, the following effects can be obtained.

(1) When a time-series medical image temporarily contains an image unsuitable for recognition such as a blurred image, the truth is obtained by using the latest recognition result stored in the recognition result storage unit 168. It is possible to present a highly reliable and reasonable recognition result.

(2) As a result, it is possible to avoid generating a recognition result (erroneous recognition result) having low truthfulness from an image unsuitable for recognition and providing it to the user.

(3) Even if the recognition result for an image unsuitable for recognition is changed, if an image suitable for recognition is obtained thereafter, the processes of steps S20 to S22 are performed, so that highly accurate recognition is possible. Is.

(4) According to the embodiment of the present invention, it is possible to provide useful diagnostic support for doctors and the like.

<< Third Embodiment: Application example to medical information management system >>
The medical image processing device according to the present invention is not limited to the form applied to the processor device 16 of the endoscope system 10 illustrated in FIG. 1, and various applications are possible. For example, a medical image processing device can be applied to a medical information management system that manages various medical information including endoscopic images.

FIG. 10 is a block diagram showing a configuration example of a medical information management system. The medical information management system 200 includes an image acquisition terminal 202, an image storage server 204, an information management device 210, a display device 218, and an input device 219. Each of the image acquisition terminal 202, the image storage server 204, and the information management device 210 is connected to the telecommunication line 230. The term "connection" is not limited to wired connection, but also includes the concept of wireless connection.

The telecommunication line 230 may be a local area network or a wide area network. The telecommunication line 230 is composed of an appropriate combination of wired and wireless.

The processor device 16 of the endoscope system 10 is connected to the telecommunication line 230. The medical image generated by the processor device 16 is taken into at least one of the image acquisition terminal 202, the image storage server 204, and the information management device 210 via the telecommunication line 230. For example, the medical image generated by the processor device 16 is sent to the image acquisition terminal 202. The image acquisition terminal 202 receives a medical image from the processor device 16.

The image acquisition terminal 202 sends the medical image received from the processor device 16 to the image storage server 204. Further, the image acquisition terminal 202 may transmit the medical image received from the processor device 16 to the information management device 210.

The image storage server 204 serves as a storage device for storing a database of various medical images. Instead of the image storage server 204, cloud storage may be used. In addition to the medical image, the image storage server 204 may store image analysis results such as a region of interest (region of interest) included in the medical image, the presence or absence of an object of interest, and the result of image classification.

Although one endoscope system 10 is shown in FIG. 10, a plurality of endoscope systems can be connected to the telecommunication line 230. Further, the telecommunication line 230 may be connected not only to an endoscopic system but also to another medical image diagnostic device such as an ultrasonic diagnostic device. The ultrasonic image obtained from the ultrasonic diagnostic apparatus is an example of a "medical image".

The information management device 210 is realized by, for example, computer hardware and software. A display device 218 and an input device 219 are connected to the information management device 210. The information management device 210 may include a part or all of the functions of the medical image processing device 160 shown in FIG. 4 or FIG. For example, the information management device 210 includes functions of an image acquisition unit 162, an image recognition unit 164, a determination unit 166, a recognition result storage unit 168, a change processing unit 170, and a display control unit 172. The display device 218 can function as a display unit 174. The function of the information management device 210 can be realized by one or a plurality of computers, and can also be realized by cloud computing. The information management device 210 may include the function of the image storage server 204. Further, the image acquisition terminal 202 can function as an image acquisition unit 162. It is also possible to omit the image acquisition terminal 202, in which case the image storage server 204 and / or the information management device 210 is configured to acquire medical images from the processor device 16.

According to the medical information management system 200 configured as described above, not only medical images obtained in real time from the electronic endoscope 12 but also moving images stored in the image storage server 204 can be reproduced. It is possible to process the flowchart described with reference to FIG. 5 or 7, and to display the recognition result as described with reference to FIGS. 8 and 9.

The information management device 210 may be installed in, for example, an operating room, an examination room, a conference room, or the like in a hospital, or may be installed in a medical institution, a research institution, or the like of an out-of-hospital facility. The information management device 210 may be a workstation that supports medical examination, treatment, diagnosis, and the like, or may be a business support device that supports medical work. The business support device may have a function of accumulating clinical information, supporting the creation of diagnostic documents, supporting the creation of reports, and the like.

<< Hardware configuration of each processing unit and control unit >>
Various processes such as the image acquisition unit 162, the image recognition unit 164, the determination unit 166, the recognition result storage unit 168, the change processing unit 170, and the display control unit 172 of the medical image processing unit 160 described with reference to FIGS. 4 and 6 are performed. The hardware structure of the processing unit to be executed is various processors as shown below.

Various processors include CPUs (Central Processing Units) and FPGAs (Field Programmable Gate Arrays), which are general-purpose processors that execute programs and function as various processing units. A programmable logic device (PLD), a dedicated electric circuit which is a processor having a circuit configuration specially designed for executing a specific process such as an ASIC (Application Specific Integrated Circuit), and the like are included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs or a combination of a CPU and an FPGA. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or a server. There is a form in which the processor functions as a plurality of processing units. Secondly, as typified by System On Chip (SoC), there is a form in which a processor that realizes the functions of the entire system including a plurality of processing units with one IC (Integrated Circuit) chip is used. is there. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

<< Modification 1 >>
The electronic endoscope is not limited to a flexible endoscope, and may be a rigid endoscope or a capsule endoscope. Further, the device for generating a time-series medical image including a subject image is not limited to an electronic endoscope, and may be, for example, an ultrasonic diagnostic device.

<< Modification 2 >>
The medical image processing device of the present disclosure can be used as a diagnostic support device that supports medical examination, treatment, diagnosis, etc. by a doctor or the like. The term "diagnostic support" includes the concept of diagnostic support and / or treatment support.

<< About the observation light of the endoscopic system >>
As the observation light, white light, light in one or a plurality of specific wavelength bands, or light in various wavelength bands according to the purpose of observation such as a combination thereof is selected. White light is light in a white wavelength band or light in a plurality of wavelength bands. The "specific wavelength band" is a band narrower than the white wavelength band. Specific examples regarding a specific wavelength band are shown below.

<First example>
A first example of a particular wavelength band is, for example, the blue or green band in the visible range. The wavelength band of the first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example is in the wavelength band of 390 nm or more and 450 nm or less or a wavelength of 530 nm or more and 550 nm or less. It has a peak wavelength in the band.

<Second example>
A second example of a particular wavelength band is, for example, the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nm or more and 615 nm or less or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example has a wavelength of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. It has a peak wavelength in the band.

<Third example>
The third example of a specific wavelength band includes a wavelength band in which the extinction coefficient differs between oxidized hemoglobin and reduced hemoglobin, and the light in the third example has a peak wavelength in a wavelength band in which the extinction coefficient differs between oxidized hemoglobin and reduced hemoglobin. Has. The wavelength band of the third example includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, a wavelength band of 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example is 400 ± 10 nm. It has a peak wavelength in the wavelength band of 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less.

<4th example>
The fourth example of the specific wavelength band is used for observing the fluorescence emitted by the fluorescent substance in the living body (fluorescence observation), and the wavelength band of the excitation light that excites the fluorescent substance, for example, 390 nm to 470 nm.

<Example 5>
A fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example is within the wavelength band of 790 nm or more and 820 nm or less or a wavelength of 905 nm or more and 970 nm or less. It has a peak wavelength in the band.

<< Switching of observation light >>
As the type of light source, a laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof may be adopted. The type and wavelength of the light source, the presence or absence of a filter, etc. are preferably configured according to the type of subject, the purpose of observation, etc., and when observing, the wavelength of the illumination light is combined according to the type of subject, the purpose of observation, etc. And / or switching is preferred. When switching the wavelength, for example, by rotating a disk-shaped filter (rotary color filter) arranged in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of the emitted light is switched. May be good.

The image sensor used in the electronic endoscope is not limited to the color image sensor in which the color filter is provided for each pixel, and may be a monochrome image sensor. When a monochrome image sensor is used, the wavelength of the illumination light can be sequentially switched to perform surface-sequential (color-sequential) imaging. For example, the wavelength of the emitted illumination light may be sequentially switched between purple, blue, green, and red, or a wide band light (white light) is irradiated and emitted by a rotary color filter (red, green, blue, etc.). The wavelength of the illumination light to be used may be switched. Further, the wavelength of the illumination light emitted by the rotary color filter by irradiating one or a plurality of narrow band lights may be switched. The narrow band light may be infrared light having two or more wavelengths having different wavelengths.

<< Example of generating a special light image >>
The processor device 16 may generate a special optical image having information in a specific wavelength band based on a normal optical image obtained by imaging with white light. The generation referred to here includes the concept of "acquisition". In this case, the processor device 16 functions as a special optical image acquisition unit. The processor device 16 transmits signals in a specific wavelength band to red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y), which are usually contained in an optical image. ) Can be obtained by performing an operation based on the color information.

<< Example of feature image generation >>
The processor device 16 is a medical image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band, and a special optical image obtained by irradiating light in a specific wavelength band. An operation based on at least one of the above can be used to generate a feature image. A feature image is a form of medical image.

<< About the program that makes the computer realize the function of the medical image processing device >>
A program for realizing the functions of the medical image processing apparatus described in the above-described embodiment on a computer is recorded on an optical disk, a magnetic disk, or a computer-readable medium such as a semiconductor memory or other tangible non-temporary information storage medium. It is possible to provide a program through an information storage medium. Further, instead of the mode in which the program is stored and provided in such a tangible non-temporary information storage medium, it is also possible to provide the program signal as a download service by using a telecommunication line such as the Internet.

It is also possible to provide a part or all of the functions of the medical image processing apparatus described in the above-described embodiment as an application server, and provide a service that provides the processing function through a telecommunication line.

<< Combination of Embodiments and Modifications >>
The components described in the above-described embodiment and the components described in the modified examples can be used in combination as appropriate, or some components can be replaced.

<< Additional notes >>
The present specification includes the disclosure of the constituent inventions described below, in addition to the above-described embodiments and modifications.

(Appendix 1)
The medical image processing apparatus includes a medical image analysis processing unit and a medical image analysis result acquisition unit, and the medical image analysis processing unit is an area of interest based on the feature amount of pixels of a medical image. The medical image analysis result acquisition unit is a medical image processing device that detects an area and acquires the analysis result of the medical image analysis processing unit.

The medical image analysis processing unit may include an image recognition unit.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of a noteworthy object based on the feature amount of the pixel of the medical image, and the medical image analysis result acquisition unit acquires the analysis result of the medical image analysis processing unit. apparatus.

(Appendix 3)
The medical image analysis result acquisition unit acquires from a recording device that records the analysis result of the medical image, and the analysis result is either a notable area included in the medical image or the presence or absence of a notable object. A medical imaging device that is, or both.

(Appendix 4)
A medical image is a medical image processing apparatus that is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band.

(Appendix 5)
A medical image is an image obtained by irradiating light in a specific wavelength band, and the specific wavelength band is a narrower band than the white wavelength band.

(Appendix 6)
A medical image processing device in which a specific wavelength band is a blue or green band in the visible range.

(Appendix 7)
A medical treatment in which a specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and light in a specific wavelength band has a peak wavelength in a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)
A medical image processing device in which a specific wavelength band is a red band in the visible range.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which the oxidation hemoglobin and the reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which the oxidation hemoglobin and the reduced hemoglobin have different absorption coefficients. Medical image processing device.

(Appendix 11)
The specific wavelength band includes 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or 600 nm or more and 750 nm or less, and the light in the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ±. A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or 600 nm or more and 750 nm or less.

(Appendix 12)
The medical image is an in-vivo image of the inside of the living body, and the in-vivo image is a medical image processing device having information on fluorescence emitted by a fluorescent substance in the living body.

(Appendix 13)
Fluorescence is a medical image processing device obtained by irradiating a living body with excitation light having a peak of 390 nm or more and 470 nm or less.

(Appendix 14)
A medical image is an in-vivo image of the inside of a living body, and a specific wavelength band is a wavelength band of infrared light.

(Appendix 15)
A medical image in which a specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and light in a specific wavelength band has a peak wavelength in a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. Processing equipment.

(Appendix 16)
The medical image acquisition unit acquires a special optical image having information in a specific wavelength band based on a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in the white band. A medical image processing device equipped with an optical image acquisition unit and a medical image is a special optical image.

(Appendix 17)
A medical image processing device that obtains a signal in a specific wavelength band by calculation based on RGB or CMY color information included in a normal optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band and a special light image obtained by irradiating light in a specific wavelength band. A medical image processing device including a feature amount image generation unit that generates a feature amount image, and the medical image is a feature amount image.

(Appendix 19)
The medical image processing apparatus according to any one of Supplementary note 1 to Supplementary note 18, and an endoscope that acquires an image by irradiating at least one of light in a white wavelength band or light in a specific wavelength band. An endoscopic device comprising.

(Appendix 20)
A diagnostic support device including the medical image processing device according to any one of Supplementary Note 1 to Supplementary Note 18.

(Appendix 21)
A medical business support device including the medical image processing device according to any one of Supplementary Note 1 to Supplementary Note 18.

[Other]
In the embodiment of the present invention described above, the constituent requirements can be appropriately changed, added, or deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made by a person having ordinary knowledge in an equivalent related field within the technical idea of the present invention.

10 Endoscope system 12 Electronic endoscope 14 Light source device 16 Processor device 18 Display device 19 Input device 20 Insertion part 22 Tip hard part 22A Tip surface 24 Curved part 26 Flexible part 30 Operation part 31 Angle knob 32 Air supply / water supply button 33 Suction button 34 Mode selector switch 35 Zoom operation unit 36 Treatment tool introduction port 40 Universal cord 42 Connector 42A Video connector 42B Light guide connector 42C Water supply connector 44 Water supply tank 50 Lighting window 52 Observation window 54 Force outlet 56 Air supply water supply nozzle 60 Objective Optical system 62 Imaging element 64 Analog front-end circuit 65 Timing generator 66 CPU
68 Imaging chip 70 CPU
72 ROM
74 RAM
76 Digital signal processing circuit 78 Display control circuit 100 1st light source 101 1st light source drive circuit 102 2nd light source 103 2nd light source drive circuit 104 CPU
105 Convergence part 120 Light guide 121 Incident end 122 Exit end 124 Phosphorant 160 Medical image processing device 162 Image acquisition unit 164 Image recognition unit 164A First recognition unit 164B Second recognition unit 166 Judgment unit 168 Recognition result storage unit 170 Change processing unit 172 Display control unit 174 Display unit 200 Medical information management system 202 Image acquisition terminal 204 Image storage server 210 Information management device 218 Display device 219 Input device 230 Telecommunications line 300 Window 301 Image display area 302 Medical image 303 Disease area 310 Recognition result display area 312 Icons IM1, IM2, IM3 Images S12 to S26 Processing steps in the medical image processing device

Claims (26)

An image acquisition unit that acquires time-series medical images including the subject image,
An image recognition unit that performs image recognition from the medical image and
A recognition result storage unit that stores the recognition result of the image recognition unit, and a recognition result storage unit.
A change processing unit that changes the recognition result of the image recognition unit to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.
It is a medical image processing device equipped with
The image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition.
When the image recognition unit recognizes that the image is unsuitable for recognition, the change processing unit changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit.
Medical image processing equipment. A determination unit for determining whether or not to save the recognition result of the image recognition unit according to the recognition result of the image recognition unit is provided.
The medical image processing apparatus according to claim 1, wherein the recognition result of the image recognition unit is stored in the recognition result storage unit according to the determination result of the determination unit. The change processing unit changes the recognition result of the image recognition unit for the medical image to which the determination unit has given a determination that the recognition result is not saved to the recognition result stored in the recognition result storage unit. The medical image processing apparatus according to claim 2. Prior Symbol judging unit, according to claim, wherein the image recognition unit when recognizing that the inappropriate image recognition, it is determined that the recognition result for the medical image is recognized that the unsuitable images and nonconservative The medical image processing apparatus according to 3. Wherein said image recognition unit recognizes that the inappropriate image recognition medical image recognition result for the medical image processing apparatus according to claim 1 which is non-stored in the recognition result storage unit. The medical image processing apparatus according to any one of claims 1 to 5 , wherein the unsuitable image is an image in which the subject is blurred. The medical image processing apparatus according to any one of claims 1 to 5 , wherein the unsuitable image is an image in which the subject is blurred. The medical image processing apparatus according to any one of claims 1 to 5 , wherein the unsuitable image is an image in which the subject is covered with water. The medical image processing apparatus according to any one of claims 1 to 5 , wherein the unsuitable image is an image having a residue on the subject. The medical image processing apparatus according to any one of claims 1 to 9 , wherein the image recognition performed by the image recognition unit includes a task of detecting a region of interest. The medical image processing apparatus according to any one of claims 1 to 10 , wherein the image recognition performed by the image recognition unit includes image classification. The image recognition unit includes a first recognition unit that recognizes whether or not the medical image acquired from the image acquisition unit is an image unsuitable for recognition.
A second recognition unit that detects a region of interest from the medical image acquired from the image acquisition unit and processes at least one of the image classifications of the medical image.
Including
The medical image according to any one of claims 1 to 11 , wherein the medical image recognized by the first recognition unit as an image unsuitable for recognition is not processed by the second recognition unit. Processing equipment. The image recognition unit performs the image recognition for each of the medical images acquired in time series.
The medical image processing apparatus according to any one of claims 1 to 12 , which outputs a recognition result for each medical image. Any of claims 1 to 13 in which the recognition result of the medical image in which the recognition result of the image recognition unit is maintained is stored in the recognition result storage unit without receiving the change processing by the change processing unit. The medical image processing apparatus according to paragraph 1. The medical image processing apparatus according to any one of claims 1 to 14 , wherein the change processing unit changes the recognition result of the image recognition unit to the latest recognition result stored in the recognition result storage unit. A display unit for displaying the recognition result of the image recognition unit is provided.
The medical image processing apparatus according to any one of claims 1 to 15 , wherein the recognition result stored in the recognition result storage unit is displayed on the display unit according to the recognition result of the image recognition unit. The medical treatment according to claim 16 , wherein the display unit displays the recognition result stored in the recognition result storage unit when displaying the recognition result stored in the recognition result storage unit. Image processing device. The display indicating that the recognition result is stored in the recognition result storage unit is a different color display in which the display color is different from the case where the recognition result of the image recognition unit is displayed as it is, and a specific symbol is added. The medical image processing apparatus according to claim 17 , which includes at least one display form of a symbol display, a character string display to which a specific character string is added, and a blinking display. From claim 1, the recognition process of the image recognition unit is stopped when the acquisition of the medical image obtained by changing the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit continues for a certain period of time. The medical image processing apparatus according to any one of 18. The medical image processing apparatus according to any one of claims 1 to 19 , wherein the medical image is an endoscopic image obtained from an endoscope or an ultrasonic image obtained from an ultrasonic diagnostic apparatus. An image acquisition step to acquire a time-series medical image including a subject image,
An image recognition step for performing image recognition from the medical image and
A recognition result saving step for saving the recognition result of the image recognition step in the recognition result saving unit, and
A change processing step that changes the recognition result of the image recognition step to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition step.
The medical image processing method comprising,
The image recognition step includes a task of recognizing that the medical image acquired in the image acquisition step is an image unsuitable for recognition.
When the image recognition step recognizes that the image is unsuitable for recognition, the change processing step changes the recognition result of the image recognition step to the recognition result stored in the recognition result storage unit.
Medical image processing method . An electronic endoscope that photographs the inside of the body cavity,
An endoscope system including a processor device for processing an image signal obtained from the electronic endoscope.
The processor device is
An image acquisition unit that acquires a time-series medical image including a subject image taken by using the electronic endoscope, and an image acquisition unit.
An image recognition unit that performs image recognition from the medical image and
A recognition result storage unit that stores the recognition result of the image recognition unit, and a recognition result storage unit.
A change processing unit that changes the recognition result of the image recognition unit to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.
Bei to give a,
The image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition.
When the image recognition unit recognizes that the image is unsuitable for recognition, the change processing unit changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit.
Endoscopic system. A processor device that processes image signals obtained from an electronic endoscope.
An image acquisition unit that acquires a time-series medical image including a subject image taken by using the electronic endoscope, and an image acquisition unit.
An image recognition unit that performs image recognition from the medical image and
A recognition result storage unit that stores the recognition result of the image recognition unit, and a recognition result storage unit.
A change processing unit that changes the recognition result of the image recognition unit to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.
Bei to give a,
The image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition.
The change processing unit is a processor device that changes the recognition result of the image recognition unit to a recognition result stored in the recognition result storage unit when the image recognition unit recognizes that the image is unsuitable for recognition. An image acquisition unit that acquires time-series medical images including the subject image,
An image recognition unit that performs image recognition from the medical image and
A recognition result storage unit that stores the recognition result of the image recognition unit, and a recognition result storage unit.
A change processing unit that changes the recognition result of the image recognition unit to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit.
A notification unit that notifies the recognition result of the image recognition unit, and
It is a diagnostic support device equipped with
The image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition.
When the image recognition unit recognizes that the image is unsuitable for recognition, the change processing unit changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit.
Diagnostic support device. On the computer
An image acquisition step to acquire a time-series medical image including a subject image,
An image recognition step for performing image recognition from the medical image and
A recognition result saving step for saving the recognition result of the image recognition step in the recognition result saving unit, and
A change processing step that changes the recognition result of the image recognition step to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition step.
A program to be executed by the,
The image recognition step includes a task of recognizing that the medical image acquired in the image acquisition step is an image unsuitable for recognition.
When the image recognition step recognizes that the image is unsuitable for recognition, the change processing step changes the recognition result of the image recognition step to the recognition result stored in the recognition result storage unit.
program. A non-temporary, computer-readable recording medium when the instructions stored in the recording medium are read by a computer.
An image acquisition step to acquire a time-series medical image including a subject image,
An image recognition step for performing image recognition from the medical image and
A recognition result saving step for saving the recognition result of the image recognition step in the recognition result saving unit, and
A change processing step that changes the recognition result of the image recognition step to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition step.
Is a recording medium that causes a computer to execute
The image recognition step includes a task of recognizing that the medical image acquired in the image acquisition step is an image unsuitable for recognition.
When the image recognition step recognizes that the image is unsuitable for recognition, the change processing step changes the recognition result of the image recognition step to the recognition result stored in the recognition result storage unit.
recoding media.

Images