JP7383105B2 - Medical image processing equipment and endoscope systems - Google Patents

Description

The present invention relates to a medical image processing device and method, an endoscope system, a processor device, a diagnosis support device, and a program, and more particularly to an image processing technique for performing image recognition using a medical image obtained by photographing an observation target.

In the medical field, tests using endoscope systems are being performed. In recent years, systems have been known that recognize lesions included in medical images by using image analysis techniques. In a typical endoscope system, a camera at the tip of a scope is used to photograph a body cavity, and a processor receives the photographed observation image and displays it on a monitor. At this time, the processor device may perform recognition processing on the received image and notify the user of the recognition result.

For example, if a lesion is photographed, the processor device recognizes whether it is cancerous or non-cancerous from image analysis, and notifies the user of the recognition result. A system with such functions supports decision-making by doctors and others regarding diagnostic findings.

Japanese Patent Application Publication No. 2004-180932 JP2015-204960A Japanese Patent Application Publication No. 2014-030488

However, for example, due to the effects of at least one of scope operation, fetal movements, and debris, the image often becomes blurred or blurred, or the lesion site is hidden by the debris. Furthermore, even when a water supply device is used to supply water to the observation site in order to secure the field of view, etc., the image may become blurred or the lesion site may be covered with water due to the influence of the water. Accurately recognize lesions, etc. from such "blurred images," "blurred images," "images containing residue," or "images with water on the subject" through image analysis. is difficult and may give different recognition results. This problem is particularly problematic in a system that performs recognition processing for each image obtained in time series, such as performing recognition processing for each frame of a moving image obtained during an examination, and notifies the user of the recognition results in real time.

In addition, similar problems exist not only in endoscope systems but also in systems that perform recognition processing from medical images obtained in time series, such as ultrasound diagnostic devices.

The present invention has been made in view of these circumstances, and when images unsuitable for image recognition are included in images obtained in time series, the recognition results obtained from the images unsuitable for image recognition are displayed to the user. It is an object of the present invention to provide a medical image processing device and method, an endoscope system, a processor device, a diagnostic support device, and a program that can avoid being provided.

In order to solve the problem, the following invention aspects are provided.

The medical image processing device according to aspect 1 includes an image acquisition unit that acquires time-series medical images including a subject image, an image recognition unit that performs image recognition from the medical images, and a recognition result that stores recognition results of the image recognition unit. The image recognition apparatus includes a storage section and a change processing section that changes the recognition result of the image recognition section to the recognition result stored in the recognition result storage section in accordance with the recognition result of the image recognition section.

According to aspect 1, recognition results of images unsuitable for image recognition, such as blurred images, among time-series medical images are changed to recognition results of other images stored in the recognition result storage unit. . The recognition result storage unit can store recognition results for images other than images unsuitable for image recognition. This makes it possible to avoid presenting inaccurate recognition results and to present valid recognition results in a time series.

The time-series medical images may be moving images, or may be still images taken at specific time intervals, such as continuous shooting or interval shooting.

Aspect 2 is the medical image processing apparatus of Aspect 1, including a determination unit that determines whether or not to save the recognition result of the image recognition unit according to the recognition result of the image recognition unit, and according to the determination result of the determination unit, This is a medical image processing device in which recognition results of an image recognition unit are stored in a recognition result storage unit.

The determination unit determines that recognition results of images that are unsuitable for image recognition, such as blurred images, are not saved in the recognition result storage unit (judgment that they are not saved), and stores images other than images that are unsuitable for image recognition. It is possible to determine whether the recognition result is to be stored in the recognition result storage unit.

Aspect 3 is the medical image processing device of Aspect 2, wherein the change processing unit stores the recognition result of the image recognition unit for the medical image for which the determination unit has determined that the recognition result is not saved, in the recognition result storage unit. This is a medical image processing device that changes the recognition results to the current recognition result.

Aspect 4 is the medical image processing apparatus of Aspect 3, in which the image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition, and the determination unit includes a task in which the image recognition unit recognizes that the medical image is an image unsuitable for recognition. This medical image processing apparatus determines, when an image is recognized to be unsuitable for recognition, that the recognition result of the medical image recognized as an unsuitable image is not saved.

Aspect 5 is the medical image processing device according to any one of Aspects 1 to 3, wherein the image recognition unit includes a task of recognizing that the medical image acquired from the image acquisition unit is an image unsuitable for recognition. The change processing unit is a medical image processing device that changes the recognition result of the image recognition unit to the recognition result stored in the recognition result storage unit when the image recognition unit recognizes that the image is unsuitable for recognition.

Aspect 6 is the medical image processing apparatus according to Aspect 5, in which the recognition result of a medical image that the image recognition unit recognizes as an image unsuitable for recognition is not stored in the recognition result storage unit. be.

Aspect 7 is a medical image processing apparatus according to any one of aspects 4 to 6, in which the inappropriate image is an image in which a subject is blurred.

Aspect 8 is a medical image processing apparatus according to any one of aspects 4 to 6, in which the inappropriate image is an image in which the subject is blurred.

Aspect 9 is a medical image processing apparatus according to any one of aspects 4 to 6, in which the inappropriate image is an image in which the subject is covered with water.

Aspect 10 is a medical image processing apparatus according to any one of aspects 4 to 6, in which the inappropriate image is an image in which a subject has a residue.

Aspect 11 is a medical image processing apparatus according to any one of Aspects 1 to 10, in which image recognition performed by the image recognition unit includes a task of detecting a region of interest.

The region of interest may be, for example, a lesion region.

Aspect 12 is a medical image processing apparatus according to any one of Aspects 1 to 11, in which image recognition performed by the image recognition unit includes image classification.

Aspect 13 is a medical image processing apparatus according to any one of Aspects 1 to 12, in which the image recognition unit recognizes whether or not the medical image acquired from the image acquisition unit is an image unsuitable for recognition. the first recognition unit, and a second recognition unit that performs at least one of detecting a region of interest from the medical image acquired from the image acquisition unit and classifying the medical image; This is a medical image processing apparatus in which a second recognition unit does not perform processing on a medical image that is recognized as an image unsuitable for recognition.

Aspect 14 is the medical image processing device according to any one of Aspects 1 to 13, wherein the image recognition unit performs image recognition for each medical image acquired in time series and outputs a recognition result for each medical image. It is a medical image processing device.

Aspect 15 is a medical image processing apparatus according to any one of Aspects 1 to 14, in which a recognition result of a medical image in which the recognition result of the image recognition unit is maintained without being subjected to change processing by the change processing unit. This is a medical image processing device that is stored in a recognition result storage unit.

Aspect 16 is the medical image processing device according to any one of Aspects 1 to 15, in which the change processing unit changes the recognition result of the image recognition unit to the latest recognition result stored in the recognition result storage unit. It is a medical image processing device.

It is preferable that the recognition results stored in the recognition result storage section be updated to the latest ones each time, and it is sufficient that only the most recent recognition results are stored in the recognition result storage section.

Aspect 17 is the medical image processing apparatus according to any one of Aspects 1 to 16, including a display unit that displays the recognition result of the image recognition unit, and in accordance with the recognition result of the image recognition unit, a recognition result storage unit. This is a medical image processing device in which stored recognition results are displayed on a display unit.

Aspect 18 is the medical image processing device of Aspect 17, when the display unit displays the recognition result stored in the recognition result storage unit, the display unit indicates that the recognition result is stored in the recognition result storage unit. This is a medical image processing device that performs display.

Aspect 19 is the medical image processing device of Aspect 18, in which the display indicating that the recognition result is stored in the recognition result storage section is displayed in a different color than when the recognition result of the image recognition section is displayed as it is. The medical image processing apparatus includes at least one display form among a different color display, a symbol display with a specific symbol added, a character string display with a specific character string added, and a blinking display.

Aspect 20 is a medical image processing apparatus according to any one of Aspects 1 to 19, in which the acquisition of medical images in which the recognition result of the image recognition unit is changed to the recognition result stored in the recognition result storage unit continues for a certain period of time. This is a medical image processing device that stops the recognition processing of the image recognition unit when

Aspect 21 is the medical image processing device according to any one of Aspects 1 to 20, wherein the medical image is an endoscopic image obtained from an endoscope or an ultrasound image obtained from an ultrasound diagnostic device. It is a processing device.

The medical image processing method according to aspect 22 includes an image acquisition step of acquiring time-series medical images including a subject image, an image recognition step of performing image recognition from the medical images, and a recognition result storage unit that stores the recognition results of the image recognition step. and a change processing step of changing the recognition result of the image recognition step to the recognition result stored in the recognition result storage section according to the recognition result of the image recognition step.

In the medical image processing method of aspect 22, the same items as those specified in aspects 2 to 21 can be combined as appropriate. In this case, elements of a processing unit or a functional unit as a means for carrying out a specified process or operation in the medical image processing apparatus can be understood as an element of a step (process) of the corresponding process or operation. Furthermore, the medical image processing method of aspect 22 may be understood as an operating method of a medical image processing apparatus.

An endoscope system according to aspect 23 is an endoscope system including an electronic endoscope that photographs the inside of a body cavity, and a processor device that processes image signals obtained from the electronic endoscope, the processor device comprising: , an image acquisition unit that acquires time-series medical images including subject images captured using an electronic endoscope; an image recognition unit that performs image recognition from the medical images; and a recognition unit that saves the recognition results of the image recognition unit. The image recognition apparatus includes a result storage section and a change processing section that changes the recognition result of the image recognition section to the recognition result stored in the recognition result storage section in accordance with the recognition result of the image recognition section.

In the endoscope system of aspect 23, the same items as those specified in aspects 2 to 21 can be combined as appropriate.

The processor device according to aspect 24 is a processor device that processes image signals obtained from an electronic endoscope, and is an image acquisition device that acquires time-series medical images including a subject image photographed using the electronic endoscope. an image recognition section that performs image recognition from medical images; a recognition result storage section that stores the recognition results of the image recognition section; and a change processing unit that changes the recognition result stored in the storage unit.

In the processor device of aspect 24, the same items as those specified in aspects 2 to 21 can be combined as appropriate.

The diagnosis support device according to aspect 25 includes an image acquisition section that acquires time-series medical images including a subject image, an image recognition section that performs image recognition from the medical images, and a recognition result storage that stores recognition results of the image recognition section. a change processing unit that changes the recognition result of the image recognition unit to a recognition result stored in the recognition result storage unit according to the recognition result of the image recognition unit; and a notification that notifies the recognition result of the image recognition unit. It is equipped with a section and a section.

In the diagnosis support device of aspect 25, the same items as those specified in aspects 2 to 21 can be combined as appropriate. The display unit that displays the recognition result of the image recognition unit can be a form of a notification unit.

The program according to aspect 26 includes an image acquisition step for acquiring time-series medical images including a subject image, an image recognition step for performing image recognition from the medical images, and a recognition result storage section for storing the recognition results of the image recognition step. A recognition result storage step for storing the recognition result in the recognition result storage section, and a changing processing step for changing the recognition result of the image recognition step to the recognition result stored in the recognition result storage section in accordance with the recognition result of the image recognition step.

In the program of aspect 26, items similar to those specified in aspects 2 to 21 can be combined as appropriate. In that case, elements of the processing unit or functional unit that are responsible for the specified processing or operation in the medical image processing device can be understood as program elements that realize the steps (functions) of the corresponding processing or operation. can.

According to the present invention, when an image unsuitable for image recognition is included in images obtained in time series, it is possible to avoid providing a user with a recognition result obtained from an image unsuitable for image recognition. It is possible.

FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. FIG. 2 is a front view showing the distal end surface of the rigid distal end portion of the electronic endoscope. FIG. 3 is a block diagram showing the control system of the endoscope system. FIG. 4 is a block diagram showing the functions of the medical image processing apparatus according to the first embodiment of the present invention. FIG. 5 is a flowchart showing the operation of the medical image processing apparatus according to the first embodiment. FIG. 6 is a block diagram showing the functions of a medical image processing apparatus according to a second embodiment of the present invention. FIG. 7 is a flowchart showing the operation of the medical image processing apparatus according to the second embodiment. FIG. 8 is a diagram showing an example of an endoscopic diagnosis support screen that displays recognition results. FIG. 9 is a diagram showing another example of the endoscopic diagnosis support screen that displays recognition results. FIG. 10 is a block diagram showing a configuration example of a medical information management system.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

《Example of configuration of endoscope system》
FIG. 1 is an overall configuration diagram showing a schematic configuration of an endoscope system according to an embodiment of the present invention. Endoscope system 10 includes an electronic endoscope 12, a light source device 14, and a processor device 16. A display device 18 and an input device 19 are connected to the processor device 16 . The electronic endoscope 12 is sometimes called a "scope,""electronicscope," or simply "endoscope."

The electronic endoscope 12 of this example is a flexible endoscope. The electronic endoscope 12 includes an insertion section 20, an operation section 30, and a universal cord 40. The insertion portion 20 is a portion inserted into the body cavity of the person to be examined. The insertion portion 20 includes, in order from the distal end toward the proximal side, a hard distal end portion 22, a curved portion 24, and a flexible portion 26. Inside the rigid tip portion 22, an illumination optical system, an objective optical system, an image sensor, etc. are arranged. The bending portion 24 is structured to smoothly bend from a reference position in four directions, up, down, left and right, in accordance with the operation of the angle knob 31. The proximal side of the flexible portion 26 is referred to as the proximal end portion of the insertion portion 20.

The operating section 30 is provided at the proximal end of the insertion section 20. The operating unit 30 includes various operating members operated by the surgeon. For example, the operation section 30 includes an angle knob 31 used for bending the bending section 24, an air/water supply button 32 for performing an air/water supply operation, and a suction button 33 for performing a suction operation. It is provided. By operating the angle knob 31, the operator can bend the bending section 24 and freely change the direction of the rigid distal end section 22. The operation unit 30 also includes a mode changeover switch 34 used for switching the observation mode, a zoom operation unit 35, and a still image photography instruction unit (not shown) for instructing photography of a still image of the region to be observed. It is provided.

Further, the operating section 30 is provided with a treatment instrument introduction port 36. The treatment instrument introduction port 36 is an opening for inserting a treatment instrument (not shown) into a treatment instrument insertion path (not shown) that passes through the insertion portion 20 . Examples of treatment tools include biopsy forceps, catheters, and high-frequency snares. Treatment tools also include guide tubes, trocar tubes, sliding tubes, and the like. The treatment instrument introduction port 36 is sometimes called a forceps port.

The universal cord 40 is a cord for connecting the electronic endoscope 12 to the light source device 14 and the processor device 16. A cable and a light guide extending from the insertion portion 20 are inserted into the universal cord 40 . The cables extending from the insertion section 20 include a communication cable used for signal transmission and a power supply cable used for power supply. A connector 42 is provided at one end of the universal cord 40.

The connector 42 is a composite type connector including a video connector 42A and a light guide connector 42B. One end of the cable is placed in the video connector 42A. Video connector 42A is detachably connected to processor device 16. One end of the light guide is arranged in the light guide connector 42B. The light guide connector 42B is detachably connected to the light source device 14. Further, the light guide connector 42B is provided with a water supply connector 42C, and the water supply tank 44 is connected via the water supply connector 42C.

The processor device 16 is electrically connected to the light source device 14 via the connector 42 . The processor device 16 comprehensively controls the operation of the endoscope system 10 including the light source device 14. The processor device 16 supplies power to the electronic endoscope 12 via a cable inserted into the universal cord 40 and controls the driving of the image sensor. The processor device 16 also receives an imaging signal transmitted from the electronic endoscope 12 via a cable, performs various signal processing on the received imaging signal, and converts it into image data. The image data converted by the processor device 16 is displayed on the display device 18 as an endoscopic image (observation image).

FIG. 2 is a front view showing the distal end surface 22A of the rigid distal end portion 22 of the electronic endoscope 12. As shown in FIG. An illumination window 50, an observation window 52, a forceps outlet 54, and an air/water supply nozzle 56 are provided on the distal end surface 22A of the rigid distal end portion 22. Although not shown in FIG. 2, an output end 122 of a light guide 120 that guides light from the light source device 14 is arranged at the back of the illumination window 50 (see FIG. 3). Illumination light is irradiated from the illumination window 50 onto the observed area. Two illumination windows 50 are arranged at symmetrical positions with the observation window 52 in between.

The observation window 52 is a window for taking in reflected light from the observed area and capturing an image of the observed area. Although not shown in FIG. 2, an objective optical system 60 and an image sensor 62 are arranged at the back of the observation window 52 to take in image light of the region to be observed in the body cavity (see FIG. 3). .

The forceps outlet 54 is connected to a forceps channel (not shown) provided in the insertion section 20 and communicates with a treatment instrument introduction port 36 (see FIG. 1) provided in the operating section 30. The treatment instrument inserted through the treatment instrument introduction port 36 is taken out from the forceps outlet 54 into the body cavity.

The air/water supply nozzle 56 sprays cleaning water or air toward the observation window 52 and/or into the body cavity in response to the operation of the air/water supply button 32 (see FIG. 1) provided on the operation unit 30. . Note that the cleaning water and air are supplied from an air and water supply device built into the light source device 14.

FIG. 3 is a block diagram showing a control system of the endoscope system 10. The rigid distal end portion 22 of the electronic endoscope 12 includes an objective optical system 60, an image sensor 62, an analog front end (AFE) circuit 64, a timing generator (TG) 65, and a CPU. (Central Processing Unit) 66 is provided.

The objective optical system 60 is configured using a zoom lens. A light guiding prism (not shown) is arranged between the objective optical system 60 and the image sensor 62. The image sensor 62 may be a CMOS (complementary metal oxide semiconductor) type image sensor or a CCD (Charged Coupled Device) type image sensor. Here, an example using a CMOS type image sensor will be described.

Although not shown in the drawings, the image sensor 62 is a solid-state image sensor of a single-chip color imaging type that includes a color filter made up of a plurality of color segments. The color filter may be, for example, a Bayer array primary color filter including red (R), green (G), and blue (B).

A large number of pixels are arranged in a matrix on the imaging surface of the image sensor 62, and each pixel is provided with a photo sensor. Light incident on the imaging surface of the image sensor 62 is accumulated as a charge in the photosensor of each pixel. The amount of signal charge accumulated in the photosensor of each pixel is sequentially read out as a pixel signal by scanning in the vertical and horizontal directions by a vertical scanning circuit and a horizontal scanning circuit (both not shown), and is read out sequentially as a pixel signal at a predetermined frame rate. Output.

The timing generator 65 generates drive pulses for the image sensor 62 and synchronization pulses for the analog front end circuit 64 under the control of the CPU 66 . The driving pulses for the image sensor 62 include a vertical scanning pulse, a horizontal scanning pulse, a reset pulse, and the like.

The image sensor 62 is driven by a drive pulse inputted from the timing generator 65, photoelectrically converts an optical image formed on an imaging surface via the objective optical system 60, and outputs the resultant image as an image signal.

Analog front end circuit 64 includes a correlated double sampling (CDS) circuit, an automatic gain control (AGC) circuit, and an A/D converter. The CDS circuit performs correlated double sampling processing on the image signal output from the image sensor 62 to remove reset noise and amplifier noise generated in the image sensor 62.

The AGC circuit amplifies the image signal from which noise has been removed by the CDS circuit with a gain (amplification factor) specified by the CPU 66. The A/D converter converts the image signal amplified by the AGC circuit into a digital signal with a predetermined number of bits and outputs the digital signal. The imaging signal digitized and output by the analog front end circuit 64 is input to the processor device 16 through a signal line.

Note that the image sensor 62, analog front end circuit 64, and timing generator 65 can be configured as a monolithic integrated circuit, and each of these circuit elements is included in one image sensor chip 68. The imaging chip 68 mounted on the electronic endoscope 12 of this example is a so-called "CMOS sensor chip" and is mounted on a support substrate (not shown).

The processor device 16 includes a CPU 70, a ROM (read-only memory) 72, a RAM (Random Access Memory) 74, a digital signal processing circuit (DSP) 76, and a display control circuit 78.

The CPU 70 controls each part within the processor device 16 and also controls the entire endoscope system 10 in an integrated manner. The ROM 72 stores various programs and control data for controlling the operation of the processor device 16. The RAM 74 temporarily stores programs and data executed by the CPU 70.

The digital signal processing circuit 76 performs various signal processing such as color interpolation, color separation, color balance adjustment, gamma correction, and image enhancement processing on the imaging signal input from the analog front end circuit 64 under the control of the CPU 70. and generate image data. Further, the digital signal processing circuit 76 performs image recognition processing. The digital signal processing circuit 76 functions as an image processing section. Further, the digital signal processing circuit 76 includes the function of an image recognition section that performs image recognition processing.

The image data output from the digital signal processing circuit 76 is input to the display control circuit 78. The display control circuit 78 converts the image data input from the digital signal processing circuit 76 into a signal format compatible with the display device 18 and causes the signal format to be displayed on the screen of the display device 18 .

The display device 18 may be, for example, a liquid crystal display, an organic electro-luminescence (OEL) display, a projector, or an appropriate combination thereof. The display device 18 can display various types of information such as various setting information necessary for processing by the processor device 16 or information indicating processing results.

Display device 18 and input device 19 function as a user interface. The input device 19 may be, for example, a keyboard, a mouse, a touch panel, an operation button, a voice input device, or an appropriate combination thereof. A user can input various instructions and/or information using the input device 19. The processor device 16 can perform various processes in response to instructions and/or information input from the input device 19.

The light source device 14 generates light that illuminates the inside of the body cavity through a light guide 120 inserted into the electronic endoscope 12 . The light source device 14 includes a first light source 100, a first light source drive circuit 101, a second light source 102, a second light source drive circuit 103, a CPU 104, and a multiplexer 105. The CPU 104 communicates with the CPU 70 of the processor device 16 and controls the first light source drive circuit 101 and the second light source drive circuit 103.

The first light source drive circuit 101 causes the first light source 100 to emit light according to instructions from the CPU 104. The second light source drive circuit 103 causes the second light source 102 to emit light according to instructions from the CPU 104.

The first light source 100 is, for example, a laser diode that emits a blue laser with a wavelength of 445 nm. The first light source 100 is pulse-driven by a first light source drive circuit 101 to control the amount of light emitted. The second light source 102 is, for example, a laser diode that emits a blue laser with a wavelength of 405 nm. The second light source 102 is pulse-driven by a second light source drive circuit 103 to control the amount of light emitted. Blue light with a wavelength of 405 nm is used for special light observation.

The combining unit 105 combines the respective output lights of the first light source 100 and the second light source 102 and outputs the combined light to the input end 121 of the light guide 120 .

A phosphor 124 is provided between the output end 122 of the light guide 120 and the illumination window 50 of the electronic endoscope 12. The blue laser light that has passed through the light guide 120 is irradiated onto the phosphor 124 to excite the phosphor 124, and part of it is transmitted through the phosphor 124 and emitted from the illumination window 50 as blue light.

The phosphor 124 is excited by blue laser light, and emits light (yellow in color) over a wide range of wavelengths, from the boundary between blue and green to the red wavelength range. This yellow light and the blue light transmitted through the phosphor 124 are mixed to form white light, which illuminates the subject through the illumination window 50. Note that the blue light transmitted through the phosphor 124 also includes a portion of the blue light emitted by the phosphor 124.

As described above, when the phosphor 124 is irradiated with blue laser light with a wavelength of 445 nm, it emits yellow light and transmits blue light with a wavelength of 445 nm, but when it is irradiated with blue laser light with a wavelength of 405 nm. In some cases, it has the property of transmitting most of it.

That is, by controlling the mixing ratio of blue laser light with a wavelength of 445 nm and blue laser light with a wavelength of 405 nm, the ratio of blue light transmitted through the phosphor 124 and yellow light emitted by the phosphor 124 can be controlled. is possible.

A color image of the subject is reproduced by using the image sensor 62 to receive reflected light from the subject illuminated with white light.

When observing the inside of a body cavity using the endoscope system 10 configured as described above, the electronic endoscope 12, light source device 14, processor device 16, and display device 18 are powered on. Then, the insertion section 20 of the electronic endoscope 12 is inserted into the body cavity, and while the interior of the body cavity is illuminated with the illumination light from the light source device 14, the image inside the body cavity captured by the image sensor 62 is displayed on the screen of the display device 18. will be observed.

The endoscope system 10 of this example has a white light observation mode and a narrowband light observation mode. The white light observation mode is a mode in which an observation image with natural colors is displayed on the display device 18 using a photographed image obtained by imaging an observation target using white light as illumination light. An image obtained by capturing an observation target in white light observation mode is called a "white light observation image." Illumination light may also be referred to as "observation light."

Narrowband light observation mode uses an image signal obtained by imaging an observation target using narrowband light in a specific wavelength band as illumination light, for example, for visualization that emphasizes blood vessels in a specific depth region of the observation target. This is a mode in which an image is generated and an image suitable for observing blood vessels is displayed on the display device 18. An image obtained by capturing an observation target in narrowband optical observation mode is called a "narrowband optical observation image."

The endoscope system 10 may have a plurality of types of narrowband light observation modes in which the types or combinations of wavelength bands of narrowband light used are different.

<<Overview of diagnosis support function in endoscope system 10>>
The processor device 16 includes an image recognition unit that recognizes the image received from the electronic endoscope 12 . For example, a convolutional neural network (CNN) is used for image recognition processing. The image recognition unit performs an image classification task of classifying lesions included in an image into a plurality of categories according to, for example, NICE classification or JNET classification. "NICE" is an abbreviation for NBI (Narrow band imaging) International Colorectal Endoscopic. "JNET" is an abbreviation for "the Japan NBI Expert Team."

The NICE classification is based on the non-magnified NBI, and is classified into Type 1, Type 2, and Type 3 for each of the three items of lesion color, microvascular pattern, and surface pattern. Ru. Type 1 is a diagnostic indicator for hyperplastic lesions, Type 2 is adenoma to intramucosal carcinoma, and Type 3 is a diagnostic indicator for SM (submucosa) deep invasive cancer. The JNET classification is a classification of NBI magnifying endoscopic findings for colorectal tumors. In the JNET classification, each item of "vessel pattern" and "surface pattern" is classified into Type 1, Type 2A, Type 2B, and Type 3.

The image recognition unit may simply perform two-class recognition of "cancerous" and "non-cancerous" instead of or in combination with detailed classification such as NICE classification.

In addition to recognition processing such as classification of lesions, the image recognition unit also performs recognition as to whether the input image is appropriate or inappropriate for image recognition. "Suitable for image recognition" means that the image is suitable for recognition processing such as classification of lesions, which is the main purpose of recognition. "Unsuitable for image recognition" means that the image is unsuitable for recognition such as classification of lesions, which is the main purpose. Examples of images unsuitable for recognition include, for example, blurred images, blurred images, images with water on the subject, and images with residue. Images that are unsuitable for recognition are called "inappropriate images."

For example, the image recognition unit performs two classifications of images, such as whether the image is suitable for recognition or unsuitable for recognition, and when it recognizes that the image is "suitable for recognition," it selects a classification such as NICE classification. perform tasks; Alternatively, the image recognition unit may add a classification such as "image unsuitable for recognition" to recognition processing of a classification task such as NICE classification.

The processor device 16 stores the recognition result when the recognition result is other than "an image unsuitable for recognition." Thereafter, the next image is received by the processor device 16 and image recognition is performed on this newly received image. When the recognition result of the newly received image is determined to be an "image unsuitable for recognition," the processor device 16 converts this recognition result (recognition result indicating that it is an "image unsuitable for recognition") into Change to the saved recognition result.

The recognition results can be displayed on the display device 18. Here, if the recognition result is changed to the saved one (previous saved recognition result), display the recognition result in a way that makes it clear that it is the previous recognition result. is preferred. Specific examples of "display that makes it clear that it is the previous recognition result" include displaying an icon next to the information indicating the recognition result, or changing the display color of the information indicating the recognition result, etc. is possible. The display device 18 is an example of a "display section." Furthermore, the display device 18 is an example of a "notification unit" that notifies the recognition result of the image recognition unit.

<<Configuration of medical image processing apparatus according to first embodiment>>
The processor device 16 is an example of a medical image processing device according to an embodiment of the present invention. FIG. 4 is a block diagram showing the functions of the medical image processing apparatus according to the first embodiment of the present invention. The medical image processing apparatus 160 shown in FIG. and.

The image acquisition unit 162 is an interface that acquires time-series images IM1, IM2, IM3, . . . taken using the electronic endoscope 12. The image acquisition unit 162 may be, for example, a connector terminal to which the video connector 42A is connected, or may be a signal input terminal of the digital signal processing circuit 76. Further, the image acquisition unit 162 may be a communication network terminal provided in the processor device 16, a media interface terminal for external storage media, a terminal for connecting an external device, or an appropriate combination thereof.

The image recognition unit 164 is a processing unit that performs image recognition from the image acquired via the image acquisition unit 162. The image recognition section 164 includes a first recognition section 164A and a second recognition section 164B. The first recognition unit 164A recognizes whether the input image is appropriate or inappropriate for image recognition. The second recognition unit 164B performs image classification processing based on the input image according to NIEC classification, JNET classification, or the like. Note that the second recognition unit 164B may perform a process of detecting a region of interest such as a lesion area, instead of or in combination with the image classification process. The recognition process performed by the first recognition unit 164A is referred to as "first recognition process." The recognition process performed by the second recognition unit 164B is referred to as "second recognition process." Each of the first recognition unit 164A and the second recognition unit 164B can be configured using, for example, a convolutional neural network (CNN). The image recognition unit 164 is configured using a learned model learned by machine learning.

When the recognition result obtained from the first recognition unit 164A is an “image suitable for recognition”, recognition processing is performed by the second recognition unit 164B. The second recognition unit 164B extracts feature amounts from the image and performs at least one of image classification, detection of a region of interest, segmentation, and similarity calculation. For example, the second recognition unit 164B performs image classification according to a predetermined classification method such as NICE classification or JNET classification.

The recognition result storage section 168 is a storage section that stores the recognition results of the image recognition section 164. The recognition result storage unit 168 may be a storage area of the RAM 74 shown in FIG. The recognition result storage unit 168 may be a storage area of a memory (not shown) other than the RAM 74. The recognition results stored in the recognition result storage unit 168 are sequentially updated to the latest information. The recognition result storage unit 168 only needs to store the latest (recent) recognition results.

The determination unit 166 determines whether or not to save the recognition result of the image recognition unit 164, depending on the recognition result of the image recognition unit 164. The determination unit 166 of this example determines whether the recognition result is saved or not based on the recognition result from the first recognition unit 164A.

If the recognition result from the first recognition unit 164A is an “image suitable for recognition”, a second recognition process is performed by the second recognition unit 164B. In this case, the determination unit 166 determines to save the recognition result, and the recognition result obtained from the second recognition unit 164B is stored in the recognition result storage unit 168. Further, the recognition result obtained from the second recognition section 164B is displayed on the display section 174 via the display control section 172. The display control section 172 corresponds to the display control circuit 78 shown in FIG. The display unit 174 may be the display device 18 (see FIGS. 1 and 2). Further, the display unit 174 may be a display device different from the display device 18.

On the other hand, if the recognition result from the first recognition unit 164A is an “image unsuitable for recognition”, the second recognition process by the second recognition unit 164B is stopped. That is, the second recognition unit 164B does not process the medical image that the first recognition unit 164A recognizes as an image unsuitable for recognition. In this case, the determination unit 166 determines that the recognition result is not saved (not saved).

The change processing unit 170 changes the recognition result of the image recognition unit 164 to the recognition result stored in the recognition result storage unit 168 according to the recognition result of the image recognition unit 164. The change processing unit 170 changes the recognition result for the image for which the determination unit 166 has determined that the recognition result is not saved, to the recognition result stored in the recognition result storage unit 168.

The recognition result after modification by the modification processing section 170 is displayed on the display section 174 via the display control section 172.

The recognition result storage unit 168 stores recognition results for images in which the recognition results of the image recognition unit 164 are maintained as they are without undergoing any modification processing by the modification processing unit 170. The change processing unit 170 changes the recognition result of the image recognition unit 164 to the latest (most recent) recognition result stored in the recognition result storage unit 168.

The functions of the image recognition section 164 can be realized by the digital signal processing circuit 76 described in FIG. 2, the CPU 70, or a combination thereof. The functions of the determination unit 166 and the change processing unit 170 can be realized by the CPU 70.

<<Operation of the medical image processing apparatus according to the first embodiment>>
FIG. 5 is a flowchart showing the operation of the medical image processing apparatus according to the first embodiment. The operation of the medical image processing device may be understood as a medical image processing method, a method of operating the medical image processing device, or a method of operating the processor device.

In step S12, the image acquisition unit 162 receives an input image. The input image here is a medical image including a subject image photographed using the electronic endoscope 12, and is one of the time-series images photographed sequentially in time series. For example, the input image is a one-frame image that constitutes a moving image. Step S12 is an example of an "image acquisition step."

In step S14, the image recognition unit 164 performs a first recognition process on the input image.

In step S16, the CPU 70 determines whether the input image is an image suitable for the second recognition process based on the recognition result of the first recognition process. If the determination result in step S16 is "Yes", that is, if the input image is an image suitable for the second recognition process, the process advances to step S18. In step S18, the image recognition unit 164 performs second recognition processing. Step S14 and step S18 are an example of an "image recognition step."

In step S20, the CPU 70 obtains a recognition result from the image recognition unit 164.

In step S22, the CPU 70 stores the recognition result obtained in step S20 in the recognition result storage unit 168. Step S22 is an example of a "recognition result storage step."

On the other hand, if the determination result in step S16 is "No determination", that is, if the input image is an image unsuitable for the second recognition process, the process proceeds to step S24.

In step S24, the CPU 70 obtains the recognition result stored in the recognition result storage unit 168. In step S26, the CPU 70 changes the recognition result of the first recognition process to the recognition result stored in the recognition result storage unit 168 to obtain the recognition result. Step S24 and step S26 are examples of "change processing steps."

The recognition result obtained in step S20 or step S26 is displayed on the display section 174.

After step S22 or step S26, the flowchart of FIG. 5 ends. The process in the flowchart of FIG. 5 is repeated for each input image acquired in time series.

According to such a configuration, when an image unsuitable for recognition is temporarily included in the images obtained in chronological order, the most recent recognition result stored in the recognition result storage unit 168 is used. , it is possible to present valid recognition results with high truthfulness.

Thereby, it is possible to avoid generating a recognition result (erroneous recognition result) with low authenticity from an image unsuitable for recognition and providing it to the user.

Further, even if the recognition result of an image unsuitable for recognition is changed, if an image suitable for recognition is obtained thereafter, the processes of steps S18 to S22 are performed, so that highly accurate recognition is possible.

<What to do when images unsuitable for recognition are continuously acquired for a certain period of time>
Although not shown in the flowchart of FIG. 5, when images unsuitable for recognition are continuously acquired for a certain period of time, the processor device 16 saves the recognition results of the image recognition unit 164 in the recognition result storage unit 168. The recognition processing of the image recognition unit 164 may be stopped when the acquisition of medical images with the recognition result changed to the current recognition result continues for a certain period of time. In this case, it is preferable to display a warning message or the like on the display unit 174 to draw the user's attention. The display section 174 is an example of a "notification section."

In the first embodiment shown in FIG. 4, an example was shown in which the "first recognition process" and the "second recognition process" were performed in stages, but instead of a configuration in which such two-stage recognition processing is performed. Therefore, a configuration may be adopted in which images are classified into a plurality of categories including the classification of "images unsuitable for recognition" in one recognition process.

<<Configuration of medical image processing apparatus according to second embodiment>>
In the first embodiment shown in FIG. 4, an example was shown in which the "first recognition process" and the "second recognition process" were performed in stages, but instead of a configuration in which such two-stage recognition processing is performed. Therefore, a configuration may be adopted in which images are classified into a plurality of categories including the classification of "images unsuitable for recognition" in one recognition process.

FIG. 6 is a block diagram showing the functions of a medical image processing apparatus according to a second embodiment of the present invention. Instead of the configuration shown in FIG. 4, the configuration shown in FIG. 6 can be adopted. In FIG. 6, elements that are the same or similar to the configuration shown in FIG. 4 are given the same reference numerals. Differences from FIG. 4 will be explained.

The image recognition unit 164 of the medical image processing device 160 shown in FIG. 6 is a processing unit that performs image recognition from the image acquired via the image acquisition unit 162. The image recognition unit 164 performs category classification in which the classification of "images unsuitable for recognition" is added to the recognition processing of the classification task according to a classification method such as NICE classification or JNET classification. The image recognition unit 164 performs image classification processing including classification of "images unsuitable for recognition" from the input image in one recognition process.

The determination unit 166 determines whether or not to save the recognition result based on the recognition result of the image recognition unit 164.

<<Operation of the medical image processing apparatus according to the second embodiment>>
FIG. 7 is a flowchart showing the operation of the medical image processing apparatus according to the second embodiment. In FIG. 7, steps that are the same or similar to those in the flowchart shown in FIG. 5 are given the same step numbers, and redundant explanations will be omitted. The differences from FIG. 5 will be explained.

The flowchart shown in FIG. 7 includes step S15 and step S17 instead of step S14, step S16, and step S18 in FIG.

In step S15, the image recognition unit 164 performs recognition processing on the input image. The recognition processing performed here is, for example, image classification processing in which a category of "image unsuitable for recognition" is added to a classification category such as NICE classification or JNET classification.

In step S17, the CPU 70 determines whether or not the intended recognition has been achieved from the input image. "Target recognition" here means recognizing categories other than "images unsuitable for recognition." If the recognition process in step S15 yields a recognition result indicating that the image is "unsuitable for recognition," the determination result in step S17 becomes "No determination." On the other hand, if a recognition result of a category other than "image unsuitable for recognition" is obtained through the recognition process in step S15, the determination result in step S17 becomes "Yes determination".

If the determination result in step S17 is "Yes", the process advances to step S20.

In step S20, the CPU 70 obtains the recognition result of the image recognition unit 164.

In step S22, the CPU 70 stores the recognition result obtained in step S20 in the recognition result storage unit 168.

On the other hand, if the determination result in step S17 is "No determination", the process advances to step S24.

In step S24, the CPU 70 obtains the recognition result stored in the recognition result storage unit 168. In step S26, the CPU 70 changes the recognition result of the image recognition unit 164 to the recognition result stored in the recognition result storage unit 168 to obtain a recognition result.

The recognition result obtained in step S20 or step S26 is displayed on the display section 174.

After step S22 or step S26, the flowchart of FIG. 7 ends. The process in the flowchart of FIG. 7 is repeated for each input image acquired in time series.

《Display example of recognition results》
FIG. 8 is an example of an endoscopic diagnosis support screen displayed on the display unit 174. The window 300 illustrated in FIG. 8 includes an image display area 301 and a recognition result display area 310. FIG. 8 shows a display example when an image suitable for recognition has been acquired. In the image display area 301, image contents of a medical image 302 captured using the electronic endoscope 12 are displayed in real time. FIG. 8 shows an example in which a medical image 302 includes a lesion area 303. Note that a single image may include a plurality of lesion areas.

In the recognition result display area 310, for example, information indicating whether the recognition result is "cancerous" or "non-cancerous" is displayed. Further, in the recognition result display area 310, for example, recognition results according to NIEC classification are displayed.

FIG. 9 is another example of the endoscopic diagnosis support screen displayed on the display unit 174. FIG. 9 shows a display example when an image unsuitable for recognition is acquired. The medical image 302 displayed in the image display area 301 of the window 300 shown in FIG. 9 is, for example, a blurred image as an example of an image unsuitable for recognition. When such an unclear image is acquired, the recognition result obtained from the image recognition section 164 is changed to the recognition result stored in the recognition result storage section 168. The recognition result display area 310 displays the changed recognition result, that is, the recognition result stored in the recognition result storage section 168.

At this time, an icon 312 is displayed next to the display of the recognition result to inform the user that it is the previous recognition result.

When the image obtained from the electronic endoscope 12 returns to an image suitable for recognition, the screen in FIG. 9 returns to the screen in FIG. 8.

<<Example of display indicating that the recognition result is stored in the recognition result storage unit 168>>
The display indicating that the recognition result is stored in the recognition result storage unit 168 is displayed in a different color or with a specific symbol added, compared to the case where the recognition result of the image recognition unit 164 is displayed as is. The display format may include at least one of a symbol display, a character string display with a specific character string added, and a blinking display, and any combination thereof may be used. The icon 312 shown in FIG. 9 is an example of symbol display. Displaying different colors is not limited to changing the color of characters, but also includes changing the background color.

《Advantages of embodiments of the present invention》
According to the configuration of the embodiment described above, the following effects can be obtained.

(1) When a time-series medical image temporarily includes an image that is unsuitable for recognition, such as a blurred image, the most recent recognition result stored in the recognition result storage unit 168 is used to It is possible to present highly accurate and valid recognition results.

(2) Thereby, it is possible to avoid generating a recognition result (erroneous recognition result) with low authenticity from an image unsuitable for recognition and providing it to the user.

(3) Furthermore, even if the recognition result of an image unsuitable for recognition is changed, if an image suitable for recognition is obtained afterwards, the processing from step S20 to step S22 is performed, so highly accurate recognition is possible. It is.

(4) According to the embodiments of the present invention, useful diagnostic support for doctors and the like can be provided.

《Third embodiment: Example of application to medical information management system》
The medical image processing device according to the present invention can be applied not only to the processor device 16 of the endoscope system 10 illustrated in FIG. 1, but also to various other applications. For example, the medical image processing device can be applied to a medical information management system that manages various medical information including endoscopic images.

FIG. 10 is a block diagram showing a configuration example of a medical information management system. The medical information management system 200 includes an image capture terminal 202, an image storage server 204, an information management device 210, a display device 218, and an input device 219. Each of the image capture terminal 202, image storage server 204, and information management device 210 is connected to a telecommunications line 230. The term "connection" is not limited to wired connections, but also includes the concept of wireless connections.

Telecommunication line 230 may be a local area network or a wide area network. The telecommunications line 230 is configured by an appropriate combination of wired and wireless lines.

Processor unit 16 of endoscopic system 10 is connected to telecommunications line 230 . Medical images generated by processor device 16 are captured via telecommunications line 230 into at least one of image capture terminal 202 , image storage server 204 , and information management device 210 . For example, medical images generated by processor device 16 are sent to image capture terminal 202. Image capture terminal 202 receives medical images from processor device 16 .

Image capture terminal 202 sends medical images received from processor device 16 to image storage server 204 . Further, the image capture terminal 202 may transmit the medical image received from the processor device 16 to the information management device 210.

The image storage server 204 serves as a storage device that stores a database of various medical images. Cloud storage may be used instead of the image storage server 204. In addition to medical images, the image storage server 204 may store image analysis results such as regions of interest included in the medical images, presence or absence of objects of interest, and image classification results.

Although one endoscope system 10 is shown in FIG. 10, a plurality of endoscope systems may be connected to the telecommunications line 230. Further, the telecommunication line 230 is not limited to the endoscope system, and other medical image diagnostic devices such as an ultrasound diagnostic device may be connected. An ultrasound image obtained from an ultrasound diagnostic device is an example of a "medical image."

The information management device 210 is realized by, for example, computer hardware and software. A display device 218 and an input device 219 are connected to the information management device 210. The information management device 210 may include some or all of the functions of the medical image processing device 160 shown in FIG. 4 or 6. For example, the information management device 210 includes the functions of an image acquisition section 162, an image recognition section 164, a determination section 166, a recognition result storage section 168, a change processing section 170, and a display control section 172. Display device 218 can function as display section 174. The functions of the information management device 210 can be realized by one or more computers, and can also be realized by cloud computing. The information management device 210 may include the functions of the image storage server 204. Further, the image capture terminal 202 can function as the image acquisition unit 162. Note that a configuration in which the image capture terminal 202 is omitted is also possible, in which case the image storage server 204 and/or the information management device 210 are configured to capture medical images from the processor device 16.

According to the medical information management system 200 configured as described above, not only medical images obtained from the electronic endoscope 12 in real time but also videos stored in the image storage server 204 can be played back. It is possible to perform the processing in the flowchart described in FIG. 5 or 7, and to display the recognition results as described in FIGS. 8 and 9.

The information management device 210 may be installed, for example, in an operating room, examination room, or conference room within a hospital, or may be installed in a medical institution, a research institution, or the like outside the hospital. The information management device 210 may be a workstation that supports examination, treatment, diagnosis, etc., or may be a business support device that supports medical work. The business support device may have functions such as accumulating clinical information, supporting the creation of diagnostic documents, and supporting the creation of reports.

《About the hardware configuration of each processing unit and control unit》
Various processes such as the image acquisition unit 162, image recognition unit 164, determination unit 166, recognition result storage unit 168, change processing unit 170, and display control unit 172 of the medical image processing device 160 described in FIGS. 4 and 6 are performed. The hardware structure of the processing unit that executes is the following various processors.

Various types of processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and function as various processing units, and FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing. These include a programmable logic device (PLD), an ASIC (Application Specific Integrated Circuit), and other dedicated electric circuits that are processors with a circuit configuration specifically designed to execute specific processing.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be configured by a plurality of FPGAs or a combination of a CPU and an FPGA. Further, the plurality of processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, one processor is configured with a combination of one or more CPUs and software, as typified by computers such as clients and servers. There is a form in which a processor functions as multiple processing units. Second, there are processors that use a single IC (Integrated Circuit) chip, as typified by System On Chip (SoC), which implements the functions of an entire system including multiple processing units. be. In this way, various processing units are configured using one or more of the various processors described above as a hardware structure.

Furthermore, the hardware structure of these various processors is more specifically an electric circuit (circuitry) that is a combination of circuit elements such as semiconductor elements.

《Modification 1》
The electronic endoscope is not limited to a flexible endoscope, but may be a rigid endoscope or a capsule endoscope. Furthermore, the device that generates time-series medical images including subject images is not limited to an electronic endoscope, and may be, for example, an ultrasonic diagnostic device.

《Modification 2》
The medical image processing device of the present disclosure can be used as a diagnosis support device that supports examination, treatment, diagnosis, etc. by a doctor or the like. The term "diagnostic support" includes the concepts of diagnostic support and/or treatment support.

《About the observation light of the endoscope system》
As the observation light, light in various wavelength bands is selected depending on the purpose of observation, such as white light, light in one or more specific wavelength bands, or a combination thereof. White light is light in a white wavelength band or light in multiple wavelength bands. The "specific wavelength band" is a band narrower than the white wavelength band. A specific example regarding a specific wavelength band is shown below.

<First example>
A first example of a specific wavelength band is, for example, a blue band or a green band in the visible range. The wavelength band of this first example includes a wavelength band of 390 nm or more and 450 nm or less, or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example includes a wavelength band of 390 nm or more and 450 nm or less, or a wavelength band of 530 nm or more and 550 nm or less. It has a peak wavelength within the band.

<2nd example>
A second example of the specific wavelength band is, for example, the visible red band. The wavelength band of this second example includes a wavelength band of 585 nm or more and 615 nm or less, or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example has a wavelength of 585 nm or more and 615 nm or less or a wavelength of 610 nm or more and 730 nm or less. It has a peak wavelength within the band.

<3rd example>
The third example of the specific wavelength band includes a wavelength band in which oxyhemoglobin and deoxyhemoglobin have different extinction coefficients, and the light of the third example has a peak wavelength in a wavelength band in which oxyhemoglobin and deoxyhemoglobin have different extinction coefficients. has. The wavelength band of the third example includes a wavelength band of 400±10 nm, a wavelength band of 440±10 nm, a wavelength band of 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example , 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

<4th example>
A fourth example of the specific wavelength band is a wavelength band of excitation light that is used to observe fluorescence emitted by a fluorescent substance in a living body (fluorescence observation) and that excites this fluorescent substance, for example, from 390 nm to 470 nm.

<Example 5>
A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of this fifth example includes a wavelength band of 790 nm or more and 820 nm or less, or a wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example includes a wavelength band of 790 nm or more and 820 nm or less or a wavelength of 905 nm or more and 970 nm or less. It has a peak wavelength within the band.

《About switching observation light》
The type of light source may be a laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof. It is preferable to configure the type of light source, wavelength, presence or absence of a filter, etc. according to the type of subject, purpose of observation, etc. Also, when observing, the wavelength of illumination light should be combined according to the type of subject, purpose of observation, etc. and/or switching is preferable. When switching the wavelength, for example, by rotating a disc-shaped filter (rotary color filter) that is placed in front of the light source and equipped with a filter that transmits or blocks light of a specific wavelength, the wavelength of the emitted light is switched. Good too.

The image sensor used in the electronic endoscope is not limited to a color image sensor in which a color filter is provided for each pixel, but may be a monochrome image sensor. When using a monochrome image sensor, it is possible to sequentially switch the wavelength of the illumination light and capture images sequentially (color sequentially). For example, the wavelength of the illumination light to be emitted may be sequentially switched between purple, blue, green, and red, or broadband light (white light) may be emitted and emitted by a rotary color filter (red, green, blue, etc.). The wavelength of the illumination light used may be switched. Alternatively, the wavelength of the illumination light emitted by a rotary color filter may be switched by irradiating one or more narrowband lights. The narrow band light may be infrared light having two or more different wavelengths.

《Example of special light image generation》
The processor device 16 may generate a special light image having information on a specific wavelength band based on a normal light image obtained by imaging using white light. Note that generation here includes the concept of "acquisition." In this case, the processor device 16 functions as a special light image acquisition section. The processor device 16 converts signals in a specific wavelength band into red (R), green (G), and blue (B), or cyan (C), magenta (M), and yellow (Y) included in the normal light image. ) can be obtained by performing calculations based on the color information.

《Example of feature image generation》
The processor device 16 generates, as medical images, normal light images obtained by irradiating white band light or light in a plurality of wavelength bands as white band light, and special light images obtained by irradiating light in a specific wavelength band. A feature image can be generated using calculations based on at least one of the following. A feature image is one form of a medical image.

《About the program that enables a computer to implement the functions of a medical image processing device》
A program that causes a computer to implement the functions of the medical image processing apparatus described in the above embodiments is recorded on a computer readable medium that is a non-transitory information storage medium such as an optical disk, a magnetic disk, or a semiconductor memory or other tangible object. It is possible to provide the program through an information storage medium. Furthermore, instead of providing the program by storing it in a tangible, non-temporary information storage medium, it is also possible to provide the program signal as a download service using a telecommunications line such as the Internet.

Further, it is also possible to provide some or all of the functions of the medical image processing apparatus described in the above embodiments as an application server, and provide a service that provides processing functions through a telecommunications line.

《About combinations of embodiments, modifications, etc.》
The components described in the above-described embodiments and the components described in the modified examples can be used in appropriate combinations, and some components can also be replaced.

《Additional notes》
In addition to each of the embodiments and modifications described above, this specification includes disclosures of the configuration inventions described below.

(Additional note 1)
The medical image processing device includes a medical image analysis processing unit and a medical image analysis result acquisition unit, and the medical image analysis processing unit selects an area to be focused on based on feature amounts of pixels of the medical image. A medical image processing device that detects a region, and a medical image analysis result acquisition unit that acquires an analysis result of a medical image analysis processing unit.

The medical image analysis processing section may include an image recognition section.

(Additional note 2)
The medical image analysis processing unit detects the presence or absence of a target of interest based on the feature amount of pixels of the medical image, and the medical image analysis result acquisition unit performs medical image processing to acquire the analysis results of the medical image analysis processing unit. Device.

(Additional note 3)
The medical image analysis result acquisition unit acquires the analysis result of the medical image from a recording device, and the analysis result is either a region of interest that is a region of interest included in the medical image, or the presence or absence of a target of interest. , or both.

(Additional note 4)
The medical image processing device is a normal light image obtained by irradiating white band light or light in a plurality of wavelength bands as white band light.

(Appendix 5)
A medical image is an image obtained by irradiating light in a specific wavelength band, and the specific wavelength band is a medical image processing device that is narrower than the white wavelength band.

(Appendix 6)
A medical image processing device in which the specific wavelength band is a blue or green band in the visible range.

(Appendix 7)
The specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light in the specific wavelength band has a peak wavelength within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Medical Image processing device.

(Appendix 8)
The specific wavelength band is the visible red band for medical image processing equipment.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light in the specific wavelength band is a medical device having a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which oxyhemoglobin and deoxyhemoglobin have different extinction coefficients, and the light in the specific wavelength band has a peak wavelength in a wavelength band in which oxyhemoglobin and deoxyhemoglobin have different extinction coefficients. Medical image processing equipment.

(Appendix 11)
The specific wavelength band includes 400±10nm, 440±10nm, 470±10nm, or a wavelength band of 600nm or more and 750nm or less, and the light in the specific wavelength band is 400±10nm, 440±10nm, 470±10nm, or 600nm or more and 750nm or less. A medical image processing device having a peak wavelength in a wavelength band of 10 nm or 600 nm or more and 750 nm or less.

(Appendix 12)
A medical image is an in-vivo image that shows the inside of a living body, and an in-vivo image is a medical image processing device that has information on fluorescence emitted by a fluorescent substance inside the living body.

(Appendix 13)
A medical image processing device generates fluorescence by irradiating the inside of a living body with excitation light having a peak of 390 nm or more and 470 nm or less.

(Appendix 14)
The medical image is an in-vivo image that shows the inside of a living body, and the specific wavelength band is a wavelength band of infrared light.

(Appendix 15)
The specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light in the specific wavelength band has a peak wavelength in a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. Processing equipment.

(Appendix 16)
The medical image acquisition unit is a special light image acquisition unit that acquires a special light image having information of a specific wavelength band based on a normal light image obtained by irradiating white band light or light of a plurality of wavelength bands as white band light. A medical image processing device that includes an optical image acquisition unit and whose medical images are special optical images.

(Appendix 17)
A medical image processing device that obtains signals in a specific wavelength band through calculations based on RGB or CMY color information included in a normal optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as white band light, and a special light image obtained by irradiating light in a specific wavelength band, A medical image processing device including a feature image generation unit that generates a feature image, the medical image being a feature image.

(Appendix 19)
A medical image processing device according to any one of Supplementary notes 1 to 18, and an endoscope that acquires an image by irradiating at least one of white wavelength band light or light in a specific wavelength band. An endoscope device comprising:

(Additional note 20)
A diagnosis support device comprising the medical image processing device according to any one of appendices 1 to 18.

(Additional note 21)
A medical work support device comprising the medical image processing device according to any one of Supplementary Notes 1 to 18.

[others]
In the embodiments of the present invention described above, constituent elements can be changed, added, or deleted as appropriate without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made within the technical idea of the present invention by those having ordinary knowledge in equivalent and related fields.

10 Endoscope system 12 Electronic endoscope 14 Light source device 16 Processor device 18 Display device 19 Input device 20 Insertion section 22 Hard tip section 22A Tip surface 24 Curved section 26 Flexible section 30 Operation section 31 Angle knob 32 Air/water supply button 33 Suction button 34 Mode changeover switch 35 Zoom operation section 36 Treatment instrument introduction port 40 Universal cord 42 Connector 42A Video connector 42B Light guide connector 42C Water supply connector 44 Water supply tank 50 Illumination window 52 Observation window 54 Forceps outlet 56 Air and water supply nozzle 60 Objective Optical system 62 Image sensor 64 Analog front end circuit 65 Timing generator 66 CPU
68 Imaging chip 70 CPU
72 ROM
74 RAM
76 Digital signal processing circuit 78 Display control circuit 100 First light source 101 First light source drive circuit 102 Second light source 103 Second light source drive circuit 104 CPU
105 Multiplexing section 120 Light guide 121 Incident end 122 Output end 124 Phosphor 160 Medical image processing device 162 Image acquisition section 164 Image recognition section 164A First recognition section 164B Second recognition section 166 Determination section 168 Recognition result storage section 170 Change processing unit 172 Display control unit 174 Display unit 200 Medical information management system 202 Image capture terminal 204 Image storage server 210 Information management device 218 Display device 219 Input device 230 Telecommunication line 300 Window 301 Image display area 302 Medical image 303 Lesion area 310 Recognition result display area 312 Icons IM1, IM2, IM3 Images S12 to S26 Processing steps in the medical image processing device

Claims (29)

comprising a processor and a memory in which a program executed by the processor is stored,
By executing the program, the processor:
Acquire multiple medical images including the subject image,
performing image recognition on each of the plurality of medical images;
Storing the recognition result of the image recognition in an image recognition result storage unit,
A medical image processing device that displays the stored recognition result,
The image recognition process includes a process of recognizing that the medical image is an image unsuitable for recognition,
The processor includes:
When the medical image is the unsuitable image, a region of interest using a second medical image other than the unsuitable image acquired before the first medical image that is the unsuitable image. performing processing for simultaneously displaying a recognition result regarding the detection of the image and a warning regarding the inappropriate image ;
Medical image processing equipment. The recognition result is displayed together with an observed image among the plurality of medical images.
The medical image processing device according to claim 1. The warning display is a display related to the first medical image,
The medical image processing device according to claim 1 or 2 . The warning display includes at least one of a different color display, a symbol display, a character string display, and a blinking display.
A medical image processing device according to any one of claims 1 to 3 . The processor includes:
ceasing detection of the region of interest if the acquisition of the inappropriate medical image continues for a certain period of time;
A medical image processing device according to any one of claims 1 to 4 . the region of interest is a lesion region;
A medical image processing device according to any one of claims 1 to 5 . The processor includes:
Performing processing to display the medical images obtained in time series,
When the acquired medical image is the inappropriate image,
displaying the recognition result regarding the detection of the region of interest using the second medical image and the warning on a screen including an image display area of the medical images in the time series;
A medical image processing device according to any one of claims 1 to 6. The processor includes:
Displaying a recognition result including a result of at least one of detection and classification of a region of interest for the first medical image when the first medical image is recognized as an image unsuitable for recognition through the image recognition. performing processing for displaying a recognition result including a result of at least one of detection and classification of a region of interest for the second medical image together with the first medical image without
A medical image processing device according to any one of claims 1 to 7 . The processor includes:
determining whether or not to save the recognition result of the image recognition according to the recognition result of the image recognition, and storing the recognition result of the image recognition in the image recognition result storage unit according to the determination result;
A medical image processing device according to any one of claims 1 to 8 . The processor includes:
Processing for displaying the recognition result stored in the image recognition result storage unit without displaying the recognition result of the image recognition for the medical image that has been determined to not store the recognition result in the determination. The medical image processing apparatus according to claim 9 , which performs the following. The processor includes:
If the image is recognized as being unsuitable for recognition through the image recognition process, determining that the recognition result of the medical image recognized as being unsuitable is not to be saved;
The medical image processing device according to claim 9 or 10 . The recognition result of the medical image that the processor recognizes as an image unsuitable for recognition is not stored in the image recognition result storage unit.
A medical image processing device according to any one of claims 1 to 11 . The processor includes:
When the image recognition recognizes that the first medical image is an image unsuitable for recognition,
performing a process of changing the recognition result for the first medical image to the recognition result for the second medical image,
A recognition result for the medical image in which the recognition result of the image recognition is maintained without undergoing the change process is stored in the image recognition result storage unit.
A medical image processing device according to any one of claims 1 to 12 . The processor includes:
performing processing for displaying the latest recognition result for the second medical image stored in the image recognition result storage unit instead of the recognition result for the first medical image;
A medical image processing device according to any one of claims 1 to 13 . comprising a display unit that displays the recognition result of the image recognition,
Any one of claims 1 to 14 , wherein the recognition result for the second medical image stored in the image recognition result storage unit is displayed on the display unit according to the recognition result of the image recognition. The medical image processing device described in . The processor includes:
When displaying a recognition result for the second medical image stored in the image recognition result storage unit, displaying a display indicating that the recognition result is stored in the image recognition result storage unit;
The medical image processing device according to any one of claims 1 to 15 . The display indicating that the recognition result is stored in the image recognition result storage unit may be displayed in a different color or with a specific symbol added compared to the case where the recognition result of the image recognition is displayed as is. including at least one display form of a symbol display with a specific character string added, a character string display with a specific character string added, and a blinking display.
The medical image processing device according to claim 16 . The processor includes:
stopping the recognition process of the image recognition if the acquisition of the medical image recognized as an image unsuitable for recognition by the image recognition continues for a certain period;
A medical image processing device according to any one of claims 1 to 17 . The processor includes:
performing processing for displaying a message to alert the user when the medical image recognized as an image unsuitable for recognition by the image recognition continues for a certain period of time;
A medical image processing device according to any one of claims 1 to 18 . The unsuitable image is an image in which the subject is blurred, an image in which the subject is blurred, an image in which the subject is covered with water, or an image in which the subject has residue.
The medical image processing device according to any one of claims 1 to 19 . The image recognition process includes a first recognition process for recognizing whether the acquired medical image is an image unsuitable for recognition;
a second recognition process that performs at least one of detecting a region of interest from the acquired medical image and classifying the medical image;
including;
The processor includes:
The second recognition process is not performed on the medical image that is recognized as an image unsuitable for recognition by the first recognition process.
A medical image processing device according to any one of claims 1 to 20 . The processor includes:
classifying the medical image into a plurality of categories, including classification of images unsuitable for recognition, by the image recognition;
The medical image processing device according to any one of claims 1 to 21 . An electronic endoscope that photographs the inside of the body cavity,
An endoscope system comprising: a processor device that processes image signals obtained from an electronic endoscope;
The processor device includes:
comprising a processor and a memory in which a program executed by the processor is stored,
By executing the program, the processor:
Acquire multiple medical images including the subject image,
performing image recognition on each of the plurality of medical images;
Storing the recognition result of the image recognition in an image recognition result storage unit,
performing a process of displaying the saved recognition result;
The image recognition process includes a process of recognizing that the medical image is an image unsuitable for recognition,
The processor includes:
When the medical image is the unsuitable image, a region of interest using a second medical image other than the unsuitable image acquired before the first medical image that is the unsuitable image. An endoscope system that performs processing for simultaneously displaying a recognition result regarding the detection of the image and a warning regarding the inappropriate image . The recognition result is displayed together with an observed image among the plurality of medical images.
The endoscope system according to claim 23. The warning display is a display related to the first medical image,
The endoscope system according to claim 23 or 24 . The warning display includes at least one of a different color display, a symbol display, a character string display, and a blinking display.
The endoscope system according to any one of claims 23 to 25 . The processor includes:
ceasing detection of the region of interest if the acquisition of the inappropriate medical image continues for a certain period of time;
The endoscope system according to any one of claims 23 to 26 . the region of interest is a lesion region;
The endoscope system according to any one of claims 23 to 27 . The processor includes:
Performing processing to display the medical images obtained in time series,
When the acquired medical image is the inappropriate image,
displaying the recognition result regarding the detection of the region of interest using the second medical image and the warning on a screen including an image display area of the medical images in the time series;
An endoscope system according to any one of claims 23 to 28.

Images