JP2023026480A - Medical image processing device, endoscope system, and operation method of medical image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a medical image processing device, an endoscope system, and an operation method of a medical image processing device which allow medical images to be highlighted according to the real-timeness thereof.

SOLUTION: A medical image processing device according to one embodiment of the present invention comprises: a medical image acquisition unit for acquiring a medical image; an information acquisition unit for acquiring information regarding an area of interest in the medical image; a mode setting unit for setting, on the basis of the information, the highlighting of the area of interest to a display mode in which the highlighting level in a case where the medical image is a non-real-time image acquired in non-real time is higher than the highlighting level in a case where the medical image is a real-time image acquired in real time; a display control unit causing the medical image to be displayed on a display device in the display mode that has been set; and a recording control unit causing the medical image and the information to be associated and recorded in a recording device. The display control unit causes the display device to display portion information being information indicating a portion captured in the medical image together with the display of the medical image.

SELECTED DRAWING: Figure 7

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to a medical image processing apparatus for highlighting medical images, an endoscope system, and a method of operating a medical image processing apparatus.

2. Description of the Related Art Emphasizing an image or changing a display mode is known as a support for a user such as a doctor to observe or diagnose a medical image. For example, Patent Literature 1 describes displaying an alert image according to the size and number of attention areas.

JP 2011-255006 A

When observing or diagnosing a medical image, the preferred mode of highlighting differs depending on the nature of the image, so it is necessary to switch between them as appropriate. For example, by superimposing a graphic or the like on a region of interest detected in an image acquired in real time during a screening examination, it is possible to notify the user of the detection of a lesion. On the other hand, the user saves the images of the lesions found during the examination and uses them for report writing and research. In addition, the saved image is displayed again as necessary for observation. When displaying an image acquired in non-real time in this way, superimposed display of figures and the like may interfere with observation and diagnosis, so it is preferable to set the highlighted display of the region of interest in a manner different from that during real-time display. However, such circumstances are not taken into consideration in the aforementioned Patent Document 1.

As described above, it has been difficult with conventional techniques to appropriately highlight and display a medical image in real time.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and provides a medical image processing apparatus, an endoscope system, and a method of operating the medical image processing apparatus that can highlight and display medical images in real time. for the purpose.

In order to achieve the above-described object, a medical image processing apparatus according to a first aspect of the present invention includes a medical image acquisition unit that acquires a medical image, an information acquisition unit that acquires information on a region of interest in the medical image, an information a mode setting unit for setting the highlighting of the region of interest to a display mode according to whether the medical image is a real-time image acquired in real time or a non-real-time image acquired in non-real time, based on A display control unit that causes a display device to display an image in a set display mode, and a recording control unit that causes a recording device to record medical images and information in association with each other.

The medical image processing apparatus according to the first aspect can highlight a medical image according to real-time characteristics (whether it is a real-time image acquired in real time or a non-real-time image acquired in non-real time). . In addition, since the medical image and the information of the region of interest are associated and recorded in a recording device (the information is recorded without being superimposed on the image), the medical image and information are used after the fact (in the case of non-real-time images) can also be displayed in a display mode according to the purpose of observation or the desire of the user.

In the first mode, the "setting of display mode" includes whether to set highlighting to on or off, and how to display the information of the attention area. Information can be displayed using letters, numbers, graphics, symbols, and combinations thereof, and may be colored.

In the first aspect and each of the following correspondences, "real-time" and "real-time images" include, for example, multiple medical images captured at a determined frame rate during an ongoing examination or observation with a substantial delay. Sequentially acquiring and displaying without, and images so sequentially displayed. "Real-time" and "real-time image" include cases where image acquisition and display are slightly delayed from perfect real-time due to image transmission/reception, image processing, and the like. On the other hand, "non-real-time" and "non-real-time image" include, for example, the case where a still image is acquired and displayed during and after inspection.

In the first aspect and the following aspects, the number of display devices is not particularly limited. Both real-time and non-real-time images may be displayed on the same display device. In this case, for example, a real-time image can be displayed, and a specific frame can be freeze-displayed according to a user's instruction or the like. Alternatively, a display device for real-time images and a display device for non-real-time images may be provided separately, and images may be displayed on the respective display devices. When separate monitors are provided, for example, real-time images may be displayed on one display device, and images acquired in an examination may be displayed on another display device in non-real time, such as when creating a report. Both aspects are included in "displaying a medical image on a display device".

The medical image processing apparatus according to the first aspect can be realized, for example, as a processor of a medical image processing system, but is not limited to such an aspect. In addition, "medical images" refer to images obtained as a result of photographing or measuring a living body such as a human body for the purpose of diagnosis, treatment, measurement, etc., such as endoscopic images, ultrasonic images, CT images (CT images). : Computed Tomography) and MRI images (MRI: Magnetic Resonance Imaging). Medical images are also called medical images.

In the first aspect of the medical image processing apparatus according to the second aspect, the aspect setting unit sets highlighting of the region of interest to ON or OFF when the medical image is a real-time image. According to the second aspect, when the highlighting is set to ON, the information of the attention area can be notified in real time by the highlighting, and the overlooking of the lesion or the like can be prevented. On the other hand, if highlighting or the like hinders diagnosis, it may be turned off. Whether to set on or off may be determined based on the user's operation, or may be determined independently of the user's operation.

In the first aspect of the medical image processing apparatus according to the third aspect, the aspect setting unit sets highlighting of the region of interest to ON or OFF when the medical image is a non-real-time image. An image with superimposed information is not suitable for reports or diagnoses, while the information on the attention area is necessary when reviewing the image after examination (prevention of oversight, research use, etc.). In the third aspect, highlighting is turned on or off for non-real-time images in consideration of such circumstances, so that highlighting can be performed according to the intended use of the image. It can be set to ON or OFF according to the user's operation, thereby allowing the user to display the medical image in a desired display mode.

A medical image processing apparatus according to a fourth aspect is any one of the first to third aspects, wherein the medical image acquisition unit acquires the medical image recorded in the recording device as a non-real-time image, and the information acquisition unit obtains information recorded in association with non-real-time images. The medical image acquisition unit may acquire medical images from a recording device connected via a network.

In any one of the first to fourth aspects, the medical image processing apparatus according to a fifth aspect includes a detector that detects a region of interest from the medical image, and the information acquisition unit obtains information from the detection result of the detector. to get A medical image to be detected may be a real-time image or a non-real-time image. In addition, in the fifth aspect, the detector can be configured by a trained model.

A medical image processing apparatus according to a sixth aspect, in any one of the first to fifth aspects, includes a classifier that classifies the medical image, and the information acquisition unit acquires information from the classification result of the classifier. do. A medical image to be classified may be a real-time image or a non-real-time image. In addition, in the sixth aspect, the classifier can be configured by a trained model.

A medical image processing apparatus according to a seventh aspect, in any one of the first to sixth aspects, is provided with a measuring instrument that measures a medical image, and the information acquisition unit acquires information from the measurement results of the measuring instrument. do. A medical image to be measured may be a real-time image or a non-real-time image. In addition, in the seventh aspect, the measuring instrument can be configured by a learned model.

A medical image processing apparatus according to an eighth aspect, in any one of the first to seventh aspects, comprises a reception unit that receives a user's display mode setting operation, and the mode setting unit based on the received setting operation Set the display mode. According to the eighth aspect, the user can display the medical image in a desired display mode (a mode of highlighting the region of interest).

A medical image processing apparatus according to a ninth aspect is a medical image processing apparatus according to any one of the first to eighth aspects, wherein a medical image showing the same region of interest is displayed on the display device. and the mode setting section sets the display mode based on the first elapsed time information. In a ninth aspect, for example, when real-time images are sequentially acquired and displayed at a predetermined frame rate, the It can be applied when setting the display mode. In the ninth mode, the display mode can be changed (including switching between ON and OFF) according to the first elapsed time.

In any one of the first to ninth aspects, a medical image processing apparatus according to a tenth aspect acquires information of a second elapsed time during which the same medical image is displayed on the display device. A time information acquisition section is provided, and the mode setting section sets the display mode based on the second elapsed time information. A tenth aspect is, for example, when a non-real-time image is observed by freeze display (fixed display), and the display mode is set according to the elapsed time (second elapsed time) after the image is displayed. can be applied to In the tenth mode, the display mode can be changed (including switching between ON and OFF) according to the second elapsed time.

In any one of the first to tenth aspects of the medical image processing apparatus according to the eleventh aspect, the display control unit causes the display device to display a list of the plurality of medical images recorded in the recording device, and sets the aspect. The unit sets the mode of highlighting for the plurality of medical images displayed as a list to a common mode. When displaying a list of a plurality of medical images, the operation is troublesome if the display mode is set for each image. However, in the eleventh mode, the highlight display mode is set to a common mode, so the operation is simple. This setting can be made according to the user's operation.

In the medical image processing apparatus according to the twelfth aspect of the eleventh aspect, the common aspect is turning on or off highlighting for a plurality of medical images. According to the twelfth aspect, highlighting for a plurality of medical images can be collectively turned on or off.

In any one of the first to twelfth aspects, the medical image processing apparatus according to the thirteenth aspect is configured such that the medical image is an image sequentially acquired in an ongoing examination or an ongoing observation. , the medical image is determined to be a real-time image. The thirteenth aspect can be applied to ongoing inspection or ongoing observation.

A medical image processing apparatus according to a fourteenth aspect is any one of the first to thirteenth aspects, further comprising a recording unit, wherein the medical image is an image recorded in the recording unit. case, a medical image is determined as the non-real-time image. The fourteenth aspect can be applied to medical images already recorded in the recording unit.

A medical image processing apparatus according to a fifteenth aspect is any one of the first to fourteenth aspects, further comprising a communication control unit, wherein the aspect setting unit acquires the medical image via the communication control unit. , the medical image is determined to be a non-real-time image. The fifteenth aspect can be applied to medical images acquired via the communication control unit.

In order to achieve the object described above, an endoscope system according to a sixteenth aspect of the present invention includes a medical image processing apparatus according to any one of the first to fifteenth aspects, a display device, and a subject. an endoscope to be inserted, the endoscope having an imaging unit for imaging a medical image. Since the endoscope system according to the sixteenth aspect includes the medical image processing apparatus according to any one of the first to fifteenth aspects, it is possible to perform enhancement processing according to the real-time nature of medical images. A real-time image can be captured by the imaging unit of the endoscope. Note that the number of display devices is not particularly limited in the sixteenth mode, as described above for the first mode. Both the real-time image and the non-real-time image may be displayed on the same display device, or a display device for the real-time image and a display device for the non-real-time image may be provided separately, and the images may be displayed on each display device. good too.

To achieve the above object, a medical image processing method according to a seventeenth aspect of the present invention includes a medical image acquisition step of acquiring a medical image, an information acquisition step of acquiring information on a region of interest in the medical image, an information a mode setting step of setting the highlighting of the region of interest to a display mode according to whether the medical image is a real-time image acquired in real time or a non-real-time image acquired in non-real time, based on; It has a display control step of displaying an image in a set display mode on a display device, and a recording control step of associating a medical image with information and recording it in a recording device. According to the seventeenth aspect, similarly to the first aspect, the medical image can be highlighted in real time.

The medical image processing method according to the eighteenth aspect may further include the same configuration (steps) as those of the second to fifteenth aspects. In addition, a non-temporary recording medium recording a program for causing a medical image processing apparatus or a computer to execute the medical image processing method of these aspects and a computer-readable code of the program can also be mentioned as an aspect of the present invention.

As described above, according to the medical image processing apparatus, the endoscope system, and the operating method of the medical image processing apparatus according to the present invention, medical images can be highlighted in real time.

FIG. 1 is a diagram showing the configuration of an endoscope system according to the first embodiment. FIG. 2 is another diagram showing the configuration of the endoscope system. FIG. 3 is a functional block diagram of an image processing unit. FIG. 4 is a diagram showing a configuration example of a convolutional neural network. FIG. 5 is a diagram showing how convolution processing is performed by a filter. FIG. 6 is a diagram showing information recorded in a recording unit. FIG. 7 is a flow chart showing the procedure of the medical image processing method according to the first embodiment. FIG. 8 is a diagram showing an example of preferred highlighting for real-time images. FIG. 9 is a diagram showing an example of highlighting that is undesirable for a real-time image. FIG. 10 is a diagram showing a display example of part information. FIG. 11 is another flow chart showing the procedure of the medical image processing method. FIG. 12 is a diagram showing an example of highlighting according to elapsed time. FIG. 13 is still another flow chart showing the procedure of the medical image processing method. FIG. 14 is a diagram showing how highlighting of a plurality of images displayed as a list is collectively turned on or off.

Hereinafter, embodiments of a medical image processing apparatus, an endoscope system, and a method of operating the medical image processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

<First embodiment>
<Configuration of endoscope system>
FIG. 1 is an external view of an endoscope system 10 (medical image processing apparatus, endoscope system), and FIG. As shown in FIGS. 1 and 2, the endoscope system 10 includes an endoscope scope 100 (medical equipment, endoscope scope, endoscope main body), an endoscope processor device 200 (medical image processing device), a light source It is composed of a device 300 (light source device) and a monitor 400 (display device). An external device (not shown) may be connected to the endoscope system 10 for acquiring information on the observation site using electromagnetic waves or ultrasonic waves.

<Configuration of endoscope>
The endoscope 100 includes a hand operating section 102 and an insertion section 104 connected to the hand operating section 102 . The operator (user) grasps and operates the hand operation unit 102, inserts the insertion unit 104 into the body of the subject (living body), and observes. Further, the hand operation unit 102 is provided with an air/water supply button 141, a suction button 142, a function button 143 to which various functions can be assigned, and a shooting button 144 for accepting shooting instruction operations (still image, moving image). . The insertion portion 104 is composed of a flexible portion 112, a bending portion 114, and a distal end hard portion 116 in order from the hand operation portion 102 side. That is, the bending portion 114 is connected to the proximal side of the distal rigid portion 116 , and the flexible portion 112 is connected to the proximal side of the bending portion 114 . A proximal operation section 102 is connected to the proximal end of the insertion section 104 . The user can bend the bending portion 114 by operating the hand operation portion 102 to change the orientation of the distal end rigid portion 116 up, down, left, or right. The hard tip portion 116 is provided with an imaging optical system 130, an illumination portion 123, a forceps port 126, and the like (see FIGS. 1 and 2).

During observation and treatment, the operation unit 208 (see FIG. 2) is operated to emit white light and/or narrowband light (red narrowband light, green narrowband light, one or more of blue narrowband light and violet narrowband light). Also, by operating the air/water supply button 141, washing water is discharged from a water supply nozzle (not shown) to wash the photographing lens 132 (photographing lens, photographing unit) of the photographing optical system 130 and the illumination lenses 123A and 123B. can be done. A conduit (not shown) communicates with the forceps opening 126 opened at the distal end rigid portion 116, and a treatment instrument (not shown) for removing a tumor or the like is inserted through the conduit and moves forward and backward as appropriate to the subject. necessary action can be taken.

As shown in FIGS. 1 and 2, a photographing lens 132 (photographing unit) is provided on the distal end surface 116A of the rigid distal portion 116. As shown in FIG. Behind the imaging lens 132, a CMOS (Complementary Metal-Oxide Semiconductor) imaging device 134 (imaging device, imaging unit), a driving circuit 136, and an AFE 138 (AFE: Analog Front End, imaging unit) are arranged. The image signal is output by the elements of The image pickup device 134 is a color image pickup device, and is composed of a plurality of light receiving elements arranged in a matrix (two-dimensional arrangement) in a specific pattern arrangement (Bayer arrangement, X-Trans (registered trademark) arrangement, honeycomb arrangement, etc.). a plurality of pixels. Each pixel of the imaging device 134 includes a microlens, a red (R), green (G), or blue (B) color filter, and a photoelectric conversion unit (photodiode, etc.). The imaging optical system 130 can also generate a color image from pixel signals of three colors of red, green, and blue, or can generate an image from pixel signals of any one or two colors of red, green, and blue. can also In the first embodiment, the imaging device 134 is a CMOS type imaging device, but the imaging device 134 may be a CCD (Charge Coupled Device) type. Each pixel of the imaging device 134 may further include a violet color filter corresponding to the violet light source 310V and/or an infrared filter corresponding to the infrared light source.

An optical image of the subject is formed on the light-receiving surface (imaging surface) of the imaging device 134 by the imaging lens 132, converted into an electrical signal, and output to the endoscope processor unit 200 via a signal cable (not shown) to produce an image. converted to a signal. As a result, an endoscope image is displayed on the monitor 400 connected to the endoscope processor device 200 .

Illumination lenses 123A and 123B of the illumination unit 123 are provided adjacent to the photographing lens 132 on the tip side end surface 116A of the tip hard portion 116 . Behind the illumination lenses 123A and 123B, an exit end of a light guide 170, which will be described later, is arranged. The incident end is located within the light guide connector 108 .

The user performs imaging at a determined frame rate while inserting or removing the endoscope 100 (insertion unit 104) configured as described above into or withdrawing from the living body to be examined (control of the imaging unit and the medical image acquisition unit 220). ), images in the living body can be sequentially captured.

<Configuration of light source device>
As shown in FIG. 2 , the light source device 300 includes a light source 310 for illumination, an aperture 330 , a condenser lens 340 , a light source control section 350 , and the like, and causes observation light to enter the light guide 170 . The light source 310 includes a red light source 310R, a green light source 310G, a blue light source 310B, and a violet light source 310V that emit narrow band lights of red, green, blue, and violet, respectively. Band light can be applied. The illuminance of the observation light from the light source 310 is controlled by the light source controller 350, and can change (increase or decrease) the illuminance of the observation light and stop the illumination as needed.

Light source 310 can emit red, green, blue, and violet narrowband light in any combination. For example, red, green, blue, and violet narrowband lights can be simultaneously emitted to irradiate white light (ordinary light) as observation light, or any one or two of them can be emitted to narrowband light. Light (special light) can also be applied. The light source 310 may further include an infrared light source that emits infrared light (an example of narrowband light). Alternatively, white light or narrow band light may be emitted as observation light by a light source that emits white light and a filter that transmits white light and narrow band light.

<Wavelength band of light source>
The light source 310 may be a light source that generates light in a white band, light in a plurality of wavelength bands as white band light, or a light source that generates light in a specific wavelength band narrower than the wavelength band of white. The specific wavelength band may be the visible blue or green band, or the visible red band. When the specific wavelength band is the visible blue or green band, it includes a wavelength band of 390 nm or more and 450 nm or less, or 530 nm or more and 550 nm or less, and a peak within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. It may have a wavelength. In addition, when the specific wavelength band is the red band in the visible range, the wavelength band of 585 nm or more and 615 nm or less, or 610 nm or more and 730 nm or less, and the light of the specific wavelength band is 585 nm or more and 615 nm or less or 610 nm or more It may have a peak wavelength within a wavelength band of 730 nm or less.

The light in the above-mentioned specific wavelength band includes a wavelength band in which oxyhemoglobin and reduced hemoglobin have different absorption coefficients, and has a peak wavelength in the wavelength band in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. good. In this case, the specific wavelength band includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm, and 400±10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm It may have peak wavelengths in the following wavelength bands.

The light generated by the light source 310 may include a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm and have a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

Also, the light source 310 may include a light source that emits excitation light with a peak of 390 nm or more and 470 nm or less. In this case, it is possible to acquire a medical image (medical image, in vivo image) having information on the fluorescence emitted by the fluorescent substance in the subject (living body). If fluorescence images are to be obtained, fluorometric dyes (fluorestin, acridine orange, etc.) may be used.

The light source type of the light source 310 (laser light source, xenon light source, LED light source (LED: Light-Emitting Diode), etc.), wavelength, presence or absence of a filter, etc. are preferably configured according to the type and part of the subject, the purpose of observation, and the like. Also, during observation, it is preferable to combine and/or switch the wavelengths of the observation light according to the type and location of the subject, the purpose of observation, and the like. When switching the wavelength, for example, by rotating a disk-shaped filter (rotary color filter) that is placed in front of the light source and provided with a filter that transmits or blocks light of a specific wavelength, the wavelength of the irradiated light is switched. good too.

Further, the image pickup device used in carrying out the present invention is not limited to a color image pickup device in which a color filter is provided for each pixel like the image pickup device 134, and may be a monochrome image pickup device. When a monochrome imaging device is used, it is possible to sequentially switch the wavelength of observation light and perform frame-sequential (color-sequential) imaging. For example, the wavelengths of emitted observation light may be sequentially switched between (violet, blue, green, and red), or broadband light (white light) may be irradiated to rotate color filters (red, green, blue, purple, etc.). The wavelength of the emitted observation light may be switched by . Alternatively, one or a plurality of narrow band lights (green, blue, violet, etc.) may be irradiated and the wavelength of observation light emitted by a rotary color filter (green, blue, violet, etc.) may be switched. The narrowband light may be infrared light of two or more wavelengths (first narrowband light, second narrowband light) having different wavelengths.

By connecting the light guide connector 108 (see FIGS. 1 and 2) to the light source device 300, the observation light emitted from the light source device 300 is transmitted to the illumination lenses 123A and 123B via the light guide 170, and the illumination lenses The observation range is irradiated from 123A and 123B.

<Processor configuration>
The configuration of the endoscope processor device 200 will be described based on FIG. The endoscope processor unit 200 receives an image signal output from the endoscope 100 via an image input controller 202 , performs necessary image processing in an image processing unit 204 , and outputs the signal via a video output unit 206 . do. As a result, an endoscopic image (medical image) is displayed on the monitor 400 (display device). These processes are performed under the control of a CPU 210 (CPU: Central Processing Unit). The communication control unit 205 exchanges information about medical images and regions of interest with a hospital system (HIS: Hospital Information System), a hospital LAN (Local Area Network) (not shown), and/or an external system or network. Controls communication. The recording unit 207 (recording device) records images of the subject (endoscopic images, medical images, medical images), attention area information, display mode setting information, and the like (see FIG. 6 and related description). . Under the control of the CPU 210 and the image processing unit 204, the audio processing unit 209 outputs a message (audio) regarding the detection of the attention area and the detection result from the speaker 209A.

A ROM 211 (ROM: Read Only Memory) is a non-volatile storage element (non-temporary recording medium), and various image processing methods are executed by the CPU 210 and/or the image processing unit 204 (medical image processing apparatus, computer). It stores computer readable code for a program that causes the A RAM 212 (RAM: Random Access Memory) is a storage element for temporary storage during various processes, and can also be used as a buffer during image acquisition.

Note that the user can issue an instruction to execute medical image processing and specify the conditions necessary for execution via the operation unit 208. etc. can be displayed on the monitor 400 .

<Functions of the image processing unit>
FIG. 3 is a functional block diagram of the image processing unit 204. As shown in FIG. The image processing unit 204 includes a medical image acquisition unit 220 (medical image acquisition unit), an information acquisition unit 222 (information acquisition unit), and a mode setting unit 226.
(mode setting unit), display control unit 228 (display control unit), recording control unit 230 (recording control unit), reception unit 232 (reception unit), first time information acquisition unit 234 (first time information acquisition unit) and a second time information acquisition unit 235 (second time information acquisition unit). The information acquisition unit 222 includes recognizers (a detector 224A (detector), a classifier 224B (classifier), and a measuring device 224C (measuring device)). Details of medical image processing using these functions will be described later.

The image processing unit 204 uses the functions described above to calculate the feature amount of a medical image, perform processing to emphasize or reduce components in a specific frequency band, and emphasize or reduce a specific target (region of interest, blood vessel at a desired depth, etc.). Processing can be performed to make it inconspicuous. The image processing unit 204 acquires a special light image having information on a specific wavelength band based on a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band. An image acquisition unit may be provided. In this case, the signal in the specific wavelength band is usually the color information of RGB (R: red, G: green, B: blue) or CMY (C: cyan, M: magenta, Y: yellow) included in the optical image. can be obtained by calculation based on In addition, the image processing unit 204 generates a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band, and a special light image obtained by irradiating light in a specific wavelength band. A feature amount image generating unit that generates a feature amount image by calculation based on at least one of (1) and (2) may be provided, and the feature amount image as a medical image (medical image) may be obtained and displayed. The processing described above is performed under the control of the CPU 210 .

<Realization of functions by various processors>
The function of each unit of the image processing unit 204 described above can be realized using various processors and recording media. Various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. In addition, the above-mentioned various processors include GPUs (Graphics Processing Units), which are processors specialized for image processing, and FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing. Programmable logic devices ( Programmable Logic Device (PLD) is also included. When learning and recognizing images as in the present invention, a configuration using a GPU is effective. Further, the various processors described above also include a dedicated electric circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration specially designed to execute specific processing.

The function of each unit may be implemented by a single processor, or may be implemented by multiple processors of the same or different types (for example, multiple FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Also, a plurality of functions may be realized by one processor. As an example of configuring a plurality of functions in one processor, first, as represented by a computer, one processor is configured by a combination of one or more CPUs and software, and this processor has a plurality of functions. There is a form of realization. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that implements the functions of the entire system with a single IC (Integrated Circuit) chip. In this way, various functions are configured using one or more of the various processors described above as a hardware structure. Further, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements. These electrical circuits may be electrical circuits that implement the above-described functions using logical sums, logical products, logical negations, exclusive logical sums, and logical operations in which these are combined.

When the processor or electric circuit described above executes software (program), the software to be executed can be read by a computer (for example, various processors and electric circuits constituting the image processing unit 204, and/or combinations thereof). The code is stored in a non-temporary recording medium such as ROM 211 (ROM: Read Only Memory), and the computer refers to the software. The software stored in the non-temporary recording medium is a program for executing the medical image processing method according to the present invention and data used for execution (data used to specify the display mode, detector 224A, classifier 224B , and parameters used by meter 224C, etc.). Instead of the ROM 211, the code may be recorded in non-temporary recording media such as various magneto-optical recording devices and semiconductor memories. When processing using software, for example, the RAM 212 (RAM: Random Access Memory) is used as a temporary storage area, and the data stored in, for example, an EEPROM (Electronically Erasable and Programmable Read Only Memory) (not shown) is referred to. can also You may use the recording part 207 as a "non-temporary recording medium."

<Recogniser based on trained model>
The above-described recognizer (detector 224A, classifier 224B, and measuring device 224C) is a trained model such as CNN (Convolutional Neural Network), SVM (Support Vector Machine) (image set composed of images of living organisms) can be configured using a model trained using The layer configuration when the recognizer is configured by CNN will be described below.

<Example of layer configuration of CNN>
FIG. 4 is a diagram showing an example of the layer configuration of the CNN. In the example shown in part (a) of FIG. 4, CNN 562 includes an input layer 562A, an intermediate layer 562B, and an output layer 562C. The input layer 562A inputs an endoscopic image (medical image) acquired by the medical image acquisition unit 220 and outputs a feature amount. The intermediate layer 562B includes a convolution layer 564 and a pooling layer 565, and receives feature values output by the input layer 562A to calculate other feature values. These layers have a structure in which a plurality of "nodes" are connected by "edges" and hold a plurality of weight parameters. The value of the weight parameter changes as learning progresses. CNN 562 may include a fully coupled layer 566 as in the example shown in part (b) of FIG. The layer configuration of the CNN 562 is not limited to the case where the convolutional layer 564 and the pooling layer 565 are repeated one by one, and any layer (for example, the convolutional layer 564) may be continuously included. Multiple contiguous all-bonded layers 566 may also be included.

<Treatment in intermediate layer>
The intermediate layer 562B calculates feature amounts by convolution and pooling processing. The convolution operation performed in the convolution layer 564 is processing for obtaining a feature map by convolution operation using a filter, and plays a role of feature extraction such as edge extraction from an image. A "feature map" of one channel (one sheet) is generated for one filter by a convolution operation using this filter. The size of the "feature map" is downscaled by the convolution, getting smaller with each layer of convolution. The pooling process performed by the pooling layer 565 is a process of reducing (or enlarging) the feature map output by the convolution operation to create a new feature map. It plays the role of giving robustness to Middle layer 562B can be composed of one or more layers that perform these processes.

FIG. 5 is a schematic diagram showing a configuration example of the intermediate layer 562B of the CNN 562 shown in FIG. In the first (first) convolutional layer of the intermediate layer 562B, a convolution operation between an image set composed of a plurality of medical images (an image set for learning at the time of learning and an image set for recognition at the time of recognition) and the filter F1 is performed. done. The image set is composed of N images (N channels) having an image size of H in the vertical direction and W in the horizontal direction. When normal light images are input, the images that make up the image set are 3-channel images of R (red), G (green), and B (blue). This image set and the filter F1 that is convoluted with this image set have N channels (N images). Become. A "feature map" of one channel (one sheet) is generated for one filter F1 by a convolution operation using this filter F1. The filter F2 used in the second convolutional layer has a filter size of 3x3xM, for example for a filter of size 3 (3x3).

Similar to the first convolutional layer, the second to nth convolutional layers perform convolution operations using filters F 2 to F n . The reason why the size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer is that it has been downscaled by the previous convolutional layers or pooling layers. is.

Among the layers of the intermediate layer 562B, low-order feature extraction (edge extraction, etc.) is performed in the convolution layer closer to the input side, and higher-order feature extraction (features related to the shape, structure, etc. of the object) is performed as it approaches the output side. extraction) is performed. When segmentation is performed for the purpose of measurement, etc., the convolutional layers in the latter half are upscaled, and in the last convolutional layer, a "feature map" of the same size as the input image set is obtained. On the other hand, when object detection is performed, position information may be output, so upscaling is not essential.

In addition to the convolution layer 564 and the pooling layer 565, the intermediate layer 562B may include a layer for batch normalization. Batch normalization is a process that normalizes the distribution of data in units of mini-batch during learning, and plays the role of speeding up learning, reducing dependence on initial values, and suppressing over-learning.

<Processing in the output layer>
The output layer 562C is a layer that detects the position of a region of interest appearing in an input medical image (normal light image, special light image) based on the feature amount output from the intermediate layer 562B, and outputs the result. be. When segmentation is performed, the output layer 562C uses the “feature map” obtained from the intermediate layer 562B to grasp the position of the region of interest in the image at the pixel level. That is, it is possible to detect whether or not each pixel of the endoscopic image belongs to the region of interest, and output the detection result. On the other hand, when object detection is performed, determination at the pixel level is not necessary, and the output layer 562C outputs the position information of the object.

For classifier 224B, output layer 562C performs discrimination (classification) on lesions and outputs discrimination results. For example, the output layer 562C classifies the endoscopic images into three categories of "neoplastic", "non-neoplastic" and "other", and the discrimination results are "neoplastic", "non-neoplastic" and "other". (The sum of the three scores is 100%) corresponding to . When outputting the discrimination result, the output layer 562C may include a fully connected layer as the last layer or layers (see part (b) of FIG. 4), or may not include .

In the case of the measuring device 224C, the output layer 562C outputs the measurement result of the region of interest. When the CNN is used for measurement, the target region of interest can be segmented, for example, as described above, and then the image processing unit 204 or the like can perform measurement based on the result. In addition, it is also possible to directly output the measurement value of the target region of interest from the measuring instrument. In the case of directly outputting the measured values, the measured values themselves are learned by the image, which causes a regression problem of the measured values.

When using the CNN with the above configuration, in the learning process, the result output by the output layer 562C is compared with the correct recognition for the image set to calculate the loss (error), and the intermediate layer 562B It is preferable to perform processing (error backpropagation) to update the weight parameters in from the output side layer toward the input side layer.

<Recognition by methods other than CNN>
The recognizers (the detector 224A, the classifier 224B, and the measurer 224C) may perform recognition (detection of attention areas, etc.) by techniques other than CNN. For example, the region of interest can be detected based on the feature amount of the pixels of the acquired medical image. In this case, for example, the detector 224A divides the detection target image into, for example, a plurality of rectangular regions, sets each divided rectangular region as a local region, and sets the feature amount ( For example, hue) is calculated, and a local region having a specific hue is determined as a region of interest from among the local regions. Similarly, classification and measurement based on feature amounts may be performed.

In the endoscope system 10, all of the detector 224A, the classifier 224B, and the measuring device 224C may be operated in parallel to switch display results, or one or two recognizers may be used. You can run it and display the results. Also, endoscope system 10 may include one or two of detector 224A, classifier 224B, and meter 224C, but not all of these recognizers.

<Information recorded in the recording unit>
FIG. 6 is a diagram showing an example of information recorded in the recording unit 207. As shown in FIG. In the example of FIG. 6, an endoscopic image 260 (medical image), a recognition result 262 (detection result, classification result, measurement result), and display mode setting information 264 (display mode when highlighting a region of interest) are recorded. be done. Other information may be recorded together. The recording control unit 230 records these pieces of information in association with each other.

<Medical image processing method>
A medical image processing method in the endoscope system 10 configured as described above will be described with reference to the flowchart of FIG.

<Initial settings>
The receiving unit 232 (receiving unit) receives the user's operation via the operation unit 208 and/or the display mode setting information 264 recorded in the recording unit 207 (for example, turning on or off highlighting, displaying The conditions necessary for executing the medical image processing method are set (step S100: initial setting step). For example, the image acquisition mode (real-time image acquisition or non-real-time image acquisition), characters, figures, symbols used for highlighting, their colors, and the like are set. Note that the conditions may be set or changed during execution of the following steps.

In the first embodiment and other aspects, "real-time" and "real-time images" are substantially multiple medical images captured at a predetermined frame rate, for example, during ongoing examinations and observations. sequential acquisition without significant delay, and images so acquired. "Real-time" and "real-time image" also include cases where image acquisition is slightly delayed from perfect real-time due to image transmission/reception, image processing, and the like. On the other hand, "non-real-time" and "non-real-time images" include, for example, cases in which a plurality of medical images that have already been recorded after an examination is sequentially acquired and displayed.

<Acquisition of endoscopic images>
The medical image acquisition unit 220 (medical image acquisition unit) acquires an endoscopic image (medical image) captured inside the body of the subject (step S110: medical image acquisition step). The medical image acquisition unit 220 sequentially captures the interior of the living body (subject) at a predetermined frame rate using the imaging unit (the imaging lens 132, the imaging device 134, the AFE 138, etc.) of the endoscope 100 (medical device). endoscopic images can be acquired in real time (real-time image acquisition).

The medical image acquisition unit 220 may acquire already captured and recorded endoscopic images in non-real time (non-real time image acquisition). For example, an endoscopic image 260 recorded in the recording unit 207 may be obtained, or an image may be obtained from an external device or system via the communication control unit 205 . In addition, if site information indicating the observation site of the subject has already been acquired, an endoscopic image (medical image) captured with observation light in a wavelength band corresponding to the site indicated by the site information can be acquired. good. For example, an image of the stomach taken with white light and an image of the esophagus taken with blue narrow band light can be acquired. The display control unit 228 causes the monitor 400 to display the acquired endoscopic image.

<Detection of attention area>
The detector 224A detects a region of interest from the endoscopic image (medical image) (step S120: information acquisition step). When detecting a region of interest, as described above, the detector 224A grasps the position of the region of interest reflected in the image at the pixel level from the "feature map" (that is, detects the region of interest for each pixel of the endoscopic image). (detect whether or not it belongs) and output the detection result. Examples of regions of interest (regions of interest) to be detected include polyps, cancers, colonic diverticula, inflammation, treatment scars (EMR scars (EMR: Endoscopic Mucosal Resection), ESD scars (ESD: Endoscopic Submucosal Dissection), clipping sites, etc.), Bleeding points, perforations, vascular atypia and the like can be mentioned. As in the case of detection, classification and measurement may be performed by the classifier 224B and the measuring device 224C in step S120. An endoscopic image (medical image) to be detected, classified, and measured may be a real-time image or a non-real-time image.

<Display mode setting>
Mode setting unit 226 determines whether the image acquired in step S110 is a real-time image (step S130: determination step, mode setting step). Based on this determination result, the mode setting unit 226 sets the display mode for the real-time image if it is a real-time image (YES in step S130) (step S140: mode setting step), and if it is a non-real-time image ( If NO in step S130), the display mode for the non-real-time image is set (step S150: mode setting step).

<Display of endoscopic image>
The display control unit 228 causes the monitor 400 (display device) to display the endoscopic image (medical image) in the display mode (the mode of highlighting the region of interest) set in step S140 or S150 (step S160: display control step).

<Display example for real-time images>
FIG. 8 is a diagram showing a display example (when highlighting is set to ON) in the case of a real-time image. Parts (b) and (c) of the figure show a state in which an attention area 802 is highlighted (superimposed on the endoscopic image 800) by a figure 804 and an arrow 806, respectively, in the endoscopic image 800. FIG. These highlighted displays directly indicate the position of the region of interest, but since the region of interest itself is not overlaid with a figure, it does not interfere with observation of the mucous membrane structure. Part (d) of the figure shows a state in which a frame 808 is displayed partly outside the screen (outside the display range of the endoscopic image 800 in the display area of the monitor 400). In this mode, notification is made in an area outside the screen near the attention area 802 . Parts (e) and (f) of the same figure show a state in which a frame 808 is displayed partly outside the screen and a state in which an asterisk 812 is displayed outside the screen, respectively. Regardless, it is a mode in which an alert is issued when the attention area exists within the observation screen. The examples shown in parts (b) to (f) of FIG. 8 are representations suitable for real-time display because they do not hinder observation to a low degree. Part (a) of FIG. 8 is a diagram (for reference) showing a state in which highlighting is not performed, and shows an endoscopic image 800 in which a region of interest 802 is shown.

<Display example for non-real-time images>
FIG. 9 is a diagram showing a display example of a non-real-time image (when highlighting is set to ON). Part (a) of the figure shows a state in which the attention area 802 is filled in, and part (b) shows the endoscopic image 820 and the recognition result display image 822 on two screens. , a state in which a symbol 824 is displayed at a position corresponding to . The display mode of FIG. 9 is not preferable for real-time display because it may interfere with observation, but it poses a problem for post-save display (including the case where an image recorded in the recording unit 207 or the like is acquired and displayed in non-real time). not. In addition, since the attention area is observed in detail in the display after storage, a display method such as parts (d) to (f) in FIG. 8 in which the position of the attention area cannot be grasped accurately is not preferable. In addition, since the display size of each image may be small in order to display a list of images acquired during an examination, a more conspicuous enhancement process is preferred. Therefore, an aspect as shown in FIG. 9 can be adopted.

When two or more screens are displayed as shown in part (b) of FIG. 9, the screens may be divided and displayed on different screens on the same monitor, or may be displayed on a plurality of monitors. When changing from the mode of displaying on two screens to the mode of displaying on one screen, the display position and/or size of the observation screen (endoscopic image, recognition result) may be changed accordingly. By doing so, it is possible to display an observation screen that fully utilizes the monitor size.

8 and 9 merely show an example of the display mode. In the case of a real-time image, the display as shown in FIG. 9 may be performed, and in the case of a non-real-time image, the display as shown in FIG. 8 may be performed according to the user's preference and the purpose of observation. Also, FIG. 9 shows an example in which highlighting is set to ON for a non-real-time image, but highlighting may be turned off in response to a user's operation or when predetermined conditions are met. In the case of real-time images as well, highlighting may be turned off according to the user's operation or when predetermined conditions are met. In addition to the display modes illustrated in FIGS. 8 and 9, color tone and gradation may be changed, frequency processing, and the like may be performed. Such setting of the display mode can be performed by the receiving unit 232 receiving a user operation (display mode setting operation) via the operation unit 208 and by the mode setting unit 226 based on the setting operation.

The mode setting unit 226 may set the display mode based on the classification result and the measurement result of the endoscopic image, and the display control unit 228 may display according to the setting. For example, the classification results of endoscopic images can be classified into "neoplastic", "non-neoplastic", and "others" by setting a display mode using characters, graphics, symbols, etc., and displaying in that mode. The color may be changed according to the classification result. Further, for example, the measurement result may be displayed numerically such as "diameter 5 mm", or a figure, symbol, or the like indicating the size of the attention area may be displayed. Again, the colors may be changed according to the results.

Note that the number of monitors (display devices) in the endoscope system 10 is not particularly limited. Both real-time and non-real-time images may be displayed on monitor 400 . In this case, for example, a real-time image can be displayed, and a specific frame can be freeze-displayed according to a user's instruction or the like. Alternatively, a monitor for real-time images and a monitor for non-real-time images may be provided separately, and images may be displayed on the respective monitors. When a separate monitor is provided, for example, a real-time image may be displayed on the monitor 400, and an image obtained by an inspection may be displayed on another monitor in a non-real-time manner, such as when creating a report. Both aspects are included in "displaying an endoscopic image (medical image) on a display device".

<Display of part information>
The display control unit 228 may display the region information (information indicating the region shown in the endoscopic image) along with the highlighted display of the endoscopic image. FIG. 10 is a diagram showing a display example of part information. Part (a) of FIG. 10 is an example in which region information 836 is displayed in characters (“gastric”) in addition to a frame 834 surrounding a region of interest 832 in an endoscopic image 830. Part (b) of FIG. 8 is an example in which region information is displayed by an arrow 842A on a schema 842 in an endoscopic image 840. FIG. In addition, the site information may be displayed by icons, and the display mode (contents, positions, shapes, colors, etc. of characters and symbols) may be changed according to the site. Note that the site information can be acquired by analysis of an endoscopic image, user input, external equipment using electromagnetic waves or ultrasonic waves, and the like.

<Acquisition and processing of images according to site>
The medical image acquisition unit 220 acquires an endoscopic image (medical image) captured with observation light in a wavelength band corresponding to the region reflected in the endoscopic image, and the display control unit 228 acquires the observation light in the wavelength band. You may display the recognition result with respect to the medical image image|photographed by the monitor 400 (display apparatus). For example, in the case of the stomach, an image taken with white light (normal light) is recognized, and in the case of the esophagus, an image taken with special light (blue narrow band light) such as BLI (Blue Laser Imaging: registered trademark) is recognized. can be provided. Depending on the site, images taken with special light such as LCI (Linked Color Imaging: registered trademark) and processed (in the case of LCI, the saturation and hue differences of colors close to the color of the mucous membrane are expanded) may be used. Image processing according to the site can be performed by the image processing unit 204 .

<Recording of endoscopic image and attention area information>
The recording control unit 230 (recording control unit) records the endoscopic image and the information on the attention area in the recording unit 207 as the endoscopic image 260 and the recognition result 262, respectively (step S170: recording control step). If classification and measurement are performed in step S120, the results are recorded. Further, when the endoscopic image is subjected to image processing, the processed endoscopic image can be recorded, and the display mode setting information 264 is recorded together with the endoscopic image 260 and the recognition result 262. good too. These pieces of information are preferably recorded in association with each other. The image processing unit 204 determines whether or not to end the processing (step S180), and if it continues (NO in step S180), returns to step S110, and performs the above-described processing for the next frame, for example.

In this manner, the endoscope system 10 can highlight medical images in real time. In addition, since the medical image and the information of the region of interest are associated and recorded in a recording device (the information is recorded without being superimposed on the image), the medical image and information are used after the fact (in the case of non-real-time images) can also be displayed in a display mode according to the purpose of observation or the desire of the user.

<Other display modes>
<Display mode setting based on elapsed time information>
The display mode for highlighting the region of interest may be set based on information on the elapsed time during which the endoscopic image is displayed on the monitor 400 . FIG. 11 is a flow chart showing processing for setting a display mode based on elapsed time information. Since steps other than steps S152 to S156 are the same as in FIG. 7, detailed description thereof will be omitted. The first time information acquisition unit 234 (first time information acquisition unit) acquires an elapsed time ( first elapsed time) (step S152: time information acquisition step), and mode setting unit 226 determines whether or not the first elapsed time is equal to or greater than the threshold value (step S154: determination step). The first time information acquisition unit 234 can acquire (calculate) information on the elapsed time based on, for example, the frame rate of shooting and the number of frames in which the same attention area is captured. If the first elapsed time is equal to or greater than the threshold value (YES in step S154), mode setting unit 226 resets the display mode based on the information on the first elapsed time (step S156: mode setting step). If the elapsed time of 1 is less than the threshold value (NO in step S154), the display mode is maintained.

FIG. 12 is a diagram showing how the display mode is reset according to the first elapsed time. In FIG. 8, a figure 804A surrounding an attention area 802 is indicated by a dotted line, and the degree of emphasis is low. The mode setting unit 226 and the display control unit 228 can switch between, for example, the state of part (b) of FIG. 8 (the graphic 804 is displayed in solid lines) and the state of FIG. 12 according to the first elapsed time. For example, when the first elapsed time is short, it is displayed as shown in FIG. 8(b) to increase the degree of emphasis to attract attention, and thereafter the degree of emphasis can be reduced as shown in FIG. Depending on the user's operation and observation purpose, the display may be performed in a reverse pattern. Also, instead of changing the line type of figures as in FIGS. 8 and 12, the shape, size, color and brightness of figures and symbols may be changed.

Also, the highlighting may be switched on and off according to the elapsed time, or the degree of highlighting may be increased (or decreased). For example, the highlighting may be turned off at the beginning of detection and then turned on (or the degree of emphasis may be increased with the first elapsed time), or vice versa (or the degree of emphasis may be decreased with the first elapsed time). ) can be used. Alternatively, it may be turned on and off repeatedly at predetermined time intervals.

For example, in the case of sequentially acquiring and displaying endoscopic images at a predetermined frame rate, such processing is performed during the elapsed time (first elapsed time) in which the same attention area is shown in a plurality of frames. This can be applied when setting the display mode accordingly.

Similarly, the display mode when highlighting a region of interest may be set based on information on the elapsed time during which the same endoscopic image is displayed. The processing in this case can be performed in the same manner as in the flowchart of FIG. The second time information acquisition unit 235 (second time information acquisition unit) acquires information on the elapsed time (second elapsed time) during which the same endoscopic image is displayed on the monitor 400 (display device). Then (step S152: time information acquisition step), mode setting unit 226 determines whether or not the second elapsed time is equal to or greater than the threshold value (step S154: determination step). If the second elapsed time is equal to or greater than the threshold (YES in step S154), mode setting unit 226 resets the display mode based on the second elapsed time information (step S156: mode setting step). If the elapsed time of 2 is less than the threshold value (NO in step S154), the display mode is maintained. The display mode can be performed in the same manner as the case based on the first elapsed time.

Such processing changes the display mode according to the elapsed time (second elapsed time) during which a specific image is continuously displayed when, for example, an endoscopic image is continuously observed by freeze display or the like. Can be applied when setting

<Batch setting of display mode for multiple medical images>
Batch setting of display modes for a plurality of medical images will be described with reference to the flowchart of FIG. 13 . 13 are the same as those in FIGS. 7 and 11 except for steps S162 and S164, and detailed description thereof will be omitted.

When checking endoscopic images after an examination, a list of a plurality of images may be displayed. For example, the display control unit 228 displays a list of a plurality of medical images recorded in the recording unit 207 (recording device) on the monitor 400 (display device) (step S160: display control step). Part (a) of FIG. 14 shows a state in which six images are displayed with highlighting turned off, and a region of interest 802 is detected in endoscopic images 850 and 852 . When the highlighting is turned on from this state, the operation is troublesome if the display mode of each image is individually set by an operation such as clicking the image. Therefore, the mode setting unit 226 determines whether or not there is a change operation (step S162: determination step). For example, when the user performs a change operation via the operation unit 208 and the reception unit 232 receives the operation, the mode setting unit 226 can determine that "a change operation has been performed."

When the change operation is performed, the mode setting unit 226 sets the mode of highlighting for the plurality of medical images displayed as a list to a common mode. For example, highlighting for a plurality of endoscopic images (medical images) is collectively set to on (or off). The display control unit 228 re-displays the displayed endoscopic images in the changed display mode (common mode) (step S164: display control step). The display control unit 228 displays frames 854 and 856 with respect to the endoscopic images 850 and 852 in which the attention area 802 is detected (the endoscopic images 850 and 852), for example, as shown in part (b) of FIG. For images other than , the frame is not displayed because the region of interest is not detected). That is, the "common mode" is "to display a frame surrounding the entire image with respect to the image showing the attention area". Such a display allows the user to easily select an image showing the region of interest (that is, an image that requires close examination) from a large number of images after the examination.

In addition to or in place of the setting of the display mode for the list display described above, the emphasis method may be changed between the list display and the single display. For example, when displaying a list, a frame surrounding the entire image is displayed as shown in part (b) of FIG. or frame can be displayed. This display mode is displayed as shown in FIG. 14(b) since only the information "the attention area is shown" or "is not shown" is sufficient when the list is displayed, and the attention area in the image is displayed when the single display is performed. Since information on the position of is required, it is displayed as shown in FIG. 8(b).

<Application to other than endoscopic images>
In the above-described embodiment, the case of detecting a region of interest using an endoscopic image, which is one aspect of a medical image (medical image), has been described. However, the medical image processing apparatus and the medical image processing method according to the present invention The present invention can also be applied to the case of using medical images other than endoscopic images, such as an ultrasonic diagnostic imaging device.

(Appendix)
In addition to the aspects described above, the configurations described below are also included in the scope of the present invention.

(Appendix 1)
The medical image analysis processing unit detects a region of interest, which is a region of interest, based on the feature amount of the pixels of the medical image,
A medical image analysis result acquisition unit is a medical image processing apparatus that acquires the analysis result of the medical image analysis processing unit.

(Appendix 2)
The medical image analysis processing unit detects the presence or absence of an object of interest based on the feature amount of the pixels of the medical image,
A medical image analysis result acquisition unit is a medical image processing apparatus that acquires the analysis result of the medical image analysis processing unit.

(Appendix 3)
The medical image analysis result acquisition unit
Obtained from a recording device that records the analysis results of medical images,
A medical image processing apparatus in which an analysis result is either a region of interest, which is a region of interest included in a medical image, or the presence or absence of a target of interest, or both.

(Appendix 4)
A medical image processing apparatus in which a medical image is a normal light image obtained by irradiating light in a white band or light in a plurality of wavelength bands as light in a white band.

(Appendix 5)
A medical image is an image obtained by irradiating light of a specific wavelength band,
A medical imaging device in which the specific wavelength band is narrower than the white wavelength band.

(Appendix 6)
A medical image processing device in which the specific wavelength band is the visible blue or green band.

(Appendix 7)
The specific wavelength band includes a wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. Image processing device.

(Appendix 8)
A medical image processing device whose specific wavelength band is the visible red band.

(Appendix 9)
The specific wavelength band includes a wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less, and the light of the specific wavelength band has a peak wavelength within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. Image processing device.

(Appendix 10)
The specific wavelength band includes a wavelength band in which oxyhemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the specific wavelength band has a peak wavelength in the wavelength band in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. Medical imaging equipment.

(Appendix 11)
The specific wavelength band includes a wavelength band of 400 ± 10 nm, 440 ± 10 nm, 470 ± 10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light in the specific wavelength band is 400 ± 10 nm, 440 ± 10 nm, 470 ± A medical image processing apparatus having a peak wavelength in a wavelength band of 10 nm or from 600 nm to 750 nm.

(Appendix 12)
A medical image is an in vivo image of the inside of a living body,
An in vivo image is a medical image processing device that has information on the fluorescence emitted by fluorescent substances in the living body.

(Appendix 13)
A medical image processing apparatus in which fluorescence is obtained by irradiating the living body with excitation light having a peak of 390 to 470 nm.

(Appendix 14)
A medical image is an in vivo image of the inside of a living body,
A medical imaging device in which the specific wavelength band is the wavelength band of infrared light.

(Appendix 15)
The specific wavelength band includes a wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less, and the light of the specific wavelength band has a peak wavelength in the wavelength band of 790 nm or more and 820 nm or less or 905 nm or more and 970 nm or less. processing equipment.

(Appendix 16)
The medical image acquisition unit acquires a special light image having information on a specific wavelength band based on a normal light image obtained by irradiating light of a plurality of wavelength bands as light of a white band or light of a white band. comprising an optical image acquisition unit,
A medical image processing device in which the medical image is a special light image.

(Appendix 17)
A medical image processing device in which a signal in a specific wavelength band is normally obtained by calculation based on RGB or CMY color information contained in an optical image.

(Appendix 18)
By calculation based on at least one of a normal light image obtained by irradiating light of a plurality of wavelength bands as white band light or light of a white band, and a special light image obtained by irradiating light of a specific wavelength band, A feature amount image generation unit that generates a feature amount image,
A medical image processing apparatus in which a medical image is a feature amount image.

(Appendix 19)
a medical image processing apparatus according to any one of Appendices 1 to 18;
an endoscope that acquires an image by irradiating at least one of light in a white wavelength band or light in a specific wavelength band;
An endoscope device comprising:

(Appendix 20)
A diagnosis support device comprising the medical image processing device according to any one of appendices 1 to 18.

(Appendix 21)
A medical service support apparatus comprising the medical image processing apparatus according to any one of Appendices 1 to 18.

Although the embodiments and other examples of the present invention have been described above, the present invention is not limited to the above aspects, and various modifications are possible without departing from the spirit of the present invention.

10 endoscope system 100 endoscope scope 102 hand operation section 104 insertion section 106 universal cable 108 light guide connector 112 flexible section 114 bending section 116 distal end rigid section 116A distal end face 123 illumination section

123A illumination lens 123B illumination lens 126 forceps port 130 photographing optical system 132 photographing lens 134 image sensor 136 drive circuit 138 AFE
141 Air supply/water supply button 142 Suction button 143 Function button 144 Shooting button 170 Light guide 200 Processor 202 Image input controller 204 Image processing unit 205 Communication control unit 206 Video output unit 207 Recording unit 208 Operation unit 209 Audio processing unit 209A Speaker 210 CPU
211 ROMs
212 RAM
220 medical image acquisition unit 222 information acquisition unit 224A detector 224B classifier 224C measuring instrument 226 mode setting unit 228 display control unit 230 recording control unit 232 reception unit 234 first time information acquisition unit 235 second time information acquisition unit 260 Endoscope image 262 Recognition result 264 Display mode setting information 300 Light source device 310 Light source 310B Blue light source 310G Green light source 310R Red light source 310V Violet light source 330 Diaphragm 340 Collecting lens 350 Light source controller 400 Monitor 562 CNN
562A Input layer 562B Intermediate layer 562C Output layer 564 Convolution layer 565 Pooling layer 566 Fully connected layer 800 Endoscope image 802 Attention area 804 Figure 804A Figure 806 Arrow 808 Frame 810 Frame 812 Star 820 Endoscope image 822 Recognition result display image 830 endoscopic image 832 attention area 834 frame 836 part information 840 endoscopic image 842 schema 842A arrow 850 endoscopic image 852 endoscopic image 854 frame 856 frame F1 filter F2 filter Fn filter S100 to S170 medical image processing method each step of

Claims (7)

a medical image acquisition unit that acquires a medical image;
an information acquisition unit that acquires information on the region of interest in the medical image;
Based on the information, the degree of highlighting of the region of interest when the medical image is a non-real-time image obtained in non-real time is set to the degree of emphasis when the medical image is a real-time image obtained in real time. a mode setting unit that sets a display mode higher than the degree of emphasis;
a display control unit that causes a display device to display the medical image in the set display mode;
a recording control unit that associates the medical image with the information and records it in a recording device;
with
The display control unit causes the display device to display, together with the display of the medical image, region information, which is information indicating a region captured in the medical image. 2. The medical image processing apparatus according to claim 1, wherein the display control unit causes the display device to display a frame surrounding the region of interest shown in the medical image and characters indicating the region information. 2. The medical image processing apparatus according to claim 1, wherein the display control unit displays the part information by an arrow on a schematic diagram and/or an icon indicating the part. The medical image processing apparatus according to any one of claims 1 to 3, wherein the display control unit changes a display mode of the part information according to the part. 5. The display control unit according to any one of claims 1 to 4, wherein the display control unit changes one or more of content, position, shape, and color of characters and/or symbols as the part information according to the part. Medical imaging equipment. a medical image processing apparatus according to any one of claims 1 to 5;
the display device;
An endoscope system comprising: an endoscope to be inserted into a subject, the endoscope having an imaging unit that captures the medical image. A method of operating a medical imaging device comprising a processor, comprising:
The processor
acquire medical images,
Acquiring information on a region of interest in the medical image;
Based on the information, the degree of highlighting of the region of interest when the medical image is a non-real-time image obtained in non-real time is set to the degree of emphasis when the medical image is a real-time image obtained in real time. Set to a display mode higher than the degree of emphasis,
causing the display device to display the medical image in the set display mode;
causing the medical image and the information to be associated and recorded in a recording device;
An operation method for causing the display device to display, together with the display of the medical image, region information, which is information indicating a region captured in the medical image.

Images