WO2022004056A1

WO2022004056A1 - Endoscope system and method for operating same

Info

Publication number: WO2022004056A1
Application number: PCT/JP2021/008740
Authority: WO
Inventors: 康太郎檜谷
Original assignee: 富士フイルム株式会社
Priority date: 2020-07-03
Filing date: 2021-03-05
Publication date: 2022-01-06
Also published as: JPWO2022004056A1

Abstract

Provided are an endoscope system and a method for operating the same which, when a lesion candidate region is detected, prevent diagnosis assistance information relating to the lesion candidate region from obstructing the visibility of a subject being observed even in situations where the movement of the subject being observed changes in an image.　When a lesion candidate region (DR) is recognized by a recognition process, a diagnosis assistance information image is generated that displays diagnosis assistance information relating to the lesion candidate region (DR) on an observation image based on a first image signal. Whether to display or not display at least first diagnosis assistance information (76) among pieces of the diagnosis assistance information is controlled by the movement of the subject being observed in the diagnosis assistance information image.

Description

Endoscope system and how to operate it

The present invention relates to an endoscopic system for displaying diagnostic support information regarding a lesion candidate area such as a suspicion rate of a lesion and a method for operating the same.

In the medical field, diagnosis using medical images is widely practiced. For example, as a device using a medical image, there is an endoscope system including a light source device, an endoscope, and a processor device. In the endoscope system, an observation object is irradiated with illumination light, and the observation object illuminated by the illumination light is imaged by an image pickup sensor to acquire an endoscope image as a medical image. The endoscopic image is displayed on the monitor and used for diagnosis.

Further, in recent endoscopic systems, lesion candidate regions are detected from images obtained by imaging an observation target. When a lesion candidate area is detected, the user is notified that the lesion candidate area has been detected by displaying the area around the lesion candidate area with diagnostic support information such as a bounding box. However, if the notification to the user reduces the visibility of the observation target, it may hinder the effective diagnosis by the user. Regarding this, in Patent Document 1, when the number of lesion candidate regions is larger than a predetermined threshold value or the size is larger than a predetermined threshold value, the alert image regarding the notification to the user is hidden. The visibility of the observation target is not reduced.

Japanese Unexamined Patent Publication No. 2011-255006

In endoscopic diagnosis, screening is performed to move the tip of an endoscope with an image sensor at a constant speed or higher, and subcommittee observation is performed to stop the movement of the tip of the endoscope to distinguish lesions. It has been. Therefore, during the diagnosis of the endoscope, the movement of the observation target in the image of the endoscope changes. Even under the situation where the movement of the observation target in the image changes in this way, when the lesion candidate area is detected, the diagnostic support information regarding the lesion candidate area should not interfere with the visibility of the observation target. I was asked.

The present invention prevents the diagnostic support information regarding the lesion candidate region from interfering with the visibility of the observation target when the lesion candidate region is detected even in a situation where the movement of the observation target in the image changes. It is an object of the present invention to provide an endoscope system and a method of operating the same.

The endoscope system of the present invention has a light source unit that emits first illumination light and second illumination light having different emission spectra, and a first illumination period that emits the first illumination light and a second illumination light that emits the second illumination light. In the case of automatically switching between the two illumination periods, the light was illuminated by the light source processor that emits the first illumination light in the first emission pattern and emits the second illumination light in the second emission pattern, and the first illumination light. Image control is provided with an image sensor that captures the first image signal obtained by imaging the observation target and outputs the second image signal that captures the observation target illuminated by the second illumination light, and an image control processor. When the processor performs recognition processing on the second image signal, acquires the movement of the observation target in the image, and recognizes the lesion candidate region by the recognition processing, the observation image based on the first image signal is obtained. On the other hand, a diagnostic support information image that displays diagnostic support information regarding the lesion candidate area is generated, and at least the first diagnostic support information of the diagnostic support information is displayed or hidden in the diagnostic support information image depending on the movement of the observation target. Control.

The image control processor displays diagnostic support information when the movement of the observation target is less than the first threshold value, and hides the diagnostic support information when the movement of the observation target is equal to or higher than the first threshold value. It is preferable to perform the first display control. The image control processor hides the diagnostic support information when the movement of the observation target is less than the second threshold value, and displays the diagnostic support information when the movement of the observation target is equal to or more than the second threshold value. It is preferable to perform the second display control.

The image control processor displays diagnostic support information when the movement of the observation target is less than the first threshold value, and hides the diagnostic support information when the movement of the observation target is equal to or higher than the first threshold value. The first display control or the diagnosis support information is hidden when the movement of the observation target is less than the second threshold value, and the diagnosis support information is displayed when the movement of the observation target is equal to or more than the second threshold value. When it is possible to perform the second display control for display, it is preferable to have a user interface for manually switching between the first display control and the second display control.

The image control processor displays diagnostic support information when the movement of the observation target is less than the first threshold value, and hides the diagnostic support information when the movement of the observation target is equal to or higher than the first threshold value. The first display control or the diagnosis support information is hidden when the movement of the observation target is less than the second threshold value, and the diagnosis support information is displayed when the movement of the observation target is equal to or more than the second threshold value. When it is possible to perform the second display control for displaying, it is preferable that the image control processor automatically switches between the first display control and the second display control.

The image control processor switches to the first display control when the movement of the observation target is less than the switching threshold value, and switches to the second display control when the movement of the observation target is equal to or more than the switching threshold value. Is preferable. When the emission spectra of the second illumination light are different in each second illumination period, and a plurality of lesion candidate images are recognized by the recognition process based on the second image signal corresponding to each second illumination light. It is preferable that the display of the second diagnostic support information, which maintains the display regardless of the movement of the observation target among the diagnostic support information, differs depending on the emission spectrum of the second illumination light.

The first light emission pattern has a first A light emission pattern in which the number of frames in the first lighting period is the same in each of the first lighting periods, and the number of frames in the first lighting period is different in each first lighting period. It is preferably one of the first B emission patterns.

In the second emission pattern, the number of frames in the second illumination period is the same in each second illumination period, and the emission spectrum of the second illumination light is the same in each second illumination period. The second B emission pattern, the second, in which the pattern and the number of frames in the second illumination period are the same in each second illumination period, and the emission spectrum of the second illumination light is different in each second illumination period. The second C emission pattern and the second illumination period in which the number of frames in the illumination period is different in each second illumination period and the emission spectrum of the second illumination light is the same in each second illumination period. The number of frames is different in each second illumination period, and the emission spectrum of the second illumination light is one of the second D emission patterns different in each second illumination period. preferable.

The present invention automatically sets a light source unit that emits a first illumination light and a second illumination light having different emission spectra, a first illumination period that emits the first illumination light, and a second illumination period that emits the second illumination light. A light source processor that emits the first illumination light in the first emission pattern and emits the second illumination light in the second emission pattern, and obtains an image of an observation target illuminated by the first illumination light. In the method of operating an endoscope system including an image pickup sensor that captures an observation target illuminated by a second illumination light and outputs a second image signal, and an image control processor. The image control processor performs recognition processing on the second image signal, acquires the movement of the observation target in the image, and recognizes the lesion candidate region by the recognition processing, observation based on the first image signal. A diagnostic support information image that displays diagnostic support information regarding a lesion candidate area is generated for the image, and at least the first diagnostic support information of the diagnostic support information is displayed or not displayed in the diagnostic support information image depending on the movement of the observation target. Control the display.

According to the present invention, even when the movement of the observation target in the image changes, when the lesion candidate region is detected, the diagnostic support information regarding the lesion candidate region does not interfere with the visibility of the observation target. can do.

It is an external view of an endoscope system. It is a block diagram which shows the function of an endoscope system. It is a graph which shows the spectrum of purple light V, blue light B, green light G, and red light R. It is explanatory drawing which shows the 1A light emission pattern or the 2nd A light emission pattern in the diagnosis support mode. It is explanatory drawing which shows the 1B light emission pattern in the diagnosis support mode. It is explanatory drawing which shows the 2nd B light emission pattern in the diagnosis support mode. It is explanatory drawing which shows the 2nd C light emission pattern in the diagnosis support mode. It is explanatory drawing which shows the 2D light emission pattern in the diagnosis support mode. It is a graph which shows the spectral transmittance of each color filter of an image pickup sensor. It is explanatory drawing which shows the 1st imaging period and the 2nd imaging period. It is explanatory drawing which shows lighting control, analysis processing, and image display in a diagnosis support mode in chronological order. It is a block diagram which shows the function of a diagnosis support information processing part. It is an image diagram which shows the 1st diagnosis support information and the 2nd diagnosis support information. It is explanatory drawing which shows the 1st display control. It is explanatory drawing which shows the 2nd display control. It is explanatory drawing which shows the switching threshold value used for automatic switching of display control. It is explanatory drawing which shows the display method of the 2nd diagnosis support information in the case of using a plurality of 2nd illumination lights having different emission spectra. It is a flowchart which shows a series flow of a diagnosis support mode.

In FIG. 1, the endoscope system 10 has an endoscope 12, a light source device 14, a processor device 16, a display 18, and a user interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 16. The endoscope 12 has an insertion portion 12a to be inserted into the body to be observed, an operation portion 12b provided at the base end portion of the insertion portion 12a, and a curved portion 12c and a tip provided on the tip end side of the insertion portion 12a. It has a portion 12d. The curved portion 12c bends by operating the angle knob 12e of the operating portion 12b. The tip portion 12d is directed in a desired direction by the bending motion of the bending portion 12c.

In addition to the angle knob 12e, the operation unit 12b includes a mode switching switch (mode switching switch) 12f used for mode switching operation, and a still image acquisition instruction unit 12g used for instructing acquisition of a still image to be observed. A zoom operation unit 12h used for operating the zoom lens 43 (see FIG. 2) is provided.

The endoscope system 10 has three modes: a normal observation mode, a special observation mode, and a diagnostic support mode. In the normal observation mode, a normal observation image having a natural hue is displayed on the display 18 by illuminating the observation target with normal light such as white light and taking an image. In the special observation mode, a special observation image emphasizing a specific structure is displayed on the display 18 by illuminating the observation target with special light having a emission spectrum different from that of normal light and taking an image. In the diagnostic support mode, when the first illumination light and the second illumination light having different emission spectra are switched to emit light and the lesion candidate region or the like is recognized by the recognition process (AI (Artificial Intelligence) or the like), the first A diagnostic support information image that displays diagnostic support information regarding the lesion candidate region is displayed on the display 18 with respect to the observation image based on the illumination light.

When the user operates the still image acquisition instruction unit 12g, a signal related to the still image acquisition instruction is sent to the endoscope 12, the light source device 14, and the processor device 16. In the processor device 16, when the still image acquisition instruction is given, when the still image acquisition instruction is given, the still image to be observed is saved in the still image storage memory 69 of the processor device 16.

The processor device 16 is electrically connected to the display 18 and the user interface 19. The display 18 outputs and displays an image to be observed, information incidental to the image to be observed, and the like. The user interface 19 has a keyboard, a mouse, a touch pad, and the like, and has a function of accepting input operations such as function settings. An external recording unit (not shown) for recording an image, image information, or the like may be connected to the processor device 16.

In FIG. 2, the light source device 14 includes a light source unit 20 and a light source processor 21 that controls the light source unit 20. The light source unit 20 has, for example, a plurality of semiconductor light sources, each of which is turned on or off, and when the light source unit 20 is turned on, the light emission amount of each semiconductor light source is controlled to emit illumination light for illuminating the observation target. In the present embodiment, the light source unit 20 is a V-LED (Violet Light Emitting Diode) 20a, a B-LED (Blue Light Emitting Diode) 20b, a G-LED (Green Light Emitting Diode) 20c, and an R-LED (Red Light). Emitting Diode) It has a 20d 4-color LED.

As shown in FIG. 3, the V-LED 20a generates purple light V having a center wavelength of 405 ± 10 nm and a wavelength range of 380 to 420 nm. The B-LED 20b generates blue light B having a center wavelength of 450 ± 10 nm and a wavelength range of 420 to 500 nm. The G-LED 20c generates green light G having a wavelength range of 480 to 600 nm. The R-LED 20d generates red light R having a center wavelength of 620 to 630 nm and a wavelength range of 600 to 650 nm.

The light source processor 21 controls the V-LED20a, B-LED20b, G-LED20c, and R-LED20d. By controlling each of the LEDs 20a to 20d independently, the light source processor 21 can emit purple light V, blue light B, green light G, or red light R by independently changing the amount of light. Further, the light source processor 21 emits white light having a light amount ratio of Vc: Bc: Gc: Rc among the purple light V, the blue light B, the green light G, and the red light R in the normal observation mode. , Each LED 20a to 20d is controlled. In addition, Vc, Bc, Gc, Rc> 0.

Further, in the special observation mode, the light source processor 21 has a light amount ratio of Vs: Bs: Gs: Rs with purple light V, blue light B, green light G, and red light R as short-wavelength narrow-band light. Each LED 20a to 20d is controlled so as to emit a special light. The light amount ratio Vs: Bs: Gs: Rs is different from the light amount ratio Vc: Bc: Gc: Rc used in the normal observation mode, and is appropriately determined according to the observation purpose. For example, when emphasizing superficial blood vessels, it is preferable to make Vs larger than other Bs, Gs, Rs, and when emphasizing mesopelagic blood vessels, Gs is more than other Vs, Gs, Rs. It is also preferable to increase the size.

Further, when the light source processor 21 automatically switches between the first illumination light and the second illumination light to emit light in the diagnosis support mode, the first illumination light is emitted in the first emission pattern and the second illumination is emitted. The light is emitted in the second emission pattern. Specifically, as shown in FIG. 4, the first light emission pattern is the first A light emission pattern in which the number of frames in the first lighting period is the same in each first lighting period, and as shown in FIG. It is preferable that the number of frames in the first lighting period is one of the first B emission patterns different in each first lighting period.

In the second emission pattern, as shown in FIG. 4, the number of frames in the second illumination period is the same in each second illumination period, and the emission spectrum of the second illumination light is in each second illumination period. As shown in FIG. 6, the number of frames in the second illumination period is the same in each of the second illumination periods, and the emission spectrum of the second illumination light is the same in the second illumination pattern. The second B emission pattern different in the two illumination periods, as shown in FIG. 7, the number of frames in the second illumination period is different in each second illumination period, and the emission spectrum of the second illumination light is different. The second C emission pattern which is the same in each second illumination period, as shown in FIG. 8, the number of frames in the second illumination period is different in each second illumination period, and the emission of the second illumination light. It is preferable that the spectrum is one of the second D emission patterns that are different in each second illumination period. The emission spectrum of the first illumination light may be the same or different in each first illumination period.

Here, the first lighting period is preferably longer than the second lighting period, and the first lighting period is preferably two frames or more. For example, in FIG. 4, when the first emission pattern is the first A pattern and the second emission pattern is the second A emission pattern (the number of frames in the second illumination period: the same, the emission spectrum of the second illumination light: the same). The first lighting period is set to 2 frames, and the second lighting period is set to 1 frame. Since the first illumination light is used to generate an observation image to be displayed on the display 18, it is preferable to obtain a bright image by illuminating the observation target with the first illumination light.

For example, the first illumination light is preferably white light. On the other hand, since the second illumination light is used for the recognition process, it is preferable to illuminate the observation target with the second illumination light to obtain an image suitable for the recognition process. For example, when the recognition process is performed based on the surface blood vessels, it is preferable that the second illumination light is purple light V. The first and second light emission patterns, which are switching patterns between the first lighting period and the second lighting period, will be described later because they are determined based on the image pickup control of the image pickup sensor 44 by the image pickup processor 45. Further, the frame refers to a unit of a period including at least a period from a specific timing to the completion of signal reading in the image pickup sensor 44.

In the present specification, the light intensity ratio includes the case where the ratio of at least one semiconductor light source is 0 (zero). Therefore, this includes the case where any one or more of the semiconductor light sources are not lit. For example, as in the case where the light amount ratio between purple light V, blue light B, green light G, and red light R is 1: 0: 0: 0, only one of the semiconductor light sources is turned on, and the other three are turned on. Even if it does not light up, it shall have a light intensity ratio.

As shown in FIG. 2, the light emitted by each of the LEDs 20a to 20d is incident on the light guide 25 via the optical path coupling portion 23 composed of a mirror, a lens, or the like. The light guide 25 is built in the endoscope 12 and a universal cord (a cord connecting the endoscope 12, the light source device 14 and the processor device 16). The light guide 25 propagates the light from the optical path coupling portion 23 to the tip portion 12d of the endoscope 12.

An illumination optical system 30a and an image pickup optical system 30b are provided at the tip end portion 12d of the endoscope 12. The illumination optical system 30a has an illumination lens 32, and the illumination light propagated by the light guide 25 is applied to the observation target through the illumination lens 32. The image pickup optical system 30b has an objective lens 42 and an image pickup sensor 44. The light from the observation target due to the irradiation of the illumination light is incident on the image pickup sensor 44 via the objective lens 42 and the zoom lens 43. As a result, an image to be observed is formed on the image pickup sensor 44. The zoom lens 43 is a lens for enlarging the observation target, and moves between the telephoto end and the wide end by operating the zoom operation unit 12h.

The image pickup sensor 44 is a primary color sensor, and is a B pixel (blue pixel) having a blue color filter, a G pixel (green pixel) having a green color filter, and an R pixel (red pixel) having a red color filter. It is equipped with three types of pixels. As shown in FIG. 9, the blue color filter BF mainly transmits light in the blue band, specifically, light in the wavelength band having a wavelength band of 380 to 560 nm. The transmittance of the blue color filter BF peaks in the vicinity of the wavelength of 460 to 470 nm. The green color filter transmits GF, mainly light in the green band, specifically, light in the wavelength band of 460 to 620 nm. The red color filter RF mainly transmits light in the red band, specifically, light in the wavelength band of 580 to 760 nm.

Further, the image sensor 44 is preferably a CCD (Charge-Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image pickup processor 45 controls the image pickup sensor 44. Specifically, the image signal is output from the image pickup sensor 44 by reading out the signal of the image pickup sensor 44 by the image pickup processor 45. In the normal observation mode, the image pickup processor 45 reads out the signal while the normal light is exposed to the image pickup sensor 44, so that the Bc image signal is output from the B pixel of the image pickup sensor 44 and the Gc image signal is output from the G pixel. Is output, and an Rc image signal is output from the R pixel. In the special observation mode, the Bs image signal is output from the B pixel of the image pickup sensor 44 and the Gs image signal is output from the G pixel by reading the signal from the image pickup processor 45 in a state where the special light is exposed to the image pickup sensor 44. Is output, and the Rs image signal is output from the R pixel.

In the diagnostic support mode, as shown in FIG. 10, the image pickup processor 45 reads out a signal from the image pickup sensor 44 in a state where the first illumination light is exposed to the image pickup sensor 44 during the first illumination period. 1 Output an image signal. The period for outputting the first image signal is defined as the first imaging period. The first image signal includes a B1 image signal output from the B pixel, a G1 image signal output from the G pixel, and an R1 image signal output from the R pixel. Further, the image pickup processor 45 outputs a second image signal from the image pickup sensor 44 by performing signal readout in a state where the image pickup sensor 44 is exposed to the second illumination light during the second illumination period. The period for outputting the second image signal is defined as the first imaging period. The second image signal includes a B2 image signal output from the B pixel, a G2 image signal output from the G pixel, and an R2 image signal output from the R pixel.

As shown in FIG. 2, the CDS / AGC (Correlated Double Sampling / Automatic Gain Control) circuit 46 performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the image pickup sensor 44. .. The image signal that has passed through the CDS / AGC circuit 46 is converted into a digital image signal by the A / D (Analog / Digital) converter 48. The digital image signal after A / D conversion is input to the processor device 16.

The processor device 16 stores programs related to various processes in a program memory (not shown). In the processor device 16, the central control unit 68 configured by the image control processor operates the program in the program memory to operate the image acquisition unit 50, the DSP (Digital Signal Processor) 52, and the noise reduction unit. The functions of the 54, the image processing switching unit 56, the image processing unit 58, and the display control unit 60 are realized. Further, with the realization of the functions of the image processing unit 58, the functions of the normal observation image generation unit 62, the special observation image generation unit 64, and the diagnosis support information processing unit 66 are realized. In the diagnostic support mode, the image control processor performs image processing based on the first image signal or the second image signal, and controls the display 18.

The image acquisition unit 50 acquires a color image input from the endoscope 12. The color image includes a blue signal (B image signal), a green signal (G image signal), and a red signal (R image signal) output from the B pixel, G pixel, and R pixel of the image pickup sensor 44. The acquired color image is transmitted to the DSP 52. The DSP 52 performs various signal processing such as defect correction processing, offset processing, gain correction processing, matrix processing, gamma conversion processing, demosaic processing, and YC conversion processing on the received color image. The noise reduction unit 54 performs noise reduction processing by, for example, a moving average method, a median filter method, or the like on a color image that has been demosaic processed by DSP 56. The color image with reduced noise is input to the image processing switching unit 56.

Depending on the set mode, the image processing switching unit 56 sets the destination of the image signal from the noise reduction unit 54 to the normal observation image generation unit 62, the special observation image generation unit 64, and the diagnosis support information processing unit 66. Switch to either. Specifically, when the normal observation mode is set, the image signal from the noise reduction unit 54 is input to the normal observation image generation unit 62. When the special observation mode is set, the image signal from the noise reduction unit 54 is input to the special observation image generation unit 64. When the diagnosis support mode is set, the image signal from the noise reduction unit 54 is input to the diagnosis support information processing unit 66.

The normal observation image generation unit 62 performs image processing for a normal observation image on the input Rc image signal, Gc image signal, and Bc image signal for one frame. Image processing for normal observation images includes 3 × 3 matrix processing, gradation conversion processing, color conversion processing such as 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. Is done. The Rc image signal, Gc image signal, and Bc image signal that have been subjected to image processing for a normal observation image are input to the display control unit 60 as normal observation images.

The special observation image generation unit 64 performs image processing for special observation images on the input Rs image signal, Gs image signal, and Bs image signal for one frame. Image processing for special observation images includes 3 × 3 matrix processing, gradation conversion processing, color conversion processing such as 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. Is done. The Rs image signal, Gs image signal, and Bs image signal that have been subjected to image processing for special observation images are input to the display control unit 60 as special observation images.

The diagnosis support information processing unit 66 performs the same image processing for normal observation images as described above on the input R1 image signal, G1 image signal, and B1 image signal for one frame. The R1 image signal, the G1 image signal, and the B1 image signal that have undergone image processing for a normal observation image signal are used as observation images. Further, the diagnosis support information processing unit 66 performs recognition processing on the input R2 image signal, G2 image signal, and B2 image signal for a specific frame. When the lesion candidate area is recognized by the recognition process, a diagnosis support information image displaying the diagnosis support information regarding the lesion candidate area is generated. The display control unit 60 displays the observation image or the diagnosis support information image on the display 18.

The display control unit 60 controls to display the image output from the image processing unit 58 on the display 18. Specifically, the display control unit 60 converts a normal observation image, a special observation image, an observation image, or a diagnosis support information image into a video signal that can be displayed in full color on the display 18. The converted video signal is input to the display 18. As a result, the display 18 displays a normal observation image, a special observation image, an observation image, or a diagnosis support information image.

In the diagnosis support mode, the display control unit 60 performs the following display control. When the first emission pattern is the first A emission pattern and the second emission pattern is the second B emission pattern (the number of frames in the second illumination period: the same, the emission spectrum of the second illumination light: different), the first illumination light When the observation target is illuminated for two frames and the second illumination light for one frame each during the emission of the first illumination light, as shown in FIG. 11, it is obtained by the illumination of the first illumination light. An observation image is obtained by performing image processing for a normal observation image on the first image signal to be obtained. When the first illumination light is emitted, the observation image is displayed on the display 18 as it is.

On the other hand, the presence or absence of the lesion candidate region DR is detected by performing recognition processing on the second image signal obtained by the emission of the second illumination light. The lesion candidate area is, for example, a lesion such as cancer, or a benign polyp. When the lesion candidate region is not detected, the observation image of the immediately preceding frame is displayed on the display 18 when the second illumination light is emitted. On the other hand, when the lesion candidate region DR is detected, the diagnosis support information is displayed on the observation image during the period when the lesion candidate region is detected when the first illumination light is emitted and when the second illumination light is emitted. The diagnostic support information image displaying the above is displayed on the display 18. The details of the display control of the diagnosis support information image will be described later.

The details of the display control of the diagnostic support information image will be described below. The diagnosis support image is generated in the diagnosis support information processing unit 66. As shown in FIG. 12, the diagnosis support information processing unit 66 includes a recognition processing unit 70, a motion acquisition unit 72, and a diagnosis support information image generation unit 74. The recognition processing unit 70 performs recognition processing based on the second image signal obtained when the second illumination light is emitted. The recognition process is preferably a process for recognizing and detecting a lesion candidate region. Specifically, it is preferable that the recognition processing unit 70 uses a learning model that has been machine-learned for the lesion candidate region, and outputs the presence or absence of the lesion candidate region when a second image signal is input to the learning model. ..

The motion acquisition unit 72 acquires the motion of the observation target in the image. The movement of the observation target is used to control the display or non-display of the first diagnosis support information, as will be described later. The movement of the observation target includes the movement of the observation target caused by the movement of the actual observation target such as the body movement, and the movement of the observation target caused by the movement of the tip portion 12d of the endoscope. It is preferable that the motion acquisition unit 72 calculates the motion of the observation target based on the first image signal or the second image signal of a plurality of frames obtained within a certain period of time. As a method of calculating the movement, the movement vector of the observation target is calculated from the first or second image signals of a plurality of frames obtained within a certain time, and the movement of the observation target is calculated based on the calculated movement vector. Is preferable.

When the lesion candidate area is recognized by the recognition process, the diagnosis support information image generation unit 74 generates a diagnosis support information image displaying the diagnosis support information regarding the lesion candidate area with respect to the observation image. The diagnostic support information includes first diagnostic support information that is displayed or hidden on the display 18 depending on the movement of the observation target, and second diagnostic support that maintains the display on the display 18 regardless of the movement of the observation target. Contains information.

Specifically, as shown in FIG. 13, the first diagnosis support information 76 is character information of a lesion candidate region such as a suspicion rate of a lesion, and is information that easily affects the user's visibility to the observed image. Is preferable. On the other hand, the second diagnosis support information 78 is preferably a bounding box or the like indicating the position information of the lesion candidate region DR, and is information that does not easily affect the user's visibility of the observed image.

The display control unit 60 controls the display or non-display of at least the first diagnosis support information among the diagnosis support information in the diagnosis support information image according to the movement of the observation target. Specifically, the display control unit 60 is different from the first display control when performing detailed observation (differentiation, etc.) of the lesion candidate area and the second display control when performing the lesion candidate area screening. Is preferable. The display or non-display of the second diagnosis support information may also be controlled.

As shown in FIG. 14, the first display control displays the first diagnosis support information 76 when the movement of the observation target is less than the first threshold value, and when the movement of the observation target is equal to or more than the first threshold value. Hides the first diagnosis support information 76. In detailed observation such as discrimination, it is preferable that the tip portion 12d of the endoscope having the image pickup sensor 44 is temporarily stopped and the movement of the observation target is relatively small. Therefore, when the movement of the observation target suitable for detailed observation is small, such as less than the first threshold value, the first diagnosis support information 76 is displayed. On the other hand, when the movement of the observation target becomes large such as becoming larger than the first threshold value and becomes unsuitable for detailed observation, the display of the first diagnosis support information 76 may hinder the visibility, so that the first diagnosis Hide support information 76. Even when the first diagnosis support information 76 is hidden in the first display control, it is preferable to maintain the display of the second diagnosis support information.

As shown in FIG. 15, the second display control displays the first diagnosis support information 76 when the movement of the observation target is equal to or higher than the second threshold value, and when the movement of the observation target is less than the second threshold value. In addition, the first diagnosis support information 76 is hidden. In the screening, since the tip portion 12d of the endoscope is moved at a constant speed or higher, the movement of the observation target is relatively large. Therefore, in order to facilitate the detection of the lesion candidate region at the time of screening, the first diagnosis support information 76 is displayed when the movement of the observation target is large such as the second threshold value or more. On the other hand, when the movement of the observation target becomes smaller than the second threshold value and becomes unsuitable for screening, the display of the first diagnosis support information 76 may hinder the visibility, so that the first diagnosis support is supported. Hide information 76. Even when the first diagnosis support information 76 is hidden in the second display control, it is preferable to maintain the display of the second diagnosis support information.

Note that switching between the first display control and the second display control may be performed manually using the user interface 19. Further, the switching between the first display control and the second display control may be manually performed by the display control switching unit 75 (see FIG. 12). As shown in FIG. 16, when the movement of the observation target is less than the switching threshold value, the display control switching unit 75 determines that the movement of the observation target is less detailed observation, and switches to the first display control. On the other hand, when the movement of the observation target is equal to or higher than the switching threshold value, it is determined that the movement of the observation target is large in the screening, and the screen is switched to the second display control. The display control unit 60 controls the display of the diagnosis support information according to the automatically switched first display control or second display control.

It should be noted that the emission spectrum of the second illumination light is different in each of the second illumination periods, as in the case where the second emission pattern of the second illumination light is the second B emission pattern or the second D emission pattern. When a plurality of lesion candidate images are recognized by the recognition process based on the second image signal corresponding to each second illumination light, the display control unit 60 maintains the display regardless of the movement of the observation target. 2 It is preferable that the display of the diagnostic support information is different depending on the emission spectrum of the second illumination light.

For example, as shown in FIG. 17, in the diagnosis support image, the lesion candidate region DRA detected by the second illumination light of the emission spectrum A and the second illumination light of the emission spectrum B different from the emission spectrum A were detected. When the lesion candidate region DRB exists, the first diagnosis support information 76a and the second diagnosis support information 78a regarding the lesion candidate region DRA, and the first diagnosis support information 76b and the second diagnosis support information 78b regarding the lesion candidate region DRB. Is displayed.

In this case, regarding the second

diagnosis support information

78a and 78b that maintain the display regardless of the movement of the observation target, the second diagnosis support information 78b is displayed in the "blue" banding box, and the second diagnosis support information 78b is ". Display in the "green" bounding box. That is, "blue" indicates that it was detected by the second illumination light of the emission spectrum A, and "green" indicates that it was detected by the second illumination light of the emission spectrum B. This makes it possible to visually grasp which lesion candidate region DRA or DRB is detected by the second illumination light of which emission spectrum.

Next, a series of flow of the diagnosis support mode will be described with reference to the flowchart of FIG. When the mode switching SW12f is operated to switch to the diagnostic support mode, the first illumination light and the second illumination light having different emission spectra are automatically switched and emitted. The image pickup sensor 44 takes an image of the observation target illuminated by the first illumination light and outputs the first image signal. Further, the observation target illuminated by the second illumination light is imaged, and the second image signal is output.

The recognition processing unit 70 performs recognition processing on the second image signal. The motion acquisition unit 72 acquires the motion of the observation target in the image. When the diagnosis support information image generation unit 74 recognizes the lesion candidate region by the recognition process, the diagnosis support information image generation unit 74 generates a diagnosis support information image that displays the diagnosis support information regarding the lesion candidate region with respect to the observation image based on the first image signal. do. The display control unit 60 controls the display or non-display of at least the first diagnosis support information among the diagnosis support information according to the movement of the observation target in the diagnosis support information image. The above series of processes is repeated while the diagnosis support mode continues.

In the above embodiment, the light source processor 21, the imaging processor 45, the image acquisition unit 50, the DPS 52, the noise reduction unit 54, the image processing switching unit 56, the normal observation image generation unit 62 included in the image processing unit 58, and the special observation image. A processing unit that executes various processes such as a generation unit 64, a diagnosis support information processing unit 66, a central control unit 68, a recognition processing unit 70, a motion acquisition unit 72, a diagnosis support information image generation unit 74, and a display control switching unit 75. The hardware structure of the processing unit) is various processors as shown below. For various processors, the circuit configuration is changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), etc., which are general-purpose processors that execute software (programs) and function as various processing units. It includes a programmable logic device (PLD), which is a possible processor, a dedicated electric circuit, which is a processor having a circuit configuration specially designed for executing various processes, and the like.

One processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). May be done. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which this processor functions as a plurality of processing units. Second, as typified by System On Chip (SoC), there is a form that uses a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Furthermore, the hardware-like structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined. The hardware structure of the storage unit is a storage device such as an HDD (hard disk drive) or SSD (solid state drive).

10 Endoscope system 12 Endoscope

12a Insertion part

12b Operation part 12c Curved part

12d Tip part

12e Angle knob 12f Mode changeover switch 12g Still image acquisition instruction part 12h Zoom operation part 14 Light source device 16 Processor device 18 Display 19 User interface 20 Light source 20a V-LED
20b B-LED
20c G-LED
20d R-LED
21 Light source processor 23 Optical path coupling part 25 Light guide 30a Illumination optical system 30b Imaging optical system 32 Illumination lens 42 Objective lens 43 Zoom lens 44 Imaging sensor 45 Imaging processor 46 CDS / AGC circuit 48 A / D converter 50 Image acquisition unit 52 DSP
54 Noise reduction unit 56 Image processing switching unit 58 Image processing unit 60 Display control unit 62 Normal observation image generation unit 64 Special observation image generation unit 66 Diagnosis support information processing unit 68 Central control unit 69 Still image storage memory 70 Recognition processing unit 72 Motion acquisition unit 74 Diagnosis support information image generation unit 75 Display

control switching unit

76, 76a, 76b First

diagnosis support information

78, 78a, 78b Second diagnosis support information DR, DRA, DRB Disease candidate area

Claims

A light source unit that emits first illumination light and second illumination light having different emission spectra from each other.
When the first illumination period for emitting the first illumination light and the second illumination period for emitting the second illumination light are automatically switched, the first illumination light is emitted in the first emission pattern, and the first emission pattern is used. 2 A light source processor that emits illumination light in the second emission pattern,
An image sensor that outputs a first image signal obtained by imaging an observation object illuminated by the first illumination light and a second image signal by imaging the observation object illuminated by the second illumination light. ,
Equipped with an image control processor
The image control processor is
The second image signal is recognized and processed.
Acquire the movement of the observation target in the image and
When the lesion candidate region is recognized by the recognition process, a diagnostic support information image that displays diagnostic support information regarding the lesion candidate region is generated for the observation image based on the first image signal.
An endoscope system that controls the display or non-display of at least the first diagnosis support information among the diagnosis support information according to the movement of the observation target in the diagnosis support information image.
The image control processor displays the diagnosis support information when the movement of the observation target is less than the first threshold value, and the diagnosis when the movement of the observation target is equal to or more than the first threshold value. The endoscope system according to claim 1, wherein the first display control for hiding the support information is performed.
The image control processor hides the diagnostic support information when the movement of the observation target is less than the second threshold value, and when the movement of the observation target is equal to or more than the second threshold value, the diagnosis support information is hidden. The endoscope system according to claim 1 or 2, wherein the second display control for displaying diagnostic support information is performed.
The image control processor displays the diagnosis support information when the movement of the observation target is less than the first threshold value, and the diagnosis when the movement of the observation target is equal to or more than the first threshold value. The first display control that hides the support information, or when the movement of the observation target is less than the second threshold value, the diagnosis support information is hidden and the movement of the observation target is equal to or higher than the second threshold value. In the case where it is possible to perform the second display control for displaying the diagnosis support information.
The endoscope system according to claim 1, further comprising a user interface for manually switching between the first display control and the second display control.
The image control processor displays the diagnosis support information when the movement of the observation target is less than the first threshold value, and the diagnosis when the movement of the observation target is equal to or more than the first threshold value. The first display control that hides the support information, or when the movement of the observation target is less than the second threshold value, the diagnosis support information is hidden and the movement of the observation target is equal to or higher than the second threshold value. In the case where it is possible to perform the second display control for displaying the diagnosis support information.
The endoscope system according to claim 1, wherein the image control processor automatically switches between the first display control and the second display control.
The image control processor switches to the first display control when the movement of the observation target is less than the switching threshold, and the first display control when the movement of the observation target is equal to or more than the switching threshold. 2. The endoscope system according to claim 5, which switches to display control.
The emission spectra of the second illumination light are different in each of the second illumination periods, and a plurality of lesion candidate images are recognized by the recognition process based on the second image signal corresponding to each second illumination light. In this case, the display of the second diagnostic support information that maintains the display regardless of the movement of the observation target among the diagnostic support information is any one of claims 1 to 6 that differs depending on the emission spectrum of the second illumination light. Endoscope system described in section.
In the first light emission pattern, the number of frames in the first lighting period is the same in each of the first lighting periods, and the number of frames in the first lighting period is the same in each first lighting period. The endoscope system according to any one of claims 1 to 7, which is one of different first B emission patterns.
In the second emission pattern, the number of frames in the second illumination period is the same in each of the second illumination periods, and the emission spectrum of the second illumination light is the same in each of the second illumination periods. A second A emission pattern,
A second B emission pattern, in which the number of frames in the second illumination period is the same in each of the second illumination periods, and the emission spectrum of the second illumination light is different in each of the second illumination periods.
A second C emission pattern, wherein the number of frames in the second illumination period is different in each of the second illumination periods, and the emission spectrum of the second illumination light is the same in each of the second illumination periods.
The number of frames in the second illumination period is different in each of the second illumination periods, and the emission spectrum of the second illumination light is different in each of the second illumination periods. The endoscope system according to any one of claims 1 to 8, which is any one of the patterns.
The light source unit that emits the first illumination light and the second illumination light having different emission spectra, the first illumination period that emits the first illumination light, and the second illumination period that emits the second illumination light are automatically set. In the case of switching, the light source processor that emits the first illumination light in the first emission pattern and emits the second illumination light in the second emission pattern, and the observation target illuminated by the first illumination light are imaged. Operation of an endoscope system including a first image signal obtained, an image sensor that captures the observation target illuminated by the second illumination light and outputs a second image signal, and an image control processor. In the method
The image control processor is
The second image signal is recognized and processed.
Acquire the movement of the observation target in the image and
When the lesion candidate region is recognized by the recognition process, a diagnostic support information image that displays diagnostic support information regarding the lesion candidate region is generated for the observation image based on the first image signal.
A method of operating an endoscope system that controls the display or non-display of at least the first diagnosis support information among the diagnosis support information according to the movement of the observation target in the diagnosis support information image.