WO2022059233A1

WO2022059233A1 - Image processing device, endoscope system, operation method for image processing device, and program for image processing device

Info

Publication number: WO2022059233A1
Application number: PCT/JP2021/010864
Authority: WO
Inventors: 裕哉木村
Original assignee: 富士フイルム株式会社
Priority date: 2020-09-15
Filing date: 2021-03-17
Publication date: 2022-03-24
Also published as: JP2023178526A

Abstract

Provided are: an image processing device that efficiently records an endoscopic image when image recognition processes are performed on the basis of a plurality of types of endoscopic images in parallel; an endoscope system; an operation method for an image processing device; and a program for an image processing device.　An image processing device (16) is provided with an image processor. The image processor acquires a plurality of types of recognition target images based on an endoscope image, performs image recognition processes for the plurality of types of recognition target images on the respective types of recognition target images in parallel, and controls the operation of recording a moving image of at least one type of recognition target image among the plurality of types of recognition target images on the basis of the results of the recognition processes of all the types of recognition target images acquired.

Description

Image processing device, endoscope system, operation method of image processing device, and program for image processing device

The present invention relates to an image processing device that performs image recognition processing, an endoscope system, an operation method of the image processing device, and a program for the image processing device.

In the medical field, examinations or diagnoses using an endoscope system equipped with a light source device, an endoscope, and a processor device are widely performed. When performing diagnosis or examination using an endoscope system, an image obtained by photographing an observation target with an endoscope for use as evidence for preparing for a medical accident or for presentation at a conference (hereinafter, Recording is performed to record an endoscopic image) as a moving image.

When recording an endoscopic image, control such as start or stop of recording is usually performed by manual operation of a recording device or the like. For example, the operation of starting recording is manually performed before the inspection is started, and the operation of stopping recording is manually performed after the inspection is completed. In addition, during endoscopy, the user may manually start and stop recording in a scene where he / she wants to keep a moving image.

Regarding the control of the recording of endoscopic images, in order to improve the operability of the user, a moving image recording operation is performed according to the type of image processing of the image signal obtained by photographing the inside of the body cavity of the subject with an endoscope. An image processing device for an endoscope that controls the above is known (Patent Document 1).

In addition, in order to automatically control the start and stop of recording a moving image of an endoscope image, the image signal is red when the image signal obtained by photographing the inside of the body cavity of the subject with an endoscope is recorded. There is known an endoscopic image recording device that controls the start and stop of recording by determining whether or not the image contains a threshold value or more (Patent Document 2).

In addition, in order to reduce oversight of lesions in endoscopic observation, image processing is performed in which observation images are sequentially recorded and displayed as a plurality of recorded images during the period from the start of detection of the lesion candidate region to the interruption. The device is known (Patent Document 3).

Japanese Unexamined Patent Publication No. 2006-271871 Japanese Unexamined Patent Publication No. 2010-51399 International Publication No. 2017/216922

In the control of operations such as start or stop in recording endoscopic images, there is a risk of forgetting to operate in the case of a manual operation in which recording is started before the inspection starts and recording is stopped after the inspection is completed. There is. Further, since the moving image recorded in this way is a photograph of the entire endoscopy, there is a possibility that it becomes difficult to reproduce a desired scene due to a large number of unnecessary parts depending on the application. Then, the storage capacity is squeezed, and the number of inspections recorded in the storage is reduced. Therefore, there is a risk of hindering the usability of the user regarding recording.

In addition, conventional methods such as a method of controlling the recording operation by detecting the type of image processing, the red color contained in the image, or the lesion candidate area of the image, etc., trigger the recording operation according to each purpose of the recording. Has been decided. Therefore, it may not be suitable for recording for other purposes.

INDUSTRIAL APPLICABILITY The present invention operates an image processing device, an endoscope system, and an image processing device that efficiently record an endoscope image when image recognition processing is performed in parallel based on a plurality of types of endoscope images. It is an object of the present invention to provide a method and a program for an image processing apparatus.

The present invention is an image processing apparatus, and performs image recognition processing based on an image obtained by photographing an observation target with an endoscope. The image processing device includes an image processor. The image processor acquires a plurality of types of recognition target images based on the image, controls the continuous display of at least one type of recognition target image among the plurality of types of recognition target images on the display, and controls the display of the plurality of types of recognition targets. Image recognition processing for images is performed in parallel for each type of recognition target image, recognition processing results obtained by image recognition processing are acquired, and multiple types of recognition processing results are obtained based on the recognition processing results of all the acquired types of recognition target images. It controls the recording operation when recording a moving image of at least one type of recognition target image among the recognition target images.

The recognition processing result includes information on whether or not the recognition target image satisfies a preset condition, and the image processor includes information that one or more of all types of recognition processing results satisfy the condition. If so, it is preferable to start or continue recording.

It is preferable that the image processor contains information that all the recognition processing results of all types do not satisfy the conditions, and the recording is stopped when the recording is continued.

When recording a moving image, the image processor preferably attaches information regarding the corresponding conditions for starting, continuing, or stopping the recording to the moving image.

The image processor acquires in advance the correspondence information in which the observation target satisfying the condition and the recognition target image obtained by shooting the observation target satisfying the condition are associated with each other, and based on the correspondence information, the newly acquired recognition target image. It is preferable to perform image recognition processing on the image.

It is preferable that the image processor acquires correspondence information for each type of recognition target image, and performs image recognition processing for the newly acquired recognition target image based on the corresponding type of correspondence information.

It is preferable that the condition is that an object other than a specific part or a living body is detected by the image processor, or that it is determined to include a region in a specific state.

The plurality of types of recognition target images include at least a first recognition target image and a second recognition target image of different types from each other, and the image processor performs a first image recognition process on the first recognition target image. For the second image to be recognized, it is preferable to perform a second image recognition process different from the first image recognition process.

It is preferable that the first image recognition process is performed with respect to the detection of a specific part or an object other than a living body in the first recognition target image.

It is preferable that the second image recognition process is performed with respect to the determination that the region of the specific state in the second recognition target image is included.

A recognition target image is generated by performing enhancement processing on an image, and the image processor distinguishes the type of recognition target image according to the presence or absence or type of enhancement processing, and each of the distinguished recognition target images is one type of recognition target image. It is preferable to obtain as.

The types of enhancement processing are color expansion processing and / or structure enhancement processing, and the image processor acquires recognition target images generated by performing different types of enhancement processing as one type of recognition target image. Is preferable.

Further, the endoscope system of the present invention includes an image processing device and a light source unit that emits illumination light to irradiate the observation target.

It is preferable that the image processor acquires an image obtained by photographing an observation target illuminated by each of a plurality of illumination lights having different spectral spectra from each other emitted by the light source unit as one type of recognition target image.

It is preferable that the image processor acquires an image obtained by photographing an observation target illuminated by white illumination light emitted by a light source unit as one kind of recognition target image.

It is preferable that the image processor acquires an image obtained by photographing an observation target illuminated by illumination light including a narrow band light of a preset wavelength band emitted by a light source unit as one kind of recognition target image.

It is preferable that the light source unit repeatedly emits each of a plurality of illumination lights having different spectral spectra in a preset order.

The light source unit emits the first illumination light and the second illumination light having different spectral spectra from each other, emits the first illumination light by the first emission pattern during the first illumination period, and emits the second illumination light by the first emission pattern, and the second illumination light during the second illumination period. A light source processor that emits light according to the second emission pattern and switches between the first illumination light and the second illumination light, a first image signal obtained by photographing an observation target illuminated by the first illumination light, and a first image signal. The image sensor includes an image sensor that outputs a second image signal obtained by photographing the observation target illuminated by the illumination light, and the image processor is the first with respect to the first recognition target image based on the first image signal. The image recognition process is performed, the second image recognition process is performed on the second recognition target image based on the second image signal, the first image recognition result by the first image recognition process and the second image by the second image recognition process. It is preferable to control the recording operation based on the recognition result.

Further, the present invention is a method of operating an image processing device, in which image recognition processing is performed based on an image obtained by photographing an observation target with an endoscope, and a plurality of types of recognition target images based on the image are performed. An image acquisition step for acquiring images, a display control step for controlling the continuous display of at least one type of recognition target image among a plurality of types of recognition target images, and an image recognition process for a plurality of types of recognition target images. , An image recognition processing step performed in parallel for each type of recognition target image, a recognition processing result acquisition step for acquiring recognition processing results obtained by image recognition processing for each type of recognition target image, and a plurality of types of recognition targets. When recording a moving image of at least one type of recognition target image among the images, the recording control step for controlling the recording operation based on the recognition processing results of all the acquired recognition target images is provided.

Further, the present invention is a program for an image processing device, which is installed in an image processing device that performs image recognition processing based on an image obtained by photographing an observation target with an endoscope, and is installed on a computer for an image. An image acquisition function that acquires multiple types of recognition target images based on the above, a display control function that controls the continuous display of at least one type of recognition target image among multiple types of recognition target images, and a plurality of types. Image recognition processing function that performs image recognition processing for the recognition target image in parallel for each type of recognition target image, and recognition processing result acquisition that acquires the recognition processing result obtained by the image recognition processing for each type of recognition target image. A function and a recording control function that controls the recording operation based on the recognition processing results of all the acquired recognition target images when recording a moving image of at least one type of recognition target image among a plurality of types of recognition target images. It is a program for an image processing device to realize.

According to the present invention, when image recognition processing is performed in parallel based on a plurality of types of endoscopic images, the endoscopic images can be efficiently recorded.

It is explanatory drawing explaining the structure of an endoscope system. It is a block diagram which shows the function of an endoscope system. It is a graph which shows the spectral spectrum of purple light V, blue light B, green light G, and red light R. It is explanatory drawing explaining the 1A light emission pattern and the 2nd A light emission pattern. It is explanatory drawing explaining 1B light emission pattern. It is explanatory drawing explaining the 2nd B light emission pattern. It is explanatory drawing explaining the 2nd C light emission pattern. It is explanatory drawing explaining the 2D light emission pattern. It is a block diagram which shows the function of an image processing part. It is a block diagram which shows the function of the diagnosis support image processing part. It is a block diagram which shows the function of the recognition target image acquisition part. It is a graph which shows the spectroscopic spectrum SP1 for the 2nd illumination light. It is a graph which shows the spectroscopic spectrum SP2 for the 2nd illumination light. It is a graph which shows the spectroscopic spectrum SP3 for the 2nd illumination light. It is a block diagram which shows the function of the 4th recognition target image generation part. It is a graph which shows the table for oxygen saturation calculation. It is a graph which shows the spectrum SP4 for the 2nd illumination light. It is a block diagram which shows the function of the 5th recognition target image generation part. It is explanatory drawing explaining the color difference expansion process. It is a block diagram which shows the function of an image recognition processing part. It is explanatory drawing explaining the function of the correspondence information acquisition part. It is explanatory drawing explaining the acquisition of the recognition target image. It is explanatory drawing explaining the recognition process result. It is explanatory drawing explaining the recognition process result and the recording control. It is explanatory drawing explaining the recognition process result by the acceptance determination and the recording control. It is explanatory drawing explaining the recognition process result by detection and the acceptance determination, and recording control. It is explanatory drawing explaining the recognition process result by detection and a plurality of match determinations, and recording control. It is explanatory drawing explaining the recording file. It is a flowchart which shows a series flow of a diagnosis support mode. It is explanatory drawing which shows the diagnosis support apparatus. It is explanatory drawing which shows the medical work support apparatus.

As shown in FIG. 1, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 16, a display 18, and a keyboard 19. The endoscope 12 photographs the observation target. The light source device 14 emits illumination light to irradiate the observation target. The processor device 16 controls the system of the endoscope system 10. The display 18 is a display unit that displays an observation image or the like based on an endoscopic image. The keyboard 19 is an input device for inputting settings to the processor device 16 and the like.

The endoscope system 10 includes three modes as observation modes: a normal observation mode, a special observation mode, and a diagnosis support mode. In the normal observation mode, a normal observation image having a natural color is displayed on the display 18 as an observation image by irradiating the observation target with normal light such as white light and taking a picture. In the special observation mode, a special image emphasizing a specific structure or the like is displayed on the display 18 as an observation image by illuminating the observation target with special light having a wavelength band or a spectral spectrum different from that of normal light. In the diagnosis support mode, image recognition processing is performed for each type of recognition target image for a plurality of types of recognition target images based on the endoscopic image. The recognition target image is an image based on the endoscopic image, and is an image to be subject to image recognition processing. Then, the display 18 continuously displays at least one of the plurality of types of recognition target images as an observation image. The results of a plurality of recognition processes performed by the image recognition process performed for each type of the image to be recognized are used for controlling the recording operation. The type of the image to be recognized depends on the spectral spectrum of the illumination light when the observation target is photographed, and / or the method of image processing for generating the image to be recognized (hereinafter referred to as image processing for generating the image to be recognized). Distinguish by. The details of the types of recognition target images will be described later.

The endoscope 12 includes an insertion portion 12a to be inserted into a subject having an observation target, an operation portion 12b provided at the base end portion of the insertion portion 12a, and a bending portion 12c provided on the distal end side of the insertion portion 12a. It has a tip portion 12d. By operating the angle knob 12e of the operation portion 12b, the curved portion 12c is curved. As a result, the tip portion 12d faces in a desired direction. Further, in addition to the angle knob 12e, the operation unit 12b is provided with a treatment tool insertion port (not shown), a scope button No. 1 12f, a scope button No. 2 12g, and a zoom operation unit 12h. The treatment tool insertion port is an entrance for inserting a treatment tool such as a biopsy forceps, a snare, or an electric knife. The treatment tool inserted into the treatment tool insertion port protrudes from the tip portion 12d. The scope button No. 1 12f is a freeze button and is used for an operation of acquiring a still image. The scope button No. 2 12g is used for the operation of switching the observation mode. Various operations can be assigned to the scope button. By operating the zoom operation unit 12h, the observation target can be enlarged or reduced for shooting.

As shown in FIG. 2, the light source device 14 includes a light source unit 20 including a light source that emits illumination light, and a light source processor 22 that controls the operation of the light source unit 20. The light source unit 20 emits illumination light that illuminates the observation target. The illumination light includes light emission such as excitation light used to emit the illumination light. The light source unit 20 includes, for example, a light source of a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, and is used to emit at least white illumination light (hereinafter referred to as white light) or white light. It emits an excitation light. The white color includes so-called pseudo-white color, which is substantially equivalent to white color in the imaging of the observation target using the endoscope 12.

The light source unit 20 includes, if necessary, a phosphor that emits light when irradiated with excitation light, an illumination light, an optical filter that adjusts the wavelength band, spectral spectrum, light amount, etc. of the excitation light. In addition, the light source unit 20 can emit illumination light composed of at least narrow band light (hereinafter referred to as narrow band light). The “narrow band” means a substantially single wavelength band in relation to the characteristics of the observation target and / or the spectral characteristics of the color filter of the image sensor 45. For example, when the wavelength band is, for example, about ± 20 nm or less (preferably about ± 10 nm or less), this light is a narrow band.

Further, the light source unit 20 can emit a plurality of illumination lights having different spectral spectra from each other. The plurality of illumination lights may include narrow band light. Further, the light source unit 20 may emit light having a specific wavelength band or spectral spectrum necessary for capturing an image used for calculating biological information such as oxygen saturation of hemoglobin contained in the observation target, for example. can.

In the present embodiment, the light source unit 20 has four color LEDs of V-LED20a, B-LED20b, G-LED20c, and R-LED20d. As shown in FIG. 3, the V-LED 20a emits purple light V having a center wavelength of 405 nm and a wavelength band of 380 to 420 nm. The B-LED 20b emits blue light B having a center wavelength of 460 nm and a wavelength band of 420 to 500 nm. The G-LED 20c emits green light G having a wavelength band of 480 to 600 nm. The R-LED 20d emits red light R having a center wavelength of 620 to 630 nm and a wavelength band of 600 to 650 nm. The center wavelengths of the V-LED 20a and the B-LED 20b have a width of about ± 20 nm, preferably about ± 5 nm to about ± 10 nm. The purple light V is a short-wavelength light used in the special observation mode or the diagnostic support mode to emphasize and display the density of superficial blood vessels, intramucosal hemorrhage, extramucosal hemorrhage, etc., and has a central wavelength or It is preferable to include 410 nm in the peak wavelength. Further, the purple light V and / or the blue light B is preferably narrow band light.

The light source processor 22 controls the timing of turning on, off, or blocking each light source constituting the light source unit 20, the light intensity, the amount of light emitted, and the like. As a result, the light source unit 20 can emit a plurality of types of illumination light having different spectral spectra for a preset period and emission amount. In the present embodiment, the light source processor 22 turns on and off the V-LED20a, B-LED20b, G-LED20c, and R-LED20d, the light intensity or the amount of light emitted at the time of lighting, the insertion and removal of the optical filter, and the like. It is controlled by inputting an independent control signal to each. The light source processor 22 independently controls each of the LEDs 20a to 20d to independently change the light intensity or the amount of light per unit time for purple light V, blue light B, green light G, or red light R. It can emit light. Therefore, the light source processor 22 can emit a plurality of illumination lights having different spectral spectra, for example, white illumination light, a plurality of types of illumination lights having different spectral spectra, or at least an illumination light composed of narrow band light. Etc. are emitted.

In the present embodiment, the light source processor 22 produces white light having a light intensity ratio of Vc: Bc: Gc: Rc between purple light V, blue light B, green light G, and red light R in the normal observation mode. Each LED 20a to 20d is controlled so as to emit light. It should be noted that each of Vc, Bc, Gc, or Rc is larger than 0 (zero) and is not 0.

Further, in the present embodiment, the light source processor 22 has a light intensity ratio of Vs: to the purple light V, the blue light B, the green light G, and the red light R as short-wavelength narrow-band light in the special observation mode. Each LED 20a to 20d is controlled so as to emit special light having Bs: Gs: Rs. The light intensity ratio Vs: Bs: Gs: Rs is different from the light intensity ratio Vc: Bc: Gc: Rc used in the normal observation mode, and is appropriately determined according to the observation purpose. Therefore, the light source unit 20 can emit a plurality of special lights having different spectral spectra from each other under the control of the light source processor 22. For example, when emphasizing superficial blood vessels, Vs is preferably larger than other Bs, Gs, and Rs, and when emphasizing mesopelagic blood vessels, Gs is referred to as other Vs, Gs, and Rs. It is preferably larger than Rs.

In the present specification, the light intensity ratio includes the case where the ratio of at least one semiconductor light source is 0 (zero) except for Vc: Bc: Gc: Rc. Therefore, this includes the case where any one or more of the semiconductor light sources are not lit. For example, as in the case where the light intensity ratio between purple light V, blue light B, green light G, and red light R is 1: 0: 0: 0, only one of the semiconductor light sources is turned on, and the other three. Even if one does not light up, it shall have a light intensity ratio.

Further, in the present embodiment, the light source processor 22 automatically switches a plurality of illumination lights having different spectral spectra to a specific pattern in order to acquire a plurality of types of recognition target images in the diagnostic support mode. To emit. Then, each of the plurality of illumination lights having different spectral spectra is repeatedly emitted in a preset order. Specifically, in the case of automatically switching between the first illumination light and the second illumination light in order to acquire two types of recognition target images in the diagnosis support mode, the first illumination light is the first illumination. It emits in the first emission pattern during the period, and emits the second illumination light in the second emission pattern in the second illumination period. The first illumination light and the second illumination light are illumination lights having different spectral spectra. For example, the first illumination light is white light. On the other hand, since the second illumination light is used for the recognition process, it is preferable that the second illumination light is an illumination light that can obtain an image suitable for a specific recognition process by illuminating the observation target with the second illumination light. For example, when the recognition process is performed based on the surface blood vessels, it is preferable that the second illumination light is purple light V.

The first light emission pattern is the light emission order of the first illumination light, the second light emission pattern is the light emission order of the second illumination light, and the element constituting each pattern is a frame which is a unit of photography. The frame means a period including at least a period from a specific timing in the image sensor 45 to the completion of signal reading. One shooting and image acquisition are performed in one frame. The first illumination light and the second illumination light emit either one, and do not emit at the same time. One light emission cycle is composed of at least one first light emission pattern and a second light emission pattern, respectively, and the first light emission pattern and the second light emission pattern are combined to form a light emission cycle. Illumination is performed by repeating the light emission cycle. Details such as the number of frames constituting each of the first light emission pattern and the second light emission pattern, the type of illumination light, and the like are set in advance. The type of illumination light is distinguished by the spectral spectrum of the illumination light. Therefore, the illumination light having a different spectral spectrum is a different kind of illumination light.

Specifically, the first light emission pattern is preferably the first A light emission pattern or the first B light emission pattern. As shown in FIG. 4, in the first A emission pattern, the number of frame FLs in the first illumination light L1 in the first illumination period P1 is determined in the emission cycle Q1 and is the same. As shown in FIG. 5, in the first B emission pattern, the number of frame FLs in the first illumination period P1 is different in each first illumination period P1 in the emission cycle Q2. In the first A emission pattern and the first B emission pattern, the first illumination light L1 has the same spectral spectrum and is white light.

The second light emission pattern is preferably a second A light emission pattern, a second B light emission pattern, a second C light emission pattern, or a second D light emission pattern. As shown in FIG. 4, in the second A emission pattern, the number of frame FLs in the second illumination period is determined in the emission cycle Q1 and is the same, and the spectral spectrum of the second illumination light L2 is the respective second illumination pattern. The second illumination light L2a is the same in the second illumination period P2. The second illumination light L2 may include illumination lights having different spectral spectra, and these are described as the second illumination light L2a and the second illumination light L2b to distinguish them, and when they are described as the second illumination light L2. , These are collectively referred to. As shown in FIG. 5, the second illumination light L2 is emitted in the second A emission pattern even in the emission cycle Q2.

As shown in FIG. 6, in the second B emission pattern, in the emission cycle Q3, the number of frames FL of the second illumination period P2 is the same in each second illumination period P2, and the spectral spectrum of the second illumination light L2 is the same. Is the second illumination light L2a or the second illumination light L2b in each of the second illumination periods P2, and is different. As shown in FIG. 7, in the second C emission pattern, in the emission cycle Q4, the number of frames FL of the second illumination period P2 is different in each second illumination period P2, and the spectral spectra of the second illumination light L2 are different from each other. The second illumination light L2a is the same in the second illumination period P2 of the above. As shown in FIG. 8, in the second D emission pattern, in the emission cycle Q5, the number of frames FL of the second illumination period P2 is different in each second illumination period P2, and the spectral spectra of the second illumination light P2 are different from each other. The second illumination light 2La or the second illumination light 2Lb in the second illumination period P2 of the above is different.

As described above, in the present embodiment, in the diagnostic support mode, the light source processor 22 repeats the light emission cycle configured by combining these first light emission patterns and the second light emission patterns. As shown in FIG. 4, the light emission cycle Q1 includes a first A light emission pattern and a second A light emission pattern. As shown in FIG. 5, the light emission cycle Q2 includes a first B light emission pattern and a second A light emission pattern. As shown in FIG. 6, the light emission cycle Q3 includes a first A light emission pattern and a second B light emission pattern. As shown in FIG. 7, the light emission cycle Q4 includes a first A light emission pattern and a second C light emission pattern. As shown in FIG. 8, the light emission cycle Q5 includes a first A light emission pattern and a second D light emission pattern. In the first emission pattern, the spectral spectrum of the first illumination light L1 may be different in each first illumination period P1.

Further, in the diagnosis support mode, the light source processor 22 may change the first light emission pattern or the second light emission pattern based on the recognition processing result described later. Changing the emission pattern includes changing the type of illumination light. Specifically, for example, the second light emission pattern is changed from the second A pattern to the second B light emission pattern based on the recognition processing result, or the second A light emission pattern using the second illumination light L2a is changed to the second light emission pattern. Switching may be performed such as changing to the second A emission pattern using the illumination light L2b.

Here, the first lighting period P1 is preferably longer than the second lighting period P2, and the first lighting period P1 is preferably two frames or more. For example, in FIG. 4, in the light emission cycle Q1 in which the first light emission pattern is the first A pattern and the second light emission pattern is the second A light emission pattern, the first illumination period P1 is set to 2 frames and the second illumination period P2 is set to 1 frame. It is supposed to be. Since the first illumination light P1 is used to generate an observation image to be displayed on the display 18, it is preferable to obtain a bright observation image by illuminating the observation target with the first illumination light P1.

As shown in FIG. 2, the light emitted by each of the LEDs 20a to 20d is incident on the light guide 41 via an optical path coupling portion (not shown) composed of a mirror, a lens, or the like. The light guide 41 is built in the endoscope 12 and the universal cord (not shown). The universal cord is a cord that connects the endoscope 12, the light source device 14, and the processor device 16. The light guide 41 propagates the light from the optical path coupling portion to the tip portion 12d of the endoscope 12.

The tip portion 12d of the endoscope 12 is provided with an illumination optical system 30a and a photographing optical system 30b. The illumination optical system 30a has an illumination lens 42, and the illumination light propagated by the light guide 41 is emitted toward the observation target through the illumination lens 42.

The photographing optical system 30b has an objective lens 43, a zoom lens 44, and an image sensor 45. The image sensor 45 was administered to the observation target via the objective lens 43 and the zoom lens 44, such as reflected light of the illumination light returning from the observation target (in addition to the reflected light, scattered light, fluorescence emitted by the observation target, or administration to the observation target. The observation target is photographed using (including fluorescence caused by the drug). The zoom lens 44 moves by operating the zoom operation unit 12h to enlarge or reduce the observation target image.

The image sensor 45 has a color filter of one of a plurality of color filters for each pixel. In the present embodiment, the image sensor 45 is a color sensor having a primary color system color filter. Specifically, the image sensor 45 has an R pixel having a red color filter (R filter), a G pixel having a green color filter (G filter), and a B pixel having a blue color filter (B filter). ..

As the image sensor 45, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor can be used. Further, although the image sensor 45 of the present embodiment is a primary color system color sensor, a complementary color system color sensor can also be used. Complementary color sensors include, for example, a cyan pixel provided with a cyan color filter, a magenta pixel provided with a magenta color filter, a yellow pixel provided with a yellow color filter, and a green pixel provided with a green color filter. Have. When the complementary color sensor is used, the image obtained from the pixels of each of the above colors can be converted into the same image as the image obtained by the primary color sensor by performing the complementary color-primary color conversion. The same applies to the case where the primary color system or complementary color system sensor has one or a plurality of types of pixels having characteristics other than the above, such as W pixels (white pixels that receive light in almost all wavelength bands). Further, although the image sensor 45 of the present embodiment is a color sensor, a monochrome sensor having no color filter may be used.

The endoscope 12 includes a photographing processor 46 that controls the image sensor 45. The control of the photographing processor 46 is different for each observation mode. In the normal observation mode, the photographing processor 46 controls the image sensor 45 so as to photograph an observation target illuminated by normal light. As a result, the Bc image signal is output from the B pixel of the image sensor 45, the Gc image signal is output from the G pixel, and the Rc image signal is output from the R pixel.

In the special observation mode, the photographing processor 46 controls the image sensor 45 and controls the image sensor 45 so as to photograph an observation target illuminated by special light. As a result, the Bs image signal is output from the B pixel of the image sensor 45, the Gs image signal is output from the G pixel, and the Rs image signal is output from the R pixel.

In the diagnostic support mode, the photographing processor 46 controls the image sensor 45, and controls the image sensor 45 so as to photograph the observation target illuminated by the first illumination light L1 or the second illumination light L2. As a result, when the first illumination light L1 is illuminated, the B1 image signal is output from the B pixel of the image sensor 45, the G1 image signal is output from the G pixel, and the R1 image signal is output from the R pixel. Further, when the second illumination light L2 is illuminated, the B2 image signal is output from the B pixel of the image sensor 45, the G2 image signal is output from the G pixel, and the R2 image signal is output from the R pixel.

The processor device 16 incorporates a program (not shown) related to processing performed by the central control unit 51, the image acquisition unit 52, the image processing unit 56, the display control unit 57, and the like, which will be described later. A central control unit 51, an image acquisition unit 52, an image processing unit 56, and a display control unit are operated by operating the program by a central control unit 51 composed of an image processor included in the processor device 16 that functions as an image processing unit. 57 functions are realized.

The central control unit 51 comprehensively controls the endoscope system 10 such as synchronous control of the irradiation timing of the illumination light and the shooting timing. When various settings are input using the keyboard 19 or the like, the central control unit 51 sets the settings to the endoscope system 10 such as the light source processor 22, the photographing processor 46, or the image processing unit 56. Enter in each part of.

The image acquisition unit 52 acquires an image of an observation target captured using pixels of each color, that is, a RAW image, from the image sensor 45. The RAW image is an image (endoscopic image) before the demosaic processing is performed. If the image is an image before the demosaic processing is performed, the RAW image also includes an image obtained by performing arbitrary processing such as noise reduction processing on the image acquired from the image sensor 45.

The image acquisition unit 52 includes a DSP (Digital Signal Processor) 53, a noise reduction unit 54, and a conversion unit 55 in order to perform various processing on the acquired RAW image as needed.

The DSP 53 includes, for example, an offset processing unit, a defect correction processing unit, a demosaic processing unit, a linear matrix processing unit, a YC conversion processing unit, and the like (none of which are shown). The DSP 53 performs various processing on the RAW image or the image generated by using the RAW image using these.

The offset processing unit performs offset processing on the RAW image. The offset process is a process of reducing the dark current component from the RAW image and setting an accurate zero level. The offset process may be referred to as a clamp process. The defect correction processing unit performs defect correction processing on the RAW image. The defect correction process is a process of correcting or generating a pixel value of a RAW pixel corresponding to a defective pixel of the image sensor 45 when the image sensor 45 includes a pixel (defective pixel) having a defect due to a manufacturing process or a change with time. Is.

The demosaic processing unit performs demosaic processing on the RAW image of each color corresponding to the color filter of each color. The demosaic process is a process of generating pixel values that are missing due to the arrangement of color filters in a RAW image by interpolation. The linear matrix processing unit performs linear matrix processing on the endoscopic image generated by assigning one or a plurality of RAW images to channels of each RGB color. The linear matrix processing is a processing for enhancing the color reproducibility of an endoscopic image. In the YC conversion process performed by the YC conversion processing unit, an endoscope image generated by assigning one or a plurality of RAW images to channels of each RGB color is used as an endoscope having a brightness channel Y, a color difference channel Cb, and a color difference channel Cr. This is the process of converting to an image.

The noise reduction unit 54 performs noise reduction processing on an endoscope image having a brightness channel Y, a color difference channel Cb, and a color difference channel Cr by using, for example, a moving average method or a median filter method. The conversion unit 55 reconverts the luminance channel Y, the color difference channel Cb, and the color difference channel Cr after the noise reduction processing into an endoscopic image having channels of each color of BGR.

The image processing unit 56 performs necessary image processing or calculation on the endoscopic image output by the image acquisition unit 52. The image processing unit 56 generates a plurality of types of recognition target images based on the endoscopic image output by the image acquisition unit 52. Further, control is performed so that at least one type of recognition target image out of a plurality of types of recognition target images is continuously displayed on the display. Further, image recognition processing for a plurality of types of recognition target images is performed in parallel for each type of recognition target image, and recognition processing results obtained by the image recognition processing are acquired for each type of recognition target image. Further, based on the recognition processing results of all the types of recognition target images acquired, the recording operation when recording a moving image of at least one type of recognition target image among the plurality of types of recognition target images is controlled.

As shown in FIG. 9, the image processing unit 56 includes a normal observation image processing unit 61, a special observation image processing unit 62, and a diagnosis support image processing unit 63. The normal observation image processing unit 61 performs image processing for normal observation images on the input Rc image signal, Gc image signal, and Bc image signal for one frame. Image processing for normal observation images includes 3 × 3 matrix processing, gradation conversion processing, color conversion processing such as three-dimensional LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. included. The Rc image signal, the Gc image signal, and the Bc image signal that have been subjected to image processing for a normal observation image are input to the display control unit 57 as a normal observation image.

The special observation image processing unit 62 performs image processing for special observation images on the input Rs image signal, Gs image signal, and Bs image signal for one frame. Image processing for special observation images includes 3x3 matrix processing, gradation conversion processing, color conversion processing such as 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. included. The Rs image signal, the Gs image signal, and the Bs image signal that have been subjected to image processing for the special observation image are input to the display control unit 57 as special observation images.

The diagnosis support image processing unit 63 performs image processing and the like in the diagnosis support mode. As shown in FIG. 10, the diagnosis support image processing unit 63 includes a recognition target image generation unit 71, an image recognition processing unit 72, a recognition result acquisition unit 73, a recording control unit 74, and a display image generation unit 75.

The recognition target image generation unit 71 generates and acquires a plurality of types of recognition target images based on the endoscopic image output by the image acquisition unit 52. The type of the image to be recognized is distinguished by one or both of the following two points. The first point is distinguished by the spectral spectrum of the illumination light when the observation target is photographed. Therefore, the recognition target image generation unit 71 acquires an image obtained by photographing an observation target illuminated by each of a plurality of illumination lights having different spectral spectra from each other emitted by the light source unit as one type of recognition target image. .. The second point is distinguished by the method of image processing for generating the image to be recognized. The method of image processing for generating a recognition target image includes color expansion processing, structure enhancement processing, and the like. When the recognition target image is distinguished by the recognition target image generation image processing method, it includes not performing the recognition target image generation image processing. Therefore, an endoscope image that does not perform image processing for generating a recognition target image on the endoscope image output by the image acquisition unit 52 is also one type of the recognition target image.

Therefore, even if the combination of the spectral spectrum of the illumination light and the image processing for generating the image to be recognized is different, it is regarded as one type of the image to be recognized. Therefore, a recognition target image in which either the spectral spectrum of the illumination light or the image processing is different is regarded as a different type of recognition target image.

As shown in FIG. 11, the recognition target image generation unit 71 is a recognition target image acquisition unit that is distinguished by the spectral spectrum of the illumination light when the observation target is photographed and / or by the method of image processing for recognition target image generation. To prepare for. Therefore, the recognition target image generation unit 71 includes a recognition target image generation unit for each type of recognition target image, and is a first recognition target image generation unit 81, a second recognition target image generation unit 82, and a third recognition target image generation unit. It includes 83, a fourth recognition target image generation unit 84, a fifth recognition target image generation unit 85, and an nth recognition target image generation unit 86. n is an integer of 6 or more. In this embodiment, the following illumination light and / or image processing for generating a recognition target image is performed, respectively.

The first recognition target image generation unit 81 performs the first image processing for generating the first recognition target image. The first image processing is processing applied to the B1 image signal, the G1 image signal, and the R1 image signal obtained by emitting the first illumination light of white light which is the spectral spectrum for the first illumination light. The first image processing is the same as the normal observation image processing in the normal observation image processing unit 61, and the same first recognition target image as the normal observation image is obtained. The first recognition target image is a kind of recognition target image. Therefore, the recognition target image generation unit 71 acquires an image obtained by photographing the observation target illuminated by the white illumination light as one type of the recognition target image.

The second recognition target image generation unit 82 performs the second image processing for generating the second recognition target image. The second image processing is a process applied to the B2 image signal, the G2 image signal, and the R2 image signal obtained by emitting the second illumination light L2 in the second illumination light spectral spectrum SP1. As shown in FIG. 12, in the second illumination light L2 emitted by the spectroscopic spectrum SP1 for the second illumination light, the purple light V has a peak intensity higher than that of the blue light B, the green light G, and the red light R of other colors. Larger light is preferred. The second image processing is a pseudo-color processing in which the B2 image signal is assigned to the B channel and the G channel for display, and the G2 image signal is assigned to the R channel for display. By this pseudo-color processing, a second recognition target image in which a blood vessel or a structure having a specific depth such as a surface blood vessel is emphasized can be obtained. The second recognition target image is a kind of recognition target image.

The third recognition target image generation unit 83 performs a third image process for generating the third recognition target image. The third image processing is a process applied to the B2 image signal, the G2 image signal, and the R2 image signal obtained by emitting the second illumination light in the second illumination light spectroscopic spectrum SP2. As shown in FIG. 13, the second illumination light emitted in the second illumination light spectral spectrum SP2 is preferably light that emits only purple light V (peak wavelength is, for example, 400 to 420 nm). The third image processing is a process of allocating the B2 image signal to the B channel, the G channel, and the R channel for display, and adjusting the color tone and the gradation balance. By the third image processing, a third recognition target image in which the polar superficial blood vessels shallower than the surface blood vessels are emphasized can be obtained. The third recognition target image is a kind of recognition target image.

The fourth recognition target image generation unit 84 performs the fourth image processing for generating the fourth recognition target image. The fourth image processing is obtained by emitting the second illumination light in the spectroscopic spectrum SP3 for the second illumination light in addition to the B1 image signal, the G1 image signal, and the R1 image signal obtained by emitting the first illumination light. This is a process applied to the B2 image signal, the G2 image signal, and the R2 image signal. As shown in FIG. 14, the second illumination light spectroscopic spectrum SP3 is bluish-purple light VB (peak wavelength is, for example, 470 to 480 nm), which is light in a wavelength range in which the extinction coefficients of oxidized hemoglobin and reduced hemoglobin are different. Is preferable.

As shown in FIG. 15, the fourth recognition target image generation unit 84 has a first signal ratio (B2 / G1) representing the ratio between the B2 image signal and the G1 image signal, and the R1 image signal and the G1 image signal. With reference to the oxygen saturation signal ratio calculation unit 84a that performs signal ratio calculation processing to calculate the second signal ratio (R1 / G1) representing the ratio, and the oxygen saturation calculation table 84b, the first signal ratio and the first signal ratio. It includes an oxygen saturation calculation unit 84c that calculates the oxygen saturation corresponding to the two signal ratios, and an oxygen saturation image generation unit 84d that generates an oxygen saturation image based on the oxygen saturation. The oxygen saturation image becomes the fourth recognition target image obtained by the fourth image processing. The fourth recognition target image is a kind of recognition target image.

The oxygen saturation calculation table 84b stores the correlation between the oxygen saturation and the first signal ratio and the second signal ratio. Specifically, as shown in FIG. 16, the oxygen saturation calculation table 84b has oxygen in a two-dimensional space centered on the first signal ratio (B2 / G1) and the second signal ratio (R1 / G1). It is composed of a two-dimensional table that defines the saturation lines ELx, EL1, EL2, EL3, ELy, and the like. For example, the isoline ELx has an oxygen saturation of 0%, the equivalence line EL1 has an oxygen saturation of 30%, the equivalence line EL2 has an oxygen saturation of 50%, and the equivalence line EL3 has an oxygen saturation of 80%. It represents that there is. The positions and shapes of the contour lines with respect to the first signal ratio (B2 / G1) and the second signal ratio (R1 / G1) are obtained in advance by physical simulation of light scattering. The first signal ratio (B2 / G1) and the second signal ratio (R1 / G1) are preferably on a log scale.

The fifth recognition target image generation unit 85 performs the fifth image processing for generating the fifth recognition target image. The fifth image processing is a color expansion processing, and specifically, with respect to the B2 image signal, the G2 image signal, and the R2 image signal obtained by emitting the second illumination light in the second illumination light spectral spectrum SP4. It is a process to be applied. As shown in FIG. 17, the second illumination light spectrum SP4 is preferably light in which the peak intensities of the purple light V and the blue light B are larger than the peak intensities of the green light G and the red light R. Further, it is preferable that the intensity of the red light R is higher than that of the second illumination light spectrum SP2.

In the fifth image processing, as shown in FIG. 18, the first signal ratio (B2 / G2) representing the ratio of the B2 image signal to the G2 image signal and the ratio of the R2 image signal to the G2 image signal are represented. The color difference between a plurality of observation target ranges is calculated based on the signal ratio calculation unit 85a for color difference expansion that performs signal ratio calculation processing for calculating the two signal ratios (G2 / R2) and the first signal ratio and the second signal ratio. A color difference expansion processing unit 85b that performs the color difference expansion processing to be expanded, and a color difference expansion image generation unit 85d that generates a color difference expansion image based on the first signal ratio and the second signal ratio after the color difference expansion processing are provided. The color difference expanded image becomes the fifth recognition target image obtained by the fifth image processing. The fifth recognition target image is a kind of recognition target image.

Regarding the color difference expansion process, as shown in FIG. 19, the distance between a plurality of observation target ranges is expanded in a two-dimensional space consisting of a first signal ratio (B2 / G2) and a second signal ratio (G2 / R2). Is preferable. Specifically, in a two-dimensional space, the first range and the second range (“2”) are maintained in a state where the position of the first range (denoted as “1”) among a plurality of observation target ranges is maintained before and after the color difference expansion process. The distance between the first range and the third range (denoted as "3"), and the distance between the first range and the fourth range (denoted as "4") can be extended. preferable. It is preferable that the color difference expansion process is performed by a method of adjusting the radius and the angle after converting the first signal ratio and the second signal ratio into polar coordinates. It is preferable that the first range is a normal part in which no lesion or the like is present, and the second to fourth ranges are an abnormal part in which a lesion or the like may be present. By the color difference expansion processing, the range A1 in the two-dimensional space before the color difference expansion processing is expanded to the range A2 after the color difference expansion processing, so that the color difference is emphasized. The image will be emphasized.

As described above, by performing image processing of various contents on the endoscopic image, a plurality of types of recognition target images are generated. The nth recognition target image generation unit 86 generates the nth kind of recognition target image. The method or content of image processing is not limited to the above. For example, in addition to the color difference expansion processing, enhancement processing such as structure enhancement processing may be performed. The types of recognition target images are distinguished according to the presence or absence of enhancement processing on the endoscopic image or the type of enhancement processing, and each of the distinguished recognition target images is acquired as one type of recognition target image. The endoscopic image to be enhanced may or may not be subjected to any one of the first image processing to the nth image processing.

The structure enhancement process is a process performed on the acquired endoscopic image so that the blood vessel in the observation target is emphasized and represented as an endoscopic image. Specifically, as the endoscope image, a B1 image signal, a G1 image signal, and an R1 image signal obtained by emitting the first illumination light, or a B2 image signal obtained by emitting the second illumination light. , G2 image signal, and R2 image signal. In the structure enhancement processing, in the acquired endoscopic image, the pixel value (brightness value) is obtained on the horizontal axis, and the density histogram, which is a graph obtained by taking the frequency on the vertical axis, is obtained, and the memory of the image processing unit 56 (not shown). The gradation correction is performed by the gradation correction table stored in advance in the above. The gradation correction table has a gradation correction curve in which the horizontal axis represents an input value and the vertical axis represents an output value, and the correspondence between the input value and the output value is shown. Gradation correction is performed based on the correction curve to widen the dynamic range of the acquired endoscopic image. As a result, in the original image before the enhancement process of structural enhancement, the density is lower in the low density portion and higher in the high density portion. Therefore, for example, the density difference between the blood vessel region and the region where no blood vessel exists. Spreads and the contrast of blood vessels improves. Therefore, in the endoscopic image processed by the structure enhancement process, the contrast of the blood vessel is improved, so that the visibility of the structure of the blood vessel is enhanced, and it is easier and more accurate, for example, of the blood vessel. It is an image that can be used for determination or the like as a specific area having a high density.

Further, the recognition target image generation unit 71 captures an image obtained by photographing an observation target illuminated by illumination light including purple light V and / or blue light B, which is preferably narrow band light, as the recognition target image. It is preferable to produce it as one kind. The generated recognition target image is sent to the image recognition processing unit 72.

The image recognition processing unit 72 performs image recognition processing for a plurality of types of recognition target images in parallel for each type of recognition target image. The image recognition process is performed in order to output a specific state reflected in the observation target image as a recognition process result. As shown in FIG. 20, the image recognition processing unit 72 includes a first image recognition unit 91, a second image recognition unit 92, a third image recognition unit 93, and a fourth image recognition unit, which are provided for each type of image to be recognized. It includes 94, a fifth image recognition unit 95, and an nth image recognition unit 96. n is an integer of 6 or more, and includes a number of image recognition units corresponding to the number of types of images to be recognized. The image recognition processing unit 72 includes an image recognition unit for each type of the recognition target image, and each image recognition unit performs image recognition processing for the corresponding recognition target image in parallel and independently. Therefore, the first image recognition unit 91 performs the image recognition processing of the first recognition target image, the second image recognition unit 92 performs the image recognition processing of the second recognition target image, and the third image recognition unit 93 performs the image recognition processing. 3 Image recognition processing of the recognition target image is performed, image recognition processing of the 4th recognition target image is carried out in the 4th image recognition unit 94, and image recognition processing of the 5th recognition target image is carried out in the 5th image recognition unit 95. Then, the nth image recognition unit 96 performs the image recognition processing of the nth recognition target image.

These image recognition processes are performed in parallel and independently. Further, in the first to nth image recognition units, how many image recognition units are used is set depending on how many types of recognition target images are acquired. Further, for example, when a plurality of recognition target images are acquired in one of the acquired plurality of types of recognition target images, it is sufficient to perform image recognition processing on at least one of the recognition target images. It is preferable to perform the image recognition process on all of the plurality of types of recognition target images because the recognition accuracy is improved and the control accuracy of the moving image recording operation is also improved.

For the image recognition processing, a conventional method of image recognition processing can be used. In the image recognition processing unit 72, each image recognition unit from the first image recognition unit 91 to the nth image recognition unit 96 may perform the image recognition processing of the same method, depending on the type of the image to be recognized. , Each image recognition unit may perform image recognition processing by a method different from each other. When performing image recognition processing of different methods depending on the type of the image to be recognized, it is preferable to select and perform the image recognition processing method that can obtain good image processing results according to the type of the image to be recognized. ..

Examples of the image recognition processing method include a method of performing pattern recognition using image processing, a method of using machine learning technology, and the like. Specifically, for example, a method using an image-based value such as a pixel value and / or a brightness value of a recognition target image, a method using a value of biological information such as oxygen saturation calculated from the image, or an observation target. Examples thereof include a method of using correspondence information in which a specific state and each recognition target image obtained by photographing an observation target including the specific state are associated in advance. By these methods of the image recognition processing, the specific state of the observation target obtained by detecting the part or the like in the observation target reflected in the recognition target image or determining the disease or the like is output as the recognition processing result.

The specific state in the observation target is a non-living object such as a site, a specific structure, or a treatment tool, including the observation target, the presence or absence of a lesion, the name of the lesion or disease, and the probability or progression of the lesion or disease. Degree or peculiar biometric information value. Further, the specific state in the observation target includes the accuracy of the image recognition result obtained by the image recognition process and the like. The site is preferably a characteristic site that appears in the endoscopic image. For example, in the case of the upper gastrointestinal tract, it is the esophagus, the sulcus, the colon, the gastric body, the pylorus, the angular incisure, or the duodenal bulb, and in the case of the colon, the cecum, the circumflex, and the ascending. Colon, transverse colon, descending colon, sigmoid colon, or cecum. Specific structures include ridges or depressions such as blood vessels, ducts, polyps or cancers, and non-biological objects include biopsy forceps, snares, or foreign body removal that can be attached to an endoscope. A treatment tool such as a device, or a treatment tool for abdominal cavity used for laparoscopic surgery. Names of lesions or diseases include those found on endoscopy of the upper gastrointestinal tract or large intestine, such as inflammation, redness, bleeding, ulcers, or polyps, or gastritis, Barrett's esophagus, cancer or. Ulcerative colitis, etc. The biological information value is a value of biological information to be observed, and is, for example, an oxygen saturation, a blood vessel density, a fluorescence value due to a dye, or the like.

The image recognition processing unit 72 may include a correspondence information acquisition unit that acquires correspondence information in which the specific state of the observation target and the recognition target image obtained by photographing the observation target in the specific state are associated in advance. preferable. Then, it is preferable that the image recognition processing unit 72 performs image recognition processing on the newly acquired recognition target image based on the corresponding information. When the specific state of the observation target is known in advance, the correspondence information corresponds to the recognition target image obtained by photographing the observation target and the information such as the specific state of the observation target or the region of the specific state. Information.

As shown in FIG. 21, by inputting a newly acquired recognition target image unknown about a specific state into the correspondence information acquisition unit, the recognition target image provided in the correspondence information acquisition unit corresponds to the specific state of the observation target. Using the attached correspondence information, it is possible to estimate a specific state in the newly acquired recognition target image and output it as a recognition processing result. Further, each correspondence information acquisition unit may perform learning or feedback to further acquire the newly acquired recognition target image and the specific state included in the recognition processing result output by estimation as correspondence information.

Each image recognition unit includes a correspondence information acquisition unit that includes correspondence information about the corresponding recognition target image. The correspondence information acquisition unit performs image recognition processing of the recognition target image based on the correspondence information, and outputs details such as an area related to a specific state of the observation target included in the recognition target image. It should be noted that the output of details regarding the specific state includes the content such as "does not include the specific state".

Also, depending on the type of image to be recognized, the specific state of the observation target that can obtain good results by image recognition processing may differ. Therefore, each type of recognition target image is provided with a corresponding information acquisition unit associated with a specific type of specific state, so that each type of recognition target image can obtain good results by image recognition processing. Therefore, it is preferable. For example, when the type of the recognition target image is a recognition target image in which a blood vessel is emphasized, the image recognition processing unit corresponding to this type of recognition target image includes correspondence information associated with a specific state regarding the observation target blood vessel. It is used as an image recognition processing unit and outputs a specific state related to the blood vessel to be observed.

As shown in FIG. 20, the first image recognition unit 91 includes a first correspondence information acquisition unit 101, and performs a first image recognition process on a first recognition target image. Similarly, the second image recognition unit 92 includes a second correspondence information acquisition unit 102, performs a second image recognition process on the second recognition target image, and the third image recognition unit 93 acquires the third correspondence information. A unit 103 is provided to perform a third image recognition process on the third recognition target image, and a fourth image recognition unit 94 is provided with a fourth corresponding information acquisition unit 104 and a fourth image with respect to the fourth recognition target image. The fifth image recognition unit 95 includes the fifth corresponding information acquisition unit 105, performs the fifth image recognition process on the fifth recognition target image, and the nth image recognition unit 96 performs the recognition process. The n-correspondence information acquisition unit 100 is provided, and the nth image recognition process is performed on the nth recognition target image.

Each correspondence information acquisition unit is, for example, a trained model in machine learning. Since the specific state of the observation target in the newly acquired image to be recognized can be obtained as the image recognition result more quickly or accurately, the image recognition process using the trained model by machine learning as the corresponding information acquisition unit is performed. Is preferable. In the present embodiment, each correspondence information acquisition unit performs image recognition processing for outputting a specific state of an observation target using a trained model in machine learning. In this case, it is preferable to use the trained model that has been trained for each type of recognition target image in order to obtain good recognition processing results. Therefore, for example, it is preferable that the first correspondence information acquisition unit 101 and the second correspondence information acquisition unit 102 are trained models that are different from each other.

The plurality of types of recognition target images include at least a first recognition target image and a second recognition target image of different types from each other, and the image recognition processing unit 72 performs first image recognition processing for the first recognition target image. It is preferable to perform a second image recognition process different from the first image recognition process for the second image to be recognized. The first image recognition process is preferably performed with respect to the detection of a specific part or an object other than a living body in the first recognition target image. It is preferable that the second image recognition process is performed with respect to the determination that the region of the specific state is included in the second recognition target image.

In the present embodiment, the first recognition target image similar to the normal observation image using the first illumination light L1 and the third recognition in which the polar surface blood vessels shallower than the surface blood vessels using the second illumination light L2a are emphasized. The target image and the fifth recognition target image, which is a color difference expanded image using the second illumination light L2b, are used by the first image recognition unit 91, the third image recognition unit 93, and the fifth image recognition unit 95, respectively. , Performs image recognition processing for each recognition target image. Since the first recognition target image is a normal observation image, the first image recognition process satisfactorily detects a specific portion. Since the third recognition target image is an image in which the polar surface blood vessels and the like are emphasized, the third image recognition process satisfactorily determines that a lesion such as a mucous membrane on the surface layer is included. Since the fifth recognition target image is a color difference expanded image, the fifth image recognition process can satisfactorily determine, for example, a severe region among the severity of ulcerative colitis as a specific region to be observed. conduct. The severity of ulcerative colitis is classified into, for example, mild, moderate, severe, or severe.

As shown in FIG. 22, in the present embodiment, the first illumination light L1 is emitted in the first A emission pattern of five frames, and the second illumination light L2 is emitted in the second B emission pattern of one frame, and an endoscopic image is acquired. do. In the second illumination light, the second illumination light L2b and the second illumination light L2a are switched in order. The fifth recognition target image IM2b is acquired based on the endoscopic image obtained by the second illumination light L2b. The third recognition target image IM2a is acquired based on the endoscopic image obtained by the second illumination light L2a. In FIG. 22, the recognition target image obtained by the second illumination light L2 is shown with diagonal lines, and the third recognition target image IM2a obtained by the second illumination light L2a and the fifth recognition obtained by the second illumination light L2b are shown. The target image IM2b is distinguished from the target image IM2b by different types of diagonal lines. The first recognition target image IM1 is acquired based on the endoscopic image obtained by the first illumination light L1.

The first image recognition unit 91 detects a specific part, for example, the rectal part, with respect to the first recognition target image IM1, and generates a first recognition processing result including information on the detection. The third image recognition unit 93 determines whether or not the severity of ulcerative colitis includes a severe region with respect to the third recognition target image IM2a, and generates a third recognition processing result including information on the determination. do. The fifth image recognition unit 95 determines whether or not the fifth recognition target image IM2 includes a bleeding spot, and generates a fifth recognition processing result including information on the determination.

Information on detection includes at least "detection" with detection or "non-detection" without detection. Further, the information regarding the determination includes at least a "pass determination" that satisfies the determination condition or a "failure determination" that does not satisfy the determination condition. Therefore, the first recognition processing result includes the information of "detection" when it is estimated that the first recognition target image includes the rectal portion in the observation target. In addition, the third recognition processing result includes information of "confirmation determination" when the third recognition target image is estimated to include a region where the severity of ulcerative colitis is severe in the observation target. In addition, the fifth recognition processing result includes information of "match determination" when it is estimated that the fifth recognition target image includes a bleeding spot in the observation target.

The recognition result acquisition unit 73 acquires the recognition processing result of the recognition target image obtained by the image recognition processing by each image recognition unit. The recognition result acquisition unit 73 acquires the recognition processing result based on the recognition processing result of all the acquired recognition target images. The recognition processing result preferably includes information on whether or not the corresponding recognition target image satisfies or does not satisfy a preset condition. The preset condition is that the recognition processing result when the image recognition processing is performed using the recognition target image is "detection" or "match judgment", and the recognition processing result is "detection" or "match". When the information of "judgment" is included, the information that the condition is satisfied is included.

In the present embodiment, there are three types of recognition processing: the first recognition processing result which is the detection result of the specific part, the third recognition processing result and the fifth recognition processing result which are the determination results including the region of the specific state. Get the result. The third recognition treatment result is a determination result of whether or not a lesion is included, and the fifth recognition treatment result is a determination result of whether or not a severe region of ulcerative colitis is included.

As shown in FIG. 23, the first recognition target image IM1 detecting a specific portion is shown with a dot pattern. The "first recognition" line indicates what the first image recognition process detects. The "result" one line below the "first recognition" indicates the information regarding the detection included in the first recognition processing result. Similarly, the "fifth recognition" line indicates what the fifth image recognition process determines, and the "result" in the line below it indicates information regarding the determination included in the fifth recognition process result. Similarly, the "third recognition" line indicates what the third image recognition process determines, and the "result" in the line below it contains information regarding the determination included in the third recognition process result. show.

Further, in the "result" of the "first recognition", the description of "specific site detection" indicates that the information that the specific site was detected by the first recognition target image IM1x was obtained, and thereafter. The description of "specific site non-detection" in the first recognition target image IM1y indicates that the information that the specific site was not detected was obtained by the first recognition target image IM1y. The code of "x" attached to the recognition target image indicates the first image for which "detection" or "match judgment" is first obtained in the image recognition process of the same type of recognition target image, and the code of "y" is used. , "Not detected" or "No" after obtaining "Detection" or "Failure" is shown. The "result" when the sign "x" or "y" is added is underlined. In the following figure, similar reference numerals indicate similar contents.

As shown in FIG. 23, when a specific part is detected in the first recognition target image IM1x, the first recognition processing result includes the information that the recognition target image IM1x "detects the specific part". In the present embodiment, a specific part is first detected in the first recognition target image IM1x, and then the specific part is continuously detected in the first recognition target image IM1 obtained thereafter, and the specific part is specified in the first recognition target image IM1y. Detection of a specific site continued until the site was not detected. In the third recognition processing result and the fifth recognition processing result, while shown in FIG. 23, no information of "pass determination" was included.

The recognition result acquisition unit 73 acquires the recognition processing result as soon as the image recognition processing unit 72 generates the recognition target result based on each recognition target image. When a recognition result acquisition unit 73 of the same type is newly obtained, the recognition result acquisition unit 73 deletes the recognition processing result immediately before the same type, and always has the latest recognition processing results based on all types of recognition target images. It is preferable that all types of recognition target images are all types within one light emission cycle due to the light emission pattern of the illumination light or the like. The emission pattern of the illumination light can be switched arbitrarily or depending on the recognition processing result or the like, but in that case, it is preferable to use all types within the emission cycle after the switching. The recognition processing results of all the acquired recognition target images are sent to the recording control unit 74.

The recording control unit 74 controls the recording operation when recording a moving image of at least one type of recognition target image among a plurality of types of recognition target images based on the recognition processing results of all types of recognition target images acquired. Based on the recognition processing results of all the acquired recognition target images, the recording operation is controlled after examining at least one recognition processing result for each type acquired by the recognition result acquisition unit 73 for all types. That is.

The moving image of the recognition target image that is the target for recording the moving image can be one or more of the multiple types of recognition target images to be acquired. When recording one type of moving image among a plurality of types of recognition target images, it is preferably the first recognition target image. This is because the first recognition target image is a normal observation image, which is a moving image taken in a natural color, and the recorded moving image has a wide range of uses and is generally useful.

The recording operation is an instruction regarding the execution of recording, and includes, for example, the start of recording when recording is not performed. Also, if recording has already been performed, it includes continuation and suspension of recording. In addition, it includes operations normally performed for recording execution, such as recording or stopping for a certain period of time, such as pausing recording. The recording control unit 74 controls operations such as start and stop of recording from all types of recognition processing results based on the plurality of types of recognition target images acquired by the recognition result acquisition unit 73.

When the recording control unit 74 includes information that at least one of the recognition processing results of all types of recognition target images acquired by the recognition result acquisition unit 73 satisfies the condition, it is preferable to start or continue recording. Further, when all the recognition processing results of all types of recognition target images acquired by the recognition result acquisition unit 73 include information that the conditions are not satisfied and the recording is continued, it is preferable to stop the recording. ..

In FIG. 24, in the present embodiment, the recording control unit 74 uses the recognition processing results of three types of recognition target images, the first recognition target image IM1, the fifth recognition target image IM2b, and the third recognition target image IM2a. Based on this, the operation of recording when recording the moving image of the first recognition target image IM1 is controlled. In the recording control unit 74, when the first recognition processing result of the first recognition target image IM1 is newly acquired, it is replaced with the first recognition processing result acquired immediately before, and the fifth recognition target image IM2b or the third recognition When each of the target images IM2a is newly acquired, the target image IM2a controls the recording operation based on the latest plurality of types of recognition target images by replacing the images acquired immediately before.

The recording control unit 74 sets the latest recognition target image IM2b "" at time t1 when the recognition processing result including "detection" is obtained by the first image recognition process of the first recognition target image IM1x. Since it is the case that the recording has not already started based on both of "No judgment", the control to start the recording is performed. If recording has already started, control is performed to continue recording, but by substantially not performing recording operation, control is performed to continue recording. Recording is continued as long as the information of "detection" or "match determination" is included in the recognition processing result of the first recognition target image or the third recognition target image. After that, at the time t2 when the recognition processing result by the first recognition target image IM1y includes "non-detection", the latest third recognition target image IM2a is also "no judgment", and both are negative recognition processing results. Therefore, control is performed to stop recording. The negative recognition processing result is "non-detection" or "negative judgment", and the positive recognition processing result is "detection" or "pass judgment". In the present embodiment, in this way, a recorded file in which the first recognition target image from the time t1 to the time t2 is recorded can be obtained.

When a positive recognition processing result by the first recognition target image IM1x is acquired as in the present embodiment, the type of the second illumination light L2 is switched from the second illumination light L2b to the second illumination light L2a. The recognition target image to be acquired may be switched from the fifth recognition target image IM2b to the third recognition target image IM2a. As a result, when a specific part is recognized by the first recognition target image IM1, it is automatically switched to the second illumination light L2 suitable for observing the specific part set in advance, and the lesion is overlooked. Can be reduced more.

The display image generation unit 75 uses at least one type of recognition target image out of a plurality of types of recognition target images to generate a display image to be displayed on the display 18. For example, when the first recognition target image similar to the normal observation image is continuously displayed on the display 18, the display image generation unit 75 uses the first recognition target image for image processing to obtain a display image. To generate a display image. The generated display image is sent to the display control unit 57.

The display control unit 57 displays the normal image on the display 18 in the normal observation mode, and displays the special observation image on the display 18 in the special observation mode. Further, in the case of the diagnosis support mode, control is performed so that at least one type of recognition target image among the recognition target images is continuously displayed on the display 18. In the frame where the type of the recognition target image to be displayed on the display 18 is not acquired, the display control unit 57 performs control such as continuously displaying the display image or the like acquired immediately before. In the present embodiment, since the display image based on the first recognition target image similar to the normal image is sent to the display control unit 57, the normal image is continuously displayed on the display 18. Although it is not displayed, the acquired fifth recognition target image and / or third recognition target image may be displayed on the display 18 as a display image according to an instruction.

With the above configuration, the processor device 16 that functions as an image processing device, the endoscope system 10 that includes the image processing device, and the like are recognized targets based on the recognition processing results of all types of acquired images to be recognized. In order to control the operation of recording a moving image, the recording is automatically controlled, and the troublesomeness of manually controlling the recording is eliminated. In addition, since it is possible to control the video recording operation in more detail than when controlling the video recording operation using the recognition processing result of one type of recognition target image, the recording that corresponds flexibly to the purpose is automatically performed. It can be carried out. In addition, since the operation of recording the recording is controlled in detail, there is no need to save the recording of unnecessary scenes, the storage can be saved, it is easy to search when using the recording later, and the endoscopy recording. It is possible to improve the usability of the. Further, since a plurality of types of recognition target images are used, even lesions that are difficult to recognize in the image displayed on the display 18 may be automatically recorded, which can reduce oversight of lesions and the like. can. As described above, the processor device 16, the endoscope system 10, and the like efficiently record the endoscope images and the like when the image recognition processes are performed in parallel based on a plurality of types of endoscope images. be able to.

In the case of FIG. 24, the recording operation was controlled because the first recognition processing result detected or did not detect a specific part, but the recording was performed based on the recognition processing results other than the first recognition processing result. The operation of may be controlled. As shown in FIG. 25, in this case, the first illumination light L1 is emitted in the first A emission pattern of five frames, and the second illumination light L2 is emitted in the second B emission pattern of one frame to acquire an endoscopic image. .. In the second illumination light, the second illumination light L2b and the second illumination light L2c are switched in order. The fifth recognition target image IM2b is acquired based on the endoscopic image obtained by the second illumination light L2b. The second recognition target image IM2c is acquired based on the endoscopic image obtained by the second illumination light L2c. In FIG. 25, the recognition target image obtained by the second illumination light L2 is shaded and shown, and the fifth recognition target image IM2a obtained by the second illumination light L2b and the second recognition obtained by the second illumination light L2c are shown. The target image IM2c is distinguished from the target image IM2c by adding a different pattern. The first recognition target image IM1 is acquired based on the endoscopic image obtained by the first illumination light L1.

At time t1, since it was determined in the recognition target image IM2bx that the area of the lesion was included, the information that the fifth recognition processing result satisfies the condition is included. Information that the recording control unit 74 satisfies one or more of the latest recognition processing results of all types, the first recognition processing result, the fifth recognition processing result, and the second recognition processing result. Since it contains, recording is started. It should be noted that, at time t1, the second recognition processing result may not be acquired, but in order to satisfy the condition in the fifth recognition processing result, even at time t1, among all types of recognition processing results. One or more meet the conditions.

In FIG. 25, the image to be recognized with a dot pattern is determined to include a lesion area. The illumination pattern of the second illumination light L2 is a pattern in which the second illumination light IM2b for the fifth recognition target image and the second illumination light IM2c for the second recognition target image are sequentially switched and used. be. After that, the determination that the region of the lesion was included in the fifth recognition target image continued, and the first recognition target image and the fifth recognition processed image did not detect or determine the condition, so recording was continued. Then, when it was determined in the recognition target image IMB2y that the area of the lesion was no longer included in the "non-lesion", the first recognition target image and the fifth recognition processed image were not detected or determined to satisfy the conditions, so that the recording was performed. It was stopped. In this way, the operation of recording can be controlled even based on the determination in the second or fifth recognition processing result. Therefore, even if a lesion or the like is not detected in the first recognition target image displayed on the display 18, recording is automatically performed, so that recording of a region of interest or the like that may be a lesion may be missed. do not have.

In the case of FIGS. 24 and 25, the recording operation was controlled by a positive detection or determination in one type of recognition processing result of the first recognition processing result or the fifth recognition processing result, but it is positive. Even if the detection or determination occurs in two or more types of recognition processing results, the recording operation is appropriately controlled. As shown in FIG. 26, in the illumination light emission pattern similar to that in FIG. 25, it is determined that the recognition target image IM2bx includes the lesion region at time t1, and recording is started. After that, the treatment tool was detected in the first recognition target image IM1x, but since the recording had already started, the recording operation was not performed and the recording was continued as it was. After that, at time t2, the treatment tool was not detected in the first recognition target image IM1y, but since the positive determination of the lesion as a result of the fifth recognition process continued, the recording stop operation was not performed. Recording was continued as it was. After that, the determination of the lesion is negative in the fifth recognition processed image IM2by, and the detection or determination is positive in the first recognition processing result, the second recognition processing result, and the fifth recognition processing result at this time. Since there are no conditions, the operation to stop recording was performed. As described above, even when there are a plurality of recognition processing results satisfying the conditions, the recording operation can be appropriately controlled.

Even if the type of image to be recognized is changed, the recording operation is properly controlled. As shown in FIG. 27, in this case, first, the first illumination light L1 is emitted in the first A emission pattern of 5 frames, and the second illumination light L2 is emitted in the second B emission pattern of 1 frame, and the endoscopic image is obtained. get. In the second illumination light, the second illumination light L2b and the second illumination light L2c are switched in order. The fifth recognition target image IM2b is acquired based on the endoscopic image obtained by the second illumination light L2b. The second recognition target image IM2c is acquired based on the endoscopic image obtained by the second illumination light L2c. When the information of "detection" is included in the first recognition processing result, the second illumination light is switched to the second illumination light L2d, and after switching to the second illumination light L2d, the second recognition processing result is "combined". The acquisition of the second illumination light L2d and the second recognition target image corresponding to the second illumination light L2d is continued until the "rejection determination" after the "determination" is included.

In FIG. 27, the recognition target image obtained by the second illumination light L2 is shaded and shown, and the fifth recognition target image IM2a obtained by the second illumination light L2b and the second recognition obtained by the second illumination light L2c are shown. The target image IM2c and the fourth recognition target image IM2d obtained by the fourth illumination light L2d are shown separately with different patterns. The first recognition target image IM1 is acquired based on the endoscopic image obtained by the first illumination light L1.

As shown in FIG. 27, first, at time t1, it was determined that the region of the lesion was included in the fifth recognition target image IM2bx, and recording of the first recognition target image was started after time t1. After that, at time t2, it was determined in the second recognition target image IM2cx that the blood vessel was dense, but since the recording had already started, the recording operation was not performed and the recording was continued as it was. Then, at time t3, the treatment tool was detected in the first recognition target image IM1x. With this as an opportunity, the second illumination light L2b and the second illumination light L2a were sequentially irradiated to the second illumination light L2d for the fourth recognition target image, and the second illumination as the second illumination light. Light L2d was continuously used. At time t4, it was determined by the fourth recognition target image IM2dx that there was a bleeding point, but since the recording was continued, the recording was continued as it was.

After that, at time t5, the treatment tool was not detected in the first recognition target image IM1y. However, as the recognition target image at this point, two types of the recognition target image, the first recognition target image and the fourth recognition target image, have been acquired, and the positive judgment by the fourth recognition target image has continued, so the recording is continued. did. After that, at time t6, the operation of stopping the recording was performed because there is no condition that the detection or determination is affirmative based on the information that the bleeding point is not determined in the fourth recognition target image IM2d. As described above, even when the type of the recognition target image is changed from the fifth recognition target image and the third recognition target image to the fourth recognition target image, the recording operation can be appropriately controlled.

Note that FIG. 27 shows a series of operations in an example of an endoscopic procedure. For example, when excising a lesion using an endoscope and a treatment tool provided at the tip of the endoscope, recording is started by determining the lesion area based on the result of the fifth recognition process, and the second recognition process is performed. Based on the results, in order to determine the range of excision of the lesion using the treatment tool, the area where the blood vessels are dense is determined, the treatment is performed using the treatment tool, and the bleeding after the treatment is based on the result of the fourth recognition treatment. Judge the point and stop recording when the bleeding point disappears. Therefore, it is possible to appropriately control the recording operation in a series of operations of the procedure.

When recording a moving image, the recording control unit 74 preferably attaches information regarding conditions corresponding to the recording operation to the moving image. The information regarding the conditions corresponding to the recording operation is the content of the recognition processing result that triggered the operation. Recording operations include starting, continuing, or stopping recording. Further, since the recognition processing result is recorded, it is preferable to attach information on the positive content of the recognition processing result even if it does not trigger the operation. When the information is attached to a moving image, it may be recorded as a chapter which is a break in the moving image recording. By attaching the recording operation and information related to the operation to the moving image, the recognition processing result and what caused the recording operation are clearly recorded. Therefore, when reviewing what has been recorded, the desired recorded portion can be easily reached, which improves usability.

It is preferable to save the video as an individual recording file. Further, depending on the case, a plurality of means for recording may be provided. By using a plurality of recording means, a plurality of recordings can be performed at the same time. For example, the first recognition target image or the like which is an image by the first illumination light L1 and the second recognition target image or the like which is an image by the second illumination light L2 may be recorded independently. In this case, it is preferable to attach information on the type of the recorded recognition target image to the recorded file. The storage of the recorded file may be built in the image processing device or may be an external storage device. Further, it may be installed on the network to which the image processing device is connected.

As shown in FIG. 28, by the recording operation such as the start and stop of recording, a plurality of recorded files are generated with file names such as "File I" and "File II" in one inspection. Each recording file is accompanied by information on the conditions corresponding to the operation of recording, the recognition processing result that triggered the start or continuation of recording, and the time information at which recording was started, continued, or stopped. The information regarding the corresponding condition is the information regarding the recognition processing result that triggered the start, continuation, or stop of recording. These are preferably attached to the recorded file as so-called tags. The file I, which is a recording file, is tagged with a "part" related to the detection of the first recognition processing result, and the recording start time is set according to the detection and non-detection of the part which is the first recognition processing result. And t1 and t2 of the stop time are attached. File II is tagged with "lesion / severe / treatment tool / bleeding point" relating to the detection and acceptance determination of the first recognition processing result and the third to fifth recognition processing results. Further, t3 of the recording start time is attached according to the judgment of the lesion or the seriousness which is the result of the fifth recognition processing or the result of the third recognition processing, and according to the judgment of whether or not the bleeding point is the result of the fourth recognition processing. The recording stop time t4 is attached.

In addition to controlling the recording operation when recording a moving image of the image to be recognized based on the results of a plurality of types of recognition processing, the recording operation may be controlled based on other information regarding the inspection by the endoscope. .. As other information, for example, when a means for displaying the shape of the endoscope is used, recording is performed when the tip portion 12d of the endoscope is located at a preset position, as can be seen from the shape of the endoscope. Can be started. The preset location is, for example, a location where there was a lesion in the past based on patient data. This ensures that the course of the lesion is recorded. Further, depending on the case, not only the recognition processing result but also the recording operation may be controlled so as to record a specific recognition target image. For example, if you want to record all oxygen saturation images, set to start recording when acquiring the 4th recognition target image and stop recording when switching to acquisition of other recognition target images. May be good.

In addition to displaying the recognition target image, the display control unit 57 may superimpose the recognition processing result information on the recognition target image to form a display image and control the display. good. For example, the display of the severely ill portion, which is the result of the third recognition process, may be superimposed on the first recognition target image by information such as text or a figure and displayed on the display 18. Further, the information of the recording operation may be controlled so as to be superimposed and displayed on the recognition target image. For example, an indicator indicating recording execution may be provided on a part of the display 18, and may be displayed by blinking in red to draw attention when recording is executed and turning off when recording is not executed.

Next, a series of processes related to the control of the moving image recording operation performed by the processor device 16 or the endoscope system 10 which is an image analysis processing device will be described with reference to the flowchart shown in FIG. 29. A plurality of types of recognition target images of the first recognition target image, the fifth recognition target image, and the third recognition target image are acquired by the illumination light of the first A emission pattern and the second B pattern (step ST110). The display control unit continuously displays the first recognition target image on the display 18. Image recognition processing is performed in parallel for each type of recognition target image for a plurality of types of recognition target images (step ST120). Therefore, the specific site is detected by the first recognition target image, the lesion is determined by the fifth recognition target image, and the serious condition is determined independently by the third recognition target image. The recognition result acquisition unit 73 acquires a plurality of recognition processing results (step ST130). The recording control unit 74 starts recording when there is one or more positive detections or determination results of "detection" or "match determination" from the plurality of types of recognition processing results (Y in step ST140) (Y). Step ST150). If there is no positive detection or judgment result of "detection" or "match judgment" (N in step ST140), the process returns to the acquisition of the image to be recognized.

After starting recording, continuously acquire multiple types of recognition target images (step ST160). Image recognition processing is performed in parallel for each type of recognition target image for a plurality of types of recognition target images (step ST170). The recognition result acquisition unit 73 acquires a plurality of recognition processing results (step ST180). The recording control unit 74 stops recording when there is no positive detection or determination result of "detection" or "match determination" from the plurality of types of recognition processing results (Y in step ST190) (step ST200). .. When there is one or more positive detections or judgment results of "detection" or "match judgment" (N in step ST190), the process returns to the acquisition of the image to be recognized and the recording is continued.

In the above-described embodiment and modification, the processor device 16 functions as an image processing device, but an image processing device including an image processing unit 56 may be provided separately from the processor device 16. In addition, as shown in FIG. 30, the image processing unit 56 has taken an image with the endoscope 12, for example, directly from the endoscope system 10 or indirectly from the PACS (Picture Archiving and Communication Systems) 910. It can be provided in the diagnostic support device 911 that acquires a RAW image. Further, as shown in FIG. 31, it is connected to various inspection devices such as the first inspection device 921, the second inspection device 922, ..., the K inspection device 923, including the endoscope system 10, via the network 926. The image processing unit 56 can be provided in the medical service support device 930.

Each of the above embodiments and modifications can be carried out by arbitrarily combining a part or all of them. Further, in each of the above embodiments and modifications, the endoscope 12 uses a so-called flexible endoscope having a flexible insertion portion 12a, but the observation target swallows and uses a capsule-type endoscope. The present invention is also suitable when a rigid endoscope (laparoscope) used for a mirror, surgery, or the like is used.

The above-described embodiment and modification are an operation method of an image processing device including an image processor and performing image recognition processing based on an image obtained by photographing an observation target with an endoscope. An image acquisition step for acquiring a plurality of types of recognition target images based on the above, a display control step for controlling to continuously display at least one type of recognition target image among a plurality of types of recognition target images on a display, and a plurality of types. Image recognition processing step that performs image recognition processing for the recognition target image in parallel for each type of recognition target image, and recognition processing result acquisition that acquires the recognition processing result obtained by the image recognition processing for each type of recognition target image. A step and a recording control step that controls a recording operation based on the recognition processing results of all types of the recognition target image when recording a moving image of at least one type of the recognition target image among a plurality of types of recognition target images. The method of operating the image recognition processing device provided is included.

Further, the above-described embodiment and modification are image processing installed in an image processing apparatus provided with an image processor and performing image recognition processing based on an image obtained by photographing an observation target with an endoscope. In the device program, the computer has an image acquisition function for acquiring multiple types of recognition target images based on images, and control for continuously displaying at least one type of recognition target image among the multiple types of recognition target images on the display. The display control function to be performed, the image recognition processing function to perform image recognition processing for a plurality of types of recognition target images in parallel for each type of recognition target image, and the recognition processing result obtained by the image recognition processing are the recognition target images. When recording a moving image of at least one type of recognition target image out of a plurality of types of recognition target images and a recognition processing result acquisition function acquired for each type, recording is performed based on the recognition processing results of all types of recognition target images. Includes a program for an image processing device to realize a recording control function that controls operation.

In the above embodiment, various types such as a central control unit 51, an image acquisition unit 52, a DSP 53, a noise reduction unit 54, a conversion unit 55, an image processing unit 56, and a display control unit 57 included in the processor device 16 which is an image processing device. The hardware structure of the processing unit that executes processing is various processors as shown below. For various processors, the circuit configuration is changed after manufacturing the CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software (program) and functions as various processing units. It includes a programmable logic device (PLD), which is a possible processor, a dedicated electric circuit, which is a processor having a circuit configuration specially designed for performing various processes, and the like.

One processing unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs or a combination of a CPU and an FPGA). May be done. Further, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client or a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which this processor functions as a plurality of processing units. Second, as typified by System On Chip (SoC), there is a form that uses a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Furthermore, the hardware-like structure of these various processors is, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.

In addition to the endoscopic system for acquiring endoscopic images, processor devices, and other related devices, the present invention is a system or device for acquiring medical images (including moving images) other than endoscopic images. It can also be used in such cases. For example, the present invention can be applied to an ultrasonic inspection device, an X-ray imaging device (including a CT (Computed Tomography) inspection device, a mammography device, etc.), an MRI (magnetic resonance imaging) device, and the like.

10 Endoscope system 12 Endoscope

12a Insertion part

12b Operation part 12c Curved part

12d Tip part

12e Angle knob 12f Scope button 1st 12g Scope button 2nd 12h Zoom operation part 14 Light source device 16 Processor device 18 Display 19 Keyboard 20 Light source Part 20a V-LED
20b B-LED
20c G-LED
20d R-LED
22 Light source processor 30a Illumination optical system 30b Imaging optical system 41 Light guide 42 Illumination lens 43 Objective lens 44 Zoom lens 45 Image sensor 46 Imaging processor 51 Central control unit 52 Image acquisition unit 53 DSP
54 Noise reduction unit 55 Conversion unit 56 Image processing unit 57 Display control unit 61 Normal observation image processing unit 62 Special observation image processing unit 63 Diagnosis support image processing unit 71 Recognition target image generation unit 72 Image recognition processing unit 73 Recognition result acquisition unit 74 Recording control unit 75 Display image generation unit 81 1st recognition target image generation unit 82 2nd recognition target image generation unit 83 3rd recognition target image generation unit 84 4th recognition target image generation unit 84a Oxygen saturation signal ratio calculation unit 84b Oxygen saturation calculation table 84c Oxygen saturation calculation unit 84d Oxygen saturation image generation unit 85 Fifth recognition target image generation unit 85a Color difference expansion signal ratio calculation unit 85b Color difference expansion processing unit 85c Color difference expansion image generation unit 86th n Recognition target image generation unit 91 1st image recognition unit 92 2nd image recognition unit 93 3rd image recognition unit 94 4th image recognition unit 95 5th image recognition unit 96 nth image recognition unit 101 1st correspondence information acquisition unit 102 2 Correspondence information acquisition unit 103 3rd correspondence information acquisition unit 104 4th correspondence information acquisition unit 105 5th correspondence information acquisition unit 106 nth correspondence information acquisition unit 910 PACS
911 Diagnostic support device 921 1st inspection device 922 2nd inspection device 923 2nd inspection device 926 Network 930 Medical business support device A1, A2 range ELx, EL1, EL2, EL3, EL4, ELy Oxygen saturation contour line P1 No. 1 Illumination period P2 2nd illumination period FL frame L1 1st illumination light L2, L2a, L2b, L2c, L2d 2nd illumination light Q1, Q2, Q3, Q4, Q5 Emission cycle SP1, SP2, SP3, SP4 2nd illumination light Spectral spectrum IM1 1st recognition target image IM2a 5th recognition target image IM2b 3rd recognition target image IM2c 2nd recognition target image t1 to t6 Time ST110 to ST200 Step

Claims

An image processing device that performs image recognition processing based on an image obtained by photographing an observation target with an endoscope.
Equipped with an image processor
The image processor is
Acquire multiple types of recognition target images based on the above images,
Control is performed to continuously display at least one type of the recognition target image among the plurality of types of the recognition target images on the display.
The image recognition process for a plurality of types of the recognition target image is performed in parallel for each type of the recognition target image.
The recognition processing result obtained by the image recognition processing is acquired, and
Image processing that controls the operation of recording when recording a moving image of at least one type of the recognition target image among a plurality of types of the recognition target images based on the recognition processing results of all the acquired types of the recognition target images. Device.
The recognition processing result includes information on whether or not the recognition target image satisfies or does not satisfy a preset condition.
The image processor according to claim 1, wherein the image processor starts or continues the recording when one or more of the recognition processing results of all kinds include the information that the condition is satisfied.
The second aspect of claim 2, wherein the image processor includes information that all of the recognition processing results of all kinds do not satisfy the conditions, and stops the recording when the recording is continued. Image processing device.
The image processor according to claim 3, wherein when the moving image is recorded, the image processor attaches information regarding the corresponding condition of start, continuation, or stop of the recording to the moving image.
The image processor previously acquires correspondence information in which the observation target satisfying the above conditions and the recognition target image obtained by photographing the observation target satisfying the conditions are associated with each other.
The image processing apparatus according to any one of claims 2 to 4, wherein the image recognition process is performed on the newly acquired image to be recognized based on the corresponding information.
The image processor acquires the correspondence information for each type of the recognition target image, and obtains the correspondence information for each type of the recognition target image.
The image processing apparatus according to claim 5, wherein the image recognition process for the newly acquired image to be recognized is performed based on the corresponding information of the corresponding type.
The condition is any one of claims 2 to 6, wherein the image processor has detected an object other than a specific part or a living body, or it has been determined that the image processor includes a region in a specific state. The image processing unit described.
The plurality of types of the recognition target images include at least a first recognition target image and a second recognition target image of different types from each other.
The image processor performs a first image recognition process on the first recognition target image, and performs a second image recognition process on the second recognition target image, which is different from the first image recognition process. The image processing apparatus according to any one of claims 1 to 7.
The image processing apparatus according to claim 8, wherein the first image recognition process is performed for detecting a specific part or an object other than a living body in the first recognition target image.
The image processing apparatus according to claim 8 or 9, wherein the second image recognition process is performed for determining that the second image to be recognized includes a region in a specific state.
The recognition target image is generated by performing enhancement processing on the image, and the recognition target image is generated.
The image processor distinguishes the types of the recognition target images according to the presence or absence or type of the enhancement processing, and any one of claims 1 to 10 for acquiring the distinguished recognition target images as one type of the recognition target image. The image processing apparatus according to item 1.
The types of the enhancement processing are color expansion processing and / or structure enhancement processing.
The image processing apparatus according to claim 11, wherein the image processor acquires the recognition target image generated by performing different types of enhancement processing as one type of the recognition target image.
The image processing apparatus according to any one of claims 1 to 12.
An endoscope system including a light source unit that emits illumination light to irradiate the observation target.
The image processor acquires the image obtained by photographing the observation target illuminated by each of a plurality of illumination lights having different spectral spectra from each other emitted by the light source unit as one kind of recognition target image. The endoscope system according to claim 13.
The internal vision according to claim 13, wherein the image processor acquires the image obtained by photographing the observation target illuminated by the white illumination light emitted by the light source unit as one kind of the recognition target image. Mirror system.
The image processor acquires the image obtained by photographing the observation target illuminated by the illumination light including the narrow band light of the preset wavelength band emitted by the light source unit as one kind of the recognition target image. 13. The endoscope system according to claim 13.
The endoscope system according to any one of claims 13 to 16, wherein the light source unit repeatedly emits light of a plurality of illumination lights having different spectral spectra in a preset order.
The light source unit emits first illumination light and second illumination light having different spectral spectra from each other.
The first illumination light is emitted by the first emission pattern during the first illumination period, the second illumination light is emitted by the second emission pattern during the second illumination period, and the first illumination light and the second illumination are emitted. A processor for light sources that switches between light and
An image that outputs a first image signal obtained by photographing an observation object illuminated by the first illumination light and a second image signal obtained by photographing the observation object illuminated by the second illumination light. Equipped with a sensor
The image processor is
The first image recognition process is performed on the first recognition target image based on the first image signal, and the second image recognition process is performed on the second recognition target image based on the second image signal.
The endoscope according to claim 13, wherein claim 8 for controlling the operation of recording based on the first image recognition result by the first image recognition process and the second image recognition result by the second image recognition process is cited. Mirror system.
It is an operation method of an image processing device that performs image recognition processing based on an image obtained by photographing an observation target with an endoscope.
An image acquisition step for acquiring a plurality of types of recognition target images based on the image, and
A display control step for controlling the continuous display of at least one type of the recognition target image among the plurality of types of the recognition target images on the display, and a display control step.
An image recognition processing step in which the image recognition processing for a plurality of types of the recognition target images is performed in parallel for each type of the recognition target image, and
A recognition processing result acquisition step of acquiring the recognition processing result obtained by the image recognition processing for each type of the recognition target image, and
Recording that controls the operation of the recording based on the recognition processing results of all the acquired recognition target images when recording a moving image of at least one of the recognition target images among the plurality of types of the recognition target images. A method of operating an image processing device including a control step.
In a program for an image processing device installed in an image processing device that performs image recognition processing based on an image obtained by photographing an observation target with an endoscope.
On the computer
An image acquisition function that acquires multiple types of recognition target images based on the image, and
A display control function that controls the continuous display of at least one of the recognition target images among a plurality of types of the recognition target images on the display, and a display control function.
An image recognition processing function that performs the image recognition processing for a plurality of types of the recognition target images in parallel for each type of the recognition target image.
A recognition processing result acquisition function for acquiring the recognition processing result obtained by the image recognition processing for each type of the recognition target image, and
Recording that controls the operation of the recording based on the recognition processing results of all the acquired recognition target images when recording a moving image of at least one of the recognition target images among the plurality of types of the recognition target images. A program for image processing equipment to realize control functions.