JP7130038B2 - Endoscope image processing device, operation method of endoscope image processing device, endoscope image processing program, and storage medium - Google Patents

Description

The present invention relates to an endoscopic image processing apparatus, an operation method of the endoscopic image processing apparatus, an endoscopic image processing program, and a storage medium , and more particularly, to an endoscopic image for performing image recognition on an endoscopic image. The present invention relates to a processing device, an operation method of an endoscope image processing device, an endoscope image processing program, and a storage medium .

Techniques for detecting a lesion by image recognition from an endoscopic image captured by an endoscopic device and classifying lesions by type are known (for example, Patent Literatures 1 and 2, etc.).

Machine learning of images such as deep learning (Non-Patent Document 1) is widely used for image recognition. Machine learning enables image recognition such as classification and detection by learning a large amount of images according to the problem.

JP-A-2000-155840 WO2017/175282

A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012

In machine learning, if there is a bias in image quality in a group of images for learning, the bias in image quality is also learned. For this reason, there is a problem that the accuracy is lowered when an image whose image quality deviates from the image quality deviation of the learned image group is image-recognized. In an examination using an endoscope, generally, imaging and examination are performed with image quality according to the preference of the operator. Therefore, if the image quality of the captured image differs from the image quality of the learned image group, the accuracy of image recognition decreases. There is a problem. On the other hand, it is also difficult to learn about all image qualities.

The present invention has been made in view of such circumstances, and provides an endoscopic image processing apparatus capable of image recognition without considering differences in image quality, an operation method of the endoscopic image processing apparatus, and an endoscopic image processing program. and to provide a storage medium .

Means for solving the above problems are as follows.

(1) an endoscopic image acquisition unit that acquires an endoscopic image captured by an endoscopic device; an image conversion processing unit that performs processing for converting the endoscopic image into an image of standard image quality; and an image recognition unit that performs image recognition on the converted image.

According to this aspect, the acquired endoscopic image is converted into an image of standard image quality, and image recognition is performed. As a result, image recognition can be performed without considering the difference in image quality of captured endoscopic images.

(2) The endoscope according to (1) above, wherein the image conversion processing unit converts an endoscopic image into an image of standard image quality based on an image quality adjustment value set in the endoscope apparatus. Image processing device.

According to this aspect, the endoscopic image is converted into the standard image quality image based on the image quality adjustment value set in the endoscope apparatus. Accordingly, image recognition can be performed without considering the difference in image quality set in the endoscope apparatus.

(3) The image conversion processing unit converts the endoscopic image into a standard image quality image based on the image quality adjustment value set in the endoscopic device when the endoscopic image is output to the display device. The endoscopic image processing apparatus according to (1) above, wherein

According to this aspect, when the endoscopic image is output to the display device, the endoscopic image is converted into the image of the standard image quality based on the image quality adjustment value set in the endoscopic device. When outputting an endoscopic image to a display device, the image quality may be adjusted according to the preference of the operator. According to this aspect, even if the image quality is adjusted, the image can be recognized with high accuracy.

(4) The image conversion processing unit converts the endoscopic image into a standard image quality image based on the image quality adjustment value set in the endoscope apparatus when the endoscopic image is saved as a still image. The endoscopic image processing apparatus according to (1) above, wherein

According to this aspect, when the endoscopic image is saved as a still image, the endoscopic image is converted into the standard image quality image based on the image quality adjustment value set in the endoscope apparatus. When saving an endoscopic image as a still image, the image quality may be adjusted before saving. According to this aspect, even if the image quality is adjusted, the image can be recognized with high accuracy.

(5) The endoscope according to (1) above, wherein the image conversion processing unit converts the endoscopic image into an image of standard image quality based on a color tone adjustment value set in the endoscope device. Image processing device.

According to this aspect, when the color tone is adjusted by the endoscope apparatus, the endoscope image is converted into the image of the standard image quality based on the adjustment value.

(6) The image conversion processing unit performs processing for converting the endoscopic image into an image of standard image quality based on the adjustment value for image quality change due to the amount of light set in the endoscope apparatus, as described in (1) above. Endoscope image processing device.

According to this aspect, when the endoscope apparatus adjusts the image quality change due to the amount of light, the endoscope image is converted into the image of the standard image quality based on the adjustment value.

(7) The image conversion processing unit processes the endoscopic image with a numerical value set for each endoscope device in which the endoscopic image was captured, and converts the endoscopic image into an image of standard image quality. The endoscopic image processing apparatus according to (1) above, wherein

According to this aspect, the endoscopic image is processed with the numerical value set for each endoscopic device, and the endoscopic image is converted into the image of the standard image quality. As a result, even when performing image recognition on endoscopic images captured by different endoscopic devices, image recognition can be performed without considering differences in image quality.

(8) The endoscopic image processing apparatus according to any one of (1) to (7) above, wherein the image recognition unit is composed of a convolutional neural network trained with images converted to the standard image quality.

According to this aspect, the image recognition unit is configured with a convolutional neural network trained with images converted to the standard image quality.

(9) The endoscope according to any one of (1) to (7) above, wherein the image recognition performed by the image recognition unit includes a process of detecting an attention area and/or a process of classifying a recognition target. Image processing device.

According to this aspect, the process of detecting an attention area such as a lesion and/or the process of classifying a recognition target such as a lesion are performed by image recognition.

(10) a step of acquiring an endoscopic image captured by an endoscopic device; a step of converting the endoscopic image into an image of standard image quality; A method of endoscopic image processing, comprising: performing recognition.

According to this aspect, the acquired endoscopic image is converted into an image of standard image quality, and image recognition is performed. As a result, image recognition can be performed without considering the difference in image quality of captured endoscopic images.

(11) A function of acquiring an endoscopic image captured by an endoscopic device, a function of converting the endoscopic image into an image of standard image quality, and an image processing function for the image converted to the standard image quality. An endoscope image processing program that allows a computer to realize a recognition function.

According to this aspect, the acquired endoscopic image is converted into an image of standard image quality, and image recognition is performed. As a result, image recognition can be performed without considering the difference in image quality of captured endoscopic images.

According to the present invention, image recognition can be performed without considering differences in image quality.

1 is a block diagram showing the overall system configuration of an endoscope system including an endoscope apparatus and an endoscope image processing apparatus; FIG. Conceptual block configuration diagram of an endoscope device External view as an example of the endoscope apparatus shown in FIG. Schematic representation of blood vessels on the mucosal surface of living tissue A diagram showing a display example of an observation image when narrow-band light is used as illumination light. A diagram showing a display example of an observation image when white light is used as illumination light. Flowchart showing a procedure of processing from acquisition of an image signal to output to a display device FIG. 2 is a block diagram showing the hardware configuration of the endoscope image processing apparatus; Block diagram showing the functions of the endoscope image processing device Diagram showing an example of recognition result display Flowchart showing the procedure of image recognition processing performed by the endoscope image processing apparatus Flowchart showing a procedure of processing from acquisition of an image signal to output to a display device Flowchart showing the procedure of image recognition processing performed by the endoscope image processing apparatus Flowchart showing a procedure of processing from acquisition of an image signal to output to a display device Flowchart showing the procedure of image recognition processing performed by the endoscope image processing apparatus

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

◆◆ First embodiment ◆◆
FIG. 1 is a block diagram showing the overall system configuration of an endoscope system including an endoscope apparatus and an endoscope image processing apparatus.

As shown in the figure, the endoscope system 1 includes an endoscope device 10, an endoscope image processing device 100 that performs image recognition on an endoscope image captured by the endoscope device 10, Prepare. The endoscope apparatus 10 and the endoscope image processing apparatus 100 are connected via a network 2 such as a LAN (Local Area Network).

《Endoscope device》
FIG. 2 is a conceptual block configuration diagram of the endoscope apparatus. 3 is an external view as an example of the endoscope apparatus shown in FIG. 2. FIG.

The endoscope apparatus 10 of the present embodiment is configured as an endoscope apparatus capable of narrow-band light observation using narrow-band light in addition to normal observation using white light.

As shown in FIGS. 2 and 3, the endoscope apparatus 10 includes an endoscope 11 and an endoscope control device 13 to which the endoscope 11 is connected. The endoscope 11 is an electronic endoscope (flexible endoscope), and is detachably connected to the endoscope control device 13 via a connection portion 27 . A display device 21 and an input device 23 are connected to the endoscope control device 13 . The display device 21 is composed of, for example, a liquid crystal monitor. The input device 23 is composed of, for example, a keyboard, a mouse, and the like.

The endoscope 11 includes, as shown in FIG. A connecting portion 27 is provided.

The endoscope insertion section 15 is composed of a flexible flexible section 31 , a bending section 33 and a distal end rigid section 35 in order from the operation section 25 side.

The distal rigid portion 35 is provided with a pair of illumination windows 37A and 37B, an observation window 38, and the like on its distal end surface. An image sensor 17 is provided inside the observation window 38 via a photographing optical system 39 . The image sensor 17 is composed of a color image sensor having a predetermined color filter array (for example, Bayer array), and is composed of a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal-Oxide Semiconductor) image sensor, or the like. .

The bending portion 33 is provided between the flexible portion 31 and the distal end hard portion 35 and bends according to the operation of the angle knob 41 provided on the operating portion 25 . By manipulating the bending portion 33, the orientation of the distal end rigid portion 35, that is, the observation direction can be changed.

In addition to the angle knob 41, the operation unit 25 includes various operation members 43 such as a button for switching observation modes (observation mode switching button) and a button for capturing a still image (shutter button). .

The connecting portion 27 is composed of a flexible cord, and has a light guide connector 19A and a video connector 19B for connecting to the endoscope control device 13 at its tip.

The endoscope control device 13 includes a light source device 47 that generates illumination light and a processor device 49 that processes image signals from the image sensor 17. The endoscope control device 13 is connected to the endoscope 11 via a light guide connector 19A and a video connector 19B. Connected. The display device 21 and the input device 23 are also connected to the processor device 49 .

The processor device 49 processes imaging signals transmitted from the endoscope 11 based on instructions from the operation unit 25 and the input device 23 of the endoscope 11, and generates image data (endoscopic image data). and output to the display device 21.

The light source device 47 includes a first laser light source LD1, which is a semiconductor light emitting element with a central wavelength of 445 nm, and a second laser light source LD2, which is a semiconductor light emitting element with a central wavelength of 405 nm, as light emitting sources. The light emission of the first laser light source LD1 and the light emission of the second laser light source LD2 are individually controlled by the light source control section 51 .

As the first laser light source LD1 and the second laser light source LD2, a broad area type InGaN-based laser diode can be used, and an InGaNAs-based laser diode, a GaNAs-based laser diode, or the like can also be used. Moreover, the structure using light-emitting bodies, such as a light emitting diode, may be sufficient as said light source.

The laser beams emitted from the first laser light source LD1 and the second laser light source LD2 are respectively input to optical fibers through a condensing lens (not shown). through the light guide connector 19A.

The blue laser light with a center wavelength of 445 nm and the violet laser light with a center wavelength of 405 nm supplied to the light guide connector 19A are combined and guided to the hard tip portion 35 by the optical fibers 57A and 57B, respectively. be. The blue laser light excites the phosphor 59, which is a wavelength conversion member, arranged at the light emitting end of the optical fibers 57A and 57B of the tip rigid portion 35 to generate fluorescence. A portion of the blue laser light is transmitted through the phosphor 59 as it is, and is emitted as white illumination light together with the fluorescence. On the other hand, the violet laser light passes through the phosphor 59 without strongly exciting it, and is emitted as illumination light with a narrow band wavelength (narrow band light).

In addition, the "white light" as used in this specification is not limited to strictly including all wavelength components of visible light, and red (Red, R), green (Green, G), blue (Blue, B) For example, light including wavelength components from green to red and light including wavelength components from blue to green are broadly included.

The phosphor 59 includes a plurality of phosphors (for example, a YAG phosphor or a phosphor containing BAM (BaMgAl10O17), etc.) that absorbs part of the blue laser beam and emits green to yellow excited light. be done. As a result, the green to yellow excitation light that uses the blue laser light as excitation light and the blue laser light that is transmitted without being absorbed by the phosphor 59 are combined to produce white (pseudo-white) illumination light.

The light source control unit 51 individually controls the amount of blue laser light emitted from the first laser light source LD1 and the amount of violet laser light emitted from the second laser light source LD2, and controls illumination light (white laser light) generated by the blue laser light. light) and illumination light (narrowband light) generated by the violet laser light are set to a light amount ratio suitable for observation.

Illumination light set to a light amount ratio suitable for observation by the light source control unit 51 is irradiated toward the observation region of the subject from a pair of illumination windows 37A and 37B provided in the distal end rigid portion 35 . Then, the area irradiated with this illumination light is imaged by the image sensor 17 through the observation window 38 and the imaging optical system 39 .

An image signal output from the image sensor 17 is transmitted to an ADC (Analog to Digital Converter) 63 through a scope cable 61, converted into a digital signal by the ADC 63, and endoscope controlled by the processor device 49. It is input to the part 65 .

The endoscope control section 65 sends the input digital image signal to the image processing section 67 . The image processing unit 67 performs required signal processing on the digital image signal to generate image data (endoscopic image data) of an endoscopic image, which is an observed image. This image data is data having intensity values (luminance values) of R, G, and B in units of pixels.

The endoscope control section 65 outputs the generated endoscope image data to the display device 21 . In addition, the endoscope control section 65 causes the storage section 69 to store the endoscope image data as necessary. The endoscope control unit 65 also outputs endoscope image data to the endoscope image processing apparatus 100 as necessary.

In this embodiment, the storage unit 69 is provided in the processor device 49, but the storage unit 69 may be connected to the processor device 49 via a network, or may be connected to the network. It may be provided in a server or the like.

FIG. 4 is a diagram schematically showing blood vessels on the mucosal surface layer of living tissue. As shown in the figure, the mucosal surface layer of the biological tissue is formed from blood vessels B1 in the deep mucosal layer to the mucosal surface layer by capillaries B2 such as a resinous vascular network. have been reported to appear in the fine structure of Therefore, in endoscopic examination, attempts have been made to observe capillaries in the surface layer of the mucous membrane with image enhancement for early detection of microlesions and diagnosis of the range of lesions.

When illumination light is incident on a living tissue, the incident light diffusely propagates through the living tissue, but the absorption and scattering characteristics of the living tissue have wavelength dependence, and the shorter the wavelength, the stronger the scattering characteristics. Tend. In other words, the depth of penetration of the light changes depending on the wavelength of the illumination light. Therefore, blood vessel information from capillaries on the surface of the mucous membrane can be obtained in the wavelength region λa of the illumination light around 400 nm, and blood vessel information including deeper blood vessels can be obtained in the wavelength region λb around 500 nm. . Therefore, a light source with a center wavelength of 360 to 800 nm, preferably 365 to 515 nm, is used for observing blood vessels in living tissue. of light sources are used. Also, fine patterns on the surface of the mucous membrane of living tissue can be highlighted in the above wavelength range in the same manner as capillaries.

FIG. 5 is a diagram showing a display example of an observation image when narrowband light is used as illumination light. As shown in the figure, when narrow-band light containing many visible short wavelength components is used as illumination light, an image in which fine capillaries on the surface of the mucous membrane and fine patterns on the surface of the mucous membrane are clearly displayed can be obtained.

FIG. 6 is a diagram showing a display example of an observation image when white light is used as illumination light. As shown in the figure, when white light is used as the illumination light, an overall image of the diseased area is obtained in which blood vessels in the relatively deep layer of the mucous membrane are projected.

That is, in an observation image obtained by simultaneously irradiating white light and narrow-band light, fine blood vessels on the mucosal surface layer of living tissue and fine patterns on the mucosal surface are emphasized, and the properties and observation position of the affected area can be easily identified. This is an observation image that facilitates endoscopic diagnosis.

Therefore, in the endoscope apparatus 10 of the present embodiment, the amounts of white light (blue laser light and phosphor emission) and narrow-band light (violet laser light) emitted from the distal end rigid portion 35 are independently controlled. It is made controllable, and reflected light from both illumination lights is included in one frame of captured image.

The ratio of the amount of emitted light between white light and narrowband light is set to an appropriate ratio such as [white light:narrowband light=1:4 to 1:8]. As a result, while the observation site is clearly projected with white light, fine patterns of superficial blood vessels and mucosal surfaces can be emphasized with narrow-band light, and an image suitable for observing fine blood vessel structures and pits can be obtained.

Next, a case of changing the image quality of the captured endoscopic image and displaying it on the display device 21 will be described.

When an endoscopic image is displayed on the display device 21, the quality of the displayed image, particularly the color tone, differs depending on the operator.

Therefore, in the endoscope apparatus 10 of the present embodiment, the color tone of the endoscope image data is corrected and output to the display device 21 as necessary.

The color tone correction is performed by the image processing section 67 under the control of the endoscope control section 65 based on the instruction from the input device 23 . The endoscope apparatus 10 of the present embodiment is implemented as follows.

The endoscope control unit 65 sets a correction matrix for matrix-correcting each intensity value (luminance value) of R, G, and B based on the color tone adjustment value input from the input device 23 . The image processing unit 67 performs matrix correction on the endoscope image data based on the set correction matrix to correct the color tone.

R0, G0, B0 are the intensity values of R, G, and B of each pixel of the endoscopic image data before color tone correction, A is the correction matrix, R, G of each pixel of the endoscopic image data after color tone correction, Assuming that the intensity values of B are R1, G1, and B1, the intensity values R1, G1, and B1 of R, G, and B of each pixel of the endoscopic image data after color tone correction are expressed by the following equation (1).

The coefficients aij of the correction matrix A are set based on the color tone adjustment values. The endoscope control unit 65 acquires information on the color tone adjustment values via the input device 23 and sets the correction matrix A. FIG.

The image processing unit 67 performs matrix correction on the intensity values of R, G, and B of each pixel of the endoscopic image data using the set correction matrix A, thereby correcting the color tone of the endoscopic image data. Then, the corrected endoscope image data is output to the display device 21 .

FIG. 7 is a flowchart showing a procedure of processing from acquisition of an image signal to output to a display device.

An image signal (digital image signal) output from the endoscope 11 is acquired (step S1), and image data of an endoscopic image (endoscopic image data) is generated from the acquired image signal (step S2). ). Then, the generated endoscope image data is subjected to color tone correction based on the adjustment value (step S3), and the endoscope image data after color tone correction is output to the display device 21 (step S4).

By performing color tone correction in this manner, the endoscope image can be displayed on the display device 21 in a color tone that suits the operator's preference.

When storing the endoscope image in the storage unit 69, the storage unit 69 stores the endoscope image data after the color tone correction. Similarly, when outputting an endoscopic image to the endoscopic image processing apparatus 100 , endoscopic image data after color tone correction is output to the endoscopic image processing apparatus 100 .

《Endoscope image processing device》
The endoscopic image processing apparatus 100 acquires endoscopic image data from the endoscopic apparatus 10, detects lesions included in the image by image recognition, classifies the detected lesions by type, and notifies them. I do.

For image recognition processing, for example, a convolutional neural network (CNN) is used.

A lesion is classified into a plurality of categories according to, for example, the NICE classification or the JNET classification. "NICE" is an abbreviation for NBI International Colorectal Endoscopic. "JNET" is an abbreviation for "the Japan NBI Expert Team." "NBI" (registered trademark) is an abbreviation for Narrow band imaging (registered trademark).

The NICE classification is a non-magnified NBI classification, and is classified into Type 1, Type 2, and Type 3 for each of three items: lesion color, microvessels, and surface pattern. be. Type 1 is a hyperplastic lesion, Type 2 is an adenoma to intramucosal carcinoma, and Type 3 is a diagnostic index for SM (submucosa) deep invasive cancer. The JNET classification is a classification of NBI magnified endoscopy findings for colon tumors. The JNET classification is classified into Type1, Type2A, Type2B, and Type3 for each item of "vessel pattern" and "surface pattern."

In the image recognition processing, instead of detailed classification such as NICE classification, or in combination with this, recognition of two classifications, simply "cancerous" or "non-cancerous" is performed. good too.

FIG. 8 is a block diagram showing the hardware configuration of the endoscope image processing apparatus.

The endoscopic image processing apparatus 100 is configured by a computer such as a personal computer or a workstation, for example. This computer includes a CPU (Central Processing Unit) 102, a RAM (Random Access Memory) 104, a ROM (read-only memory) 106, a communication interface (interface, I/F) 108, and an input/output interface 110. It is connected to network 2 via interface 108 . A storage device 112 , an input device 114 and a display device 116 are also connected via the input/output interface 110 . The storage device 112 is, for example, a storage device such as a hard disk drive (HDD). The input device 114 is composed of input devices such as a keyboard and a mouse. The display device 116 is composed of a display device such as a liquid crystal monitor.

FIG. 9 is a block diagram showing functions of the endoscope image processing apparatus.

As shown in the figure, the endoscopic image processing apparatus 100 executes a predetermined control program (endoscopic image processing program) to perform an endoscopic image acquisition unit 120, an image conversion processing unit 122, an image recognition unit 120, and an image recognition unit 120. It functions as a unit 124 and a display control unit 126 . The programs are stored in ROM 106 or storage device 112 .

The endoscope image acquisition unit 120 acquires endoscope image data from the endoscope apparatus 10 . The endoscope image data may be either moving image data or still image data.

The image conversion processing unit 122 performs processing for converting the endoscopic image acquired by the endoscopic image acquiring unit 120 into an image of standard image quality. Here, in the present embodiment, “reference image quality” refers to image quality before color tone correction performed in the endoscope apparatus 10 .

As described above, in the endoscope apparatus 10 of the present embodiment, color tone correction of the output image is performed as necessary. On the other hand, as will be described later, image recognition uses a model learned by machine learning (learned model). In machine learning, if there is a bias in image quality in a group of images for learning, the bias in image quality is also learned. As a result, the accuracy is lowered when an image whose image quality deviates from the image quality bias of the learned image group is image-recognized. When the color tone is corrected, the image quality deviation of the learned image group is deviated, and the accuracy of image recognition is lowered.

Therefore, in the present embodiment, image processing is performed on endoscope image data acquired from the endoscope apparatus 10, and the image quality is converted into image quality before color tone correction. Specifically, the acquired endoscopic image data is subjected to matrix correction using a correction matrix B that is an inverse matrix A ⁻¹ of the correction matrix A at the time of color tone correction, and converted into image data of standard image quality. .

Now, let R1, G1, B1 be the intensity values of R, G, and B of each pixel of the acquired endoscopic image data, B be the correction matrix, and R, G, and B of each pixel of the endoscopic image data after conversion. R2, G2, and B2 are the intensity values of R, G, and B of each pixel of the endoscopic image data after conversion, and are expressed by the following equation (2).

The image conversion processing unit 122 acquires the information of the correction matrix A for color tone correction from the endoscope apparatus 10, and sets the correction matrix B which is the inverse matrix thereof. Then, using the set correction matrix B, the R, G, and B intensity values of each pixel of the acquired endoscopic image data are matrix-corrected to convert the image quality. That is, the image quality is converted to that before color tone correction.

The image recognition unit 124 performs image recognition processing on the endoscopic image that has been converted into the standard image quality image by the image conversion processing unit 122 . In this embodiment, a process of detecting a lesion (region of interest) from an acquired endoscopic image and classifying the detected lesion according to the NIEC classification, the JNET classification, or the like is performed. The image recognition unit 124 is configured using a learned model learned by machine learning. This model is composed of, for example, a convolutional neural network (CNN). At the time of model generation, learning is performed using a group of endoscopic images that have been converted to standard image quality.

The display control unit 126 acquires the recognition result by the image recognition unit 124 and causes the display device 116 to display the recognition result in a predetermined display mode.

FIG. 10 is a diagram showing an example of display of recognition results.

As shown in the figure, when a lesion is detected, the recognition result is displayed superimposed on the endoscopic image that has undergone image recognition. The recognition result is displayed, for example, in the form of enclosing the detected lesion area with a frame. In addition, lesion classification results are displayed near the frame.

FIG. 11 is a flow chart showing the procedure of image recognition processing performed by the endoscope image processing apparatus.

First, endoscope image data is acquired from the endoscope apparatus 10 by the endoscope image acquisition unit 120 (step S11).

Next, based on the color tone adjustment value, a process of converting the image quality to be recognized into the standard image quality is performed (step S12). This processing is performed by the image conversion processing unit 122 . The image conversion processing unit 122 converts the image quality of the acquired endoscopic image data by performing matrix correction. The correction matrix B used at this time is obtained from the correction matrix A for color tone correction when the endoscopic image data to be recognized is generated. That is, it is obtained as an inverse matrix of the correction matrix A.

Next, image recognition is performed by the image recognition unit 124 on the endoscopic image after conversion to the standard image quality (step S13), and the recognition result is output to the display device 21 (step S14).

As described above, in the endoscopic image processing apparatus 100 of the present embodiment, when recognizing an endoscopic image, the image quality of the endoscopic image to be recognized is converted into the standard image quality, and the endoscope image after conversion is converted to the standard image quality. Image recognition is performed on the image. As a result, in the endoscope apparatus 10, image recognition can be performed with high accuracy even when original color tone correction is performed. That is, image recognition can be performed without considering differences in image quality.

◆◆ Second embodiment ◆◆
An endoscope device may automatically correct image quality. For example, there is a case where a change in image quality caused by a change in the amount of illumination light is automatically corrected. In the present embodiment, when the image quality of the endoscopic image to be recognized has been corrected, the image quality is converted to the standard image quality based on the correction information, and the converted endoscopic image is converted to the image quality. perform recognition.

《Auto Correction of Image Quality》
Here, the automatic correction of image quality based on changes in the amount of light will be described. The hardware configuration of the endoscope apparatus is the same as that of the first embodiment. Therefore, description of the hardware configuration is omitted.

<Light intensity control>
First, control of the amount of illumination light will be described. In the endoscope apparatus of the present embodiment, the light intensity of the illumination light, that is, the emission of the first laser light source LD1 and the second laser light source LD2, depends on the imaging distance (the distance from the imaging surface of the image sensor to the subject). The amount of light is automatically controlled. The endoscope control unit 65 controls the amount of light emitted from the first laser light source LD1 and the amount of light emitted from the second laser light source LD2 so as to achieve an optimum light amount ratio according to the photographing distance. Information on the shooting distance is acquired using, for example, autofocus information. In addition, a method of estimating based on exposure information (shutter speed, aperture value, brightness (EV value (Exposure Value)), etc.) and illumination light amount information set by the light source control unit 51 may be adopted. can also The light amount ratio is set by, for example, preparing in advance a table showing the relationship between the photographing distance and the optimum light amount ratio, and referring to this table.

The endoscope control unit 65 refers to the table and adjusts the amount of light emitted from the second laser light source LD2 to the amount of light emitted from the first laser light source LD1 so that the capillaries and the like on the surface of the mucous membrane are emphasized as the photographing distance is shorter. set relatively larger than Further, as the photographing distance becomes longer, the amount of light emitted from the first laser light source LD1 is made relatively higher than the amount of light emitted from the second laser light source LD2 so that the illumination light has a high illuminance in order to ensure the brightness of the distant view. set large.

<Image Correction>
As described above, the emitted light from the first laser light source LD1 is finally applied to the subject as white light, and the emitted light from the second laser light source LD2 is applied to the subject as narrow-band light. Therefore, when the light amount ratio of the emitted light of the first laser light source LD1 and the second laser light source LD2 is changed, the color of the illumination light is changed. For example, when the light intensity of the second laser light source LD2 increases with respect to the light intensity of the first laser light source LD1, the illumination light becomes bluish. As a result, the color tone of the endoscopic image, which is the observation image, changes according to the light amount ratio.

Therefore, the color tone of the endoscopic image is corrected according to the color of the illumination light that changes according to the set light amount ratio, so that the endoscopic image is not affected by the color change of the illumination light.

The endoscope control unit 65 corrects the endoscope image data based on the set light amount ratio information so that the color tone of the image does not change. For correction, for example, a color tone correction table is prepared in advance for each light amount ratio, and this color tone correction table is referred to.

The color tone correction table can be expressed as a correction matrix that performs matrix correction of R, G, and B intensity values for each pixel of the endoscopic image, for example.

R3, G3, and B3 are the intensity values of R, G, and B of each pixel of the endoscopic image data obtained by imaging, C is the correction matrix, and R and G of each pixel of the endoscopic image data after color tone correction. , and B are R4, G4, and B4, the intensity values R4, G4, and B4 of R, G, and B of each pixel of the endoscopic image data after color tone correction are expressed by the following equation (3). .

The coefficient cij of the correction matrix C is set so as to cancel the change in color of the illumination light due to the change in the light amount ratio, and a plurality of types of correction matrices C corresponding to a plurality of types of light amount ratios are prepared in advance.

The image processing unit 67 performs matrix correction on the intensity values of R, G, and B of each pixel of the endoscopic image data using the set correction matrix C, thereby correcting the color tone of the endoscopic image data. Then, the corrected endoscope image data is output to the display device 21 .

FIG. 12 is a flowchart showing a procedure of processing from acquisition of an image signal to output to a display device.

An image signal output from the endoscope 11 is acquired (step S21), and image data of an endoscopic image (endoscopic image data) is generated from the acquired image signal (step S22). Then, the generated endoscopic image data is subjected to color tone correction based on changes in the amount of light (step S23), and the endoscopic image data after color tone correction is output to the display device 21 (step S24). When outputting an endoscopic image to the endoscopic image processing apparatus 100 , endoscopic image data after color tone correction is output to the endoscopic image processing apparatus 100 .

《Image Recognition》
When the endoscopic image to be recognized has been automatically color-corrected based on changes in the amount of light, the endoscopic image processing apparatus 100 converts the image quality to the reference image quality based on the correction information, and after conversion, image recognition is performed on the endoscopic image.

FIG. 13 is a flowchart showing the procedure of image recognition processing performed by the endoscope image processing apparatus.

First, endoscope image data is acquired from the endoscope apparatus 10 by the endoscope image acquisition unit 120 (step S31). Next, based on the correction information of the color tone correction based on the change in the amount of light, a process of converting the image quality to be recognized into the standard image quality is performed (step S32). The image conversion processing unit 122 converts the image quality of the acquired endoscopic image data by performing matrix correction. The correction matrix used at this time is obtained from the correction matrix C obtained when the endoscopic image data to be recognized is subjected to color tone correction based on changes in the amount of light. That is, it is obtained as an inverse matrix C ⁻¹ of the correction matrix C.

Next, image recognition is performed by the image recognition unit 124 on the endoscopic image converted to the standard image quality (step S33), and the recognition result is output to the display device 21 (step S34).

In this manner, by converting the image into the image of the standard image quality and recognizing the image, even if the color tone is automatically corrected by the endoscope apparatus 10, the image can be accurately recognized. That is, image recognition can be performed without considering differences in image quality.

◆◆ Third Embodiment ◆◆
When the color tone of the output image is adjusted by the operator and the color tone is automatically corrected according to the change in the amount of light, the following processing is performed.

<Processing with an endoscope>
FIG. 14 is a flowchart showing a procedure of processing from acquisition of an image signal to output to a display device.

An image signal output from the endoscope 11 is acquired (step S41), and image data of an endoscopic image (endoscopic image data) is generated from the acquired image signal (step S42). First, color tone correction is performed on the generated endoscopic image data based on changes in the amount of light (step S43). Next, color tone correction is performed based on the color tone adjustment value (step S44). After that, the endoscope image data after the color tone correction is output to the display device 21 (step S45). When outputting an endoscopic image to the endoscopic image processing apparatus 100 , endoscopic image data after color tone correction is output to the endoscopic image processing apparatus 100 .

<<Processing by Endoscope Image Processing Device (Endoscope Image Processing Method)>>
FIG. 15 is a flow chart showing the procedure of image recognition processing performed by the endoscope image processing apparatus.

First, endoscope image data is acquired from the endoscope apparatus 10 by the endoscope image acquisition unit 120 (step S51).

Next, a process of converting the image quality of the object to be recognized is performed based on the correction information for the color tone correction based on the change in the amount of light (step S52). Next, a process of converting the image quality of the recognition target is performed based on the correction information for color tone correction based on the color tone adjustment value (step S53). By performing this two-step conversion process, the image quality of the recognition object is converted into the standard image quality.

Next, image recognition is performed by the image recognition unit 124 on the endoscopic image after conversion to the standard image quality (step S54). Then, the recognition result is output to the display device 21 (step S55).

In this way, when a plurality of image quality corrections are applied to an endoscopic image to be recognized, conversion processing corresponding to each image quality correction is performed to convert the image into a standard image quality image. By converting the image into the standard image quality image and recognizing the image, the image recognition can be performed with high accuracy.

◆◆Modified example◆◆
《Image quality correction performed in the endoscope device》
In the above-described embodiment, the case of correcting the color tone in the endoscope apparatus has been described as an example, but the type of image quality correction performed in the endoscope apparatus is not limited to this. In addition, various image quality corrections such as brightness correction, contrast correction, and sharpness correction can be performed.

When recognizing an endoscopic image whose image quality has been corrected, it is converted into an image of standard image quality by performing inverse conversion processing according to the content of the image quality correction.

Further, in the above embodiments, matrix correction is used as a color tone correction method, but the image quality correction method is not limited to this. For example, when adjusting image quality changes due to changes in the amount of light, image processing may be performed using preset image processing parameters for adjusting image quality changes due to the amount of light, and the image quality may be corrected. Further, when adjusting the color tone to the operator's preference, the color tone may be corrected by performing image processing using image processing parameters for adjusting the color tone. When recognizing an image, image processing is performed based on the image processing parameters used for correction, and the image is converted into an image of standard image quality.

<<Image conversion processing performed by an endoscope image processing device>>
As described above, in the endoscopic image processing apparatus, image recognition is performed by converting the image quality of the endoscopic image to be recognized into the standard image quality.

The process of converting to the standard image quality is performed based on the details of the image quality correction applied to the endoscopic image to be recognized. That is, based on the content of the image quality correction, the image is converted into an image of standard image quality by performing inverse conversion processing. At this time, for example, by acquiring information on the image quality adjustment value set in the endoscope apparatus, the image can be easily converted into an image of standard image quality.

In an endoscope apparatus, image quality such as color tone is usually adjusted when an endoscope image, which is an observation image, is displayed on a display device. Therefore, when outputting an endoscopic image to a display device, it is preferable to acquire information about an image quality adjustment value set in the endoscopic device and perform a process of converting the image into a standard image quality image.

Further, in the endoscope apparatus, when an endoscope image is saved as a still image, image quality adjustment for the still image may be performed. In this way, when performing image quality adjustment for a still image, when recognizing a still image of the endoscope image, the information of the adjustment value for the image quality for the still image set in the endoscope device is acquired. It is preferable to perform a process of converting the image into an image of standard image quality.

In addition, since the content of image quality correction generally differs depending on the endoscope apparatus used to capture the endoscopic image, the endoscopic image is processed using numerical values set for each endoscope apparatus to obtain the endoscopic image. is preferably converted into an image of standard image quality. In particular, in the case of color tone adjustment, since adjustment values differ for each operator, it is preferable to perform image conversion processing according to the settings of each operator to convert the image into an image of standard image quality.

<<Modified example of image recognition unit>>
In the above embodiment, the image recognition unit is configured by a convolutional neural network learned by machine learning. However, the configuration of the image recognition unit is not limited to this. In addition, the image recognition unit can be configured with a trained model generated by machine learning using a known technique.

Note that machine learning is performed using a group of images that have been converted to the standard image quality. Therefore, the “reference image quality” can also be said to be the image quality of the image group for learning when the image recognition unit is generated. The “process of converting to the standard image quality” can also be said to be a process of converting to the image quality of the image group for learning when the image recognition unit is generated.

<<Method of Acquisition of Endoscope Image to be Recognized>>
In the above embodiment, the endoscope image to be recognized is directly acquired from the endoscope apparatus, but the acquisition destination of the endoscope image is not limited to this. For example, an endoscopic image stored in another storage device connected via a network, an endoscopic image recorded in a server, or the like may be acquired as an endoscopic image to be recognized.

Also, the endoscopic image to be recognized may be either a moving image or a still image. A moving image is constructed as a time-series image including a plurality of frames.

◆◆Other variations◆◆
<About the hardware configuration of the endoscope image processing device>
The functions of the endoscopic image processing device can be realized by various processors. The various processors include CPUs (Central Processing Units), which are general-purpose processors that execute programs and function as various processing units, and FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacturing. Programmable Logic Devices (PLDs), ASICs (Application Specific Integrated Circuits), and other dedicated electric circuits, which are processors having circuit configurations specially designed to execute specific processing, are included.

One processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same type or different types. For example, one processing unit may be composed of a plurality of FPGAs or a combination of CPUs and FPGAs. Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with a single processor, first, as represented by a computer such as a client or a server, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. be. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor elements.

Also, the functions of the endoscope image processing device can be installed in a processor device that constitutes the endoscope device.

"Endoscope"
The electronic endoscope is not limited to a flexible endoscope, and may be a rigid endoscope. Alternatively, a capsule endoscope may be used.

《About the illumination light of the endoscope device》
As the illumination light, white light, light in one or more specific wavelength bands, light in various wavelength bands such as a combination thereof, or the like is selected according to the purpose of observation. A "specific wavelength band" is a band narrower than the white wavelength band. Specific examples for specific wavelength bands are shown below.

<First example>
A first example of a specific wavelength band is, for example, the blue band or the green band of the visible range. The wavelength band of this first example includes a wavelength band of 390 nm or more and 450 nm or less or a wavelength band of 530 nm or more and 550 nm or less, and the light of the first example is within the wavelength band of 390 nm or more and 450 nm or less or 530 nm or more and 550 nm or less. It has a peak wavelength within the wavelength band.

<Second example>
A second example of a specific wavelength band is, for example, the visible red band. The wavelength band of this second example includes a wavelength band of 585 nm or more and 615 nm or less or a wavelength band of 610 nm or more and 730 nm or less, and the light of the second example is within the wavelength band of 585 nm or more and 615 nm or less or 610 nm or more and 730 nm or less. It has a peak wavelength within the wavelength band.

<Third example>
A third example of the specific wavelength band includes a wavelength band in which oxyhemoglobin and reduced hemoglobin have different absorption coefficients, and the light in the third example peaks in the wavelength band in which oxidized hemoglobin and reduced hemoglobin have different absorption coefficients. have a wavelength. The wavelength band of this third example includes a wavelength band of 400±10 nm, a wavelength band of 440±10 nm, a wavelength band of 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less. It has a peak wavelength within a wavelength band of 10 nm, 440±10 nm, 470±10 nm, or 600 nm or more and 750 nm or less.

<Fourth example>
A fourth example of the specific wavelength band is a wavelength band of excitation light used for observing fluorescence emitted by a fluorescent substance in vivo (fluorescence observation) and exciting the fluorescent substance, for example, from 390 nm to 470 nm.

<Fifth example>
A fifth example of the specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes the wavelength band of 790 nm or more and 820 nm or less or the wavelength band of 905 nm or more and 970 nm or less, and the light of the fifth example is within the wavelength band of 790 nm or more and 820 nm or less or the wavelength band of 905 nm or more and 970 nm or less. It has a peak wavelength within the wavelength band.

《Switching illumination》
As for the type of light source, a laser light source, a xenon light source, an LED light source (LED: Light-Emitting Diode), or an appropriate combination thereof can be adopted. The type and wavelength of the light source, the presence or absence of a filter, etc. are preferably configured according to the type of subject and the purpose of observation. are preferably combined and/or switched. When switching the wavelength, for example, by rotating a disk-shaped filter (rotary color filter) provided with a filter that transmits or blocks light of a specific wavelength, placed in front of the light source, the wavelength of the irradiated light is changed. You can switch.

The image sensor used for the endoscope is not limited to a color image sensor in which a color filter is provided for each pixel, and may be a monochrome image sensor. When a monochrome image sensor is used, it is possible to sequentially switch the wavelength of illumination light and perform frame-sequential (color-sequential) imaging. For example, the wavelength of emitted illumination light may be sequentially switched between violet, blue, green, and red, or white light may be emitted and illumination light emitted by a rotary color filter (red, green, blue, etc.). wavelength may be switched. Alternatively, one or a plurality of narrow band lights may be applied and the wavelength of the illumination light emitted by the rotary color filter may be switched. The narrowband light may be infrared light having two or more wavelengths different from each other.

<<Generation example of special light image>>
The processor device 49 may generate a special light image having information of a specific wavelength band based on a normal light image obtained by imaging using white light. The processor unit 49 converts signals in specific wavelength bands into red (R), green (G) and blue (B), or cyan (Cyan, C), magenta (Magenta, M) and It can be obtained by performing a calculation based on the color information of yellow (Yellow, Y).

《About the program that realizes the function of the endoscope image processing device on the computer》
A program that causes a computer to implement the functions of the endoscopic image processing apparatus described in the above embodiments is recorded on a computer-readable medium that is a non-temporary information storage medium that is a tangible object such as an optical disk, a magnetic disk, or a semiconductor memory. , it is possible to provide the program through this information storage medium. Instead of storing the program in a tangible non-temporary information storage medium and providing the program, it is also possible to provide the program signal as a download service using an electric communication line such as the Internet.

It is also possible to provide a part or all of the functions of the endoscopic image processing apparatus described in the above embodiment as an application server, and provide a service of providing processing functions through electric communication lines.

<<Combination of Embodiments and Modifications>>
The components described in the above embodiments and the components described in the modified examples can be used in combination as appropriate, and part of the components can be replaced.

1 endoscope system 2 network 10 endoscope device 11 endoscope 13 endoscope control device 15 endoscope insertion section 17 image sensor 19A light guide connector 19B video connector 21 display device 23 input device 25 operation section 27 connection section 31 flexible portion 33 bending portion 35 tip rigid portion 37A illumination window 37B illumination window 38 observation window 39 photographing optical system 41 angle knob 43 operation member 47 light source device 49 processor device 51 light source control section 53 combiner 55 coupler 57A optical fiber 57B optical fiber 59 Phosphor 61 Scope cable 65 Endoscope control unit 67 Image processing unit 69 Storage unit 100 Endoscope image processing device 106 ROM
108 Communication interface 110 Input/output interface 112 Storage device 114 Input device 116 Display device 120 Endoscope image acquisition unit 122 Image conversion processing unit 124 Image recognition unit 126 Display control unit λa Wavelength region λb near 400 nm Wavelength region S1 near wavelength 500 nm -S4 Procedure of processing from acquisition of image signal to output to display device S11-S14 Procedure of image recognition processing performed by endoscope image processing apparatus S21-S24 From acquisition of image signal to output to display device Procedures S31-S34 for processing of image recognition performed by the endoscope image processing apparatus S41-S45 Procedures for processing from acquisition of an image signal to output to a display device S51-S55 for processing of an endoscope image processing apparatus Image recognition processing procedure performed in

Claims (11)

an endoscope image acquisition unit that acquires an endoscope image captured by an endoscope device;
an image conversion processing unit that performs a process of converting the endoscopic image into an image of standard image quality based on an image quality adjustment value set in the endoscope apparatus;
an image recognition unit configured with a trained model and performing image recognition on the image converted to the standard image quality;
An endoscopic image processing device comprising: The image conversion processing unit converts the endoscopic image into an image of the standard image quality based on an image quality adjustment value set in the endoscopic device when the endoscopic image is output to a display device. perform the process to
The endoscopic image processing apparatus according to claim 1. The image conversion processing unit converts the endoscopic image into an image of the standard image quality based on an image quality adjustment value set in the endoscopic device when the endoscopic image is saved as a still image. do the processing to
The endoscopic image processing apparatus according to claim 1. The image conversion processing unit performs processing for converting the endoscopic image into an image of the standard image quality based on a color tone adjustment value set in the endoscopic device.
The endoscopic image processing apparatus according to claim 1. The image conversion processing unit performs a process of converting the endoscopic image into an image of the standard image quality based on an adjustment value for image quality change due to the amount of light set in the endoscope device.
The endoscopic image processing apparatus according to claim 1. The image conversion processing unit processes the endoscopic image with a numerical value set for each of the endoscopic devices used to capture the endoscopic image, and transforms the endoscopic image into an image of the standard image quality. do the conversion process,
The endoscopic image processing apparatus according to claim 1. The image recognition unit is composed of a convolutional neural network trained with the image converted to the standard image quality,
The endoscope image processing apparatus according to any one of claims 1 to 6. The image recognition performed by the image recognition unit includes a process of detecting an attention area and/or a process of classifying a recognition target.
The endoscope image processing apparatus according to any one of claims 1 to 6. A method of operating an endoscopic image processing apparatus comprising an endoscopic image acquisition unit, an image conversion processing unit, and an image recognition unit configured with a trained model ,
a step in which the endoscopic image acquisition unit acquires an endoscopic image captured by an endoscopic device;
a step in which the image conversion processing unit performs a process of converting the endoscopic image into a standard image quality image based on an image quality adjustment value set in the endoscope apparatus;
a step in which the image recognition unit performs image recognition on the image converted to the standard image quality;
A method of operating an endoscopic imaging device comprising: A function of acquiring an endoscopic image captured by an endoscope device;
a function of performing processing for converting the endoscopic image into an image of standard image quality based on an image quality adjustment value set in the endoscopic device;
A function of performing image recognition using a trained model on the image converted to the standard image quality;
An endoscope image processing program that realizes on a computer. A non-transitory computer-readable storage medium, wherein when instructions stored on the storage medium are read by a computer,
A function of acquiring an endoscopic image captured by an endoscope device;
a function of performing processing for converting the endoscopic image into an image of standard image quality based on an image quality adjustment value set in the endoscopic device;
A function of performing image recognition using a trained model on the image converted to the standard image quality;
A storage medium that enables a computer to realize an endoscopic image processing function including

Images