JP2024052487A - Processor device, endoscope system, and method of operating same - Google Patents

Abstract

[Problem] To provide a processor device, an endoscope system, and an operation method thereof that can position the tip of an endoscope at an appropriate position or distance relative to a chart in an adjustment jig, regardless of the structure, size, etc., of the tip of the endoscope. [Solution] An error determination unit (421) obtains, from an endoscopic image, information on a first determination region group including at least three first determination regions Jm1-Jm3, and information on a second determination region group that is provided outside the first determination region group and includes at least three second determination regions Jn1-Jn3, and performs an error determination as to whether or not an error has occurred regarding the positional relationship between a chart (408) and the tip (12d) of the endoscope, based on the information on the first determination region group and the information on the second determination region group. [Selected Figure] Fig. 74

Description

The present invention relates to a processor device and an endoscope system that use a chart to perform calibration processes such as white balance adjustment, and a method of operating the same.

In recent years, oxygen saturation imaging has become known in the medical field where endoscopes are used. In oxygen saturation imaging, an object to be observed is illuminated with illumination light including a wavelength band whose absorption coefficient changes with changes in the oxygen saturation of hemoglobin in the blood, and an image is captured (for example, Patent Documents 1 and 2). Then, based on the endoscopic image obtained by imaging, an oxygen saturation image whose color tone changes according to the oxygen saturation is displayed on a display.

In the above oxygen saturation imaging, a signal ratio is calculated based on multiple endoscopic images, and oxygen saturation is calculated as a numerical value based on the calculated signal ratio. In this case, the spectral characteristics of the oxygen saturation illumination light that illuminates the observation target differs from one endoscopic system to another, which can result in individual differences in the signal ratio used to calculate the oxygen saturation. Furthermore, even when the same observation target is imaged, the signal ratio obtained can differ depending on the combination of the light source device and endoscope (camera system) that make up the endoscopic system.

As described above, in order to correct differences in signal ratios caused by individual differences, etc., white balance adjustment is performed in Patent Documents 1 and 2. In Patent Documents 1 to 3, the tip of an endoscope is inserted into an adjustment jig equipped with a chart for white balance adjustment, and white balance adjustment is performed based on an image obtained by capturing an image of the chart. In Patent Document 2, the edge of the inner bottom surface of the adjustment jig is extracted by edge detection, and if the edge image SC of the circle line fits within the guide frame GF, execution of the white balance adjustment process is permitted.

International Publication No. 2021/065939 JP 2019-136412 A JP 2014-236629 A

As described above, when adjusting white balance by imaging a chart, there are cases where accurate white balance adjustment cannot be performed if there is variation in the position or distance between the tip of the endoscope and the chart due to the wavelength dependency of the light distribution characteristics of the illumination light and color unevenness of the camera shading. In addition, the size of the tip of the endoscope varies depending on the organ to be observed, and the appropriate position or distance from the chart also differs depending on the endoscope. For this reason, it is difficult to position the tip of the endoscope at the appropriate position or distance using the positioning mechanism inside the adjustment jig for endoscopes of various sizes.

Therefore, there was a demand for a system that would enable the tip of the endoscope to be positioned at an appropriate position or distance relative to the chart in the adjustment jig, regardless of the structure, size, etc., of the tip of the endoscope. In addition, there was also a demand for a system that would enable positioning relative to the chart using only the low-resource calculation processing that has traditionally been provided in the processor device of the endoscope system, without providing a mechanical positioning structure in the adjustment jig and without using high-resource calculations such as edge extraction as in Patent Document 2.

The present invention aims to provide a processor device, an endoscope system, and an operating method thereof that can position the tip of an endoscope at an appropriate position or distance relative to a chart in an adjustment jig, regardless of the structure, size, etc., of the tip of the endoscope.

The processor device of the present invention includes a processor, which acquires an endoscopic image obtained by imaging a chart from the tip of an endoscope, acquires from the endoscopic image information on a first judgment area group including at least three first judgment areas, and information on a second judgment area group that is provided outside the first judgment area group and includes at least three second judgment areas, and performs an error judgment as to whether or not an error has occurred regarding the positional relationship between the chart and the tip based on the information on the first judgment area group and the information on the second judgment area group.

The information of the three first judgment regions is brightness, and it is preferable that the processor judges an error to have occurred if there is at least one first judgment region among the three first judgment regions whose brightness is less than a first threshold value. The information of the three second judgment regions is brightness, and it is preferable that the processor judges an error to have occurred if there is at least one second judgment region among the three second judgment regions whose brightness is equal to or greater than a second threshold value.

The information of the second judgment area is preferably brightness, and the processor preferably determines an error when there are a predetermined number or more of second judgment areas with a brightness equal to or greater than the second threshold among the plurality of second judgment areas. The information of the second judgment area is preferably brightness, and the processor preferably determines an error when the number of second judgment areas with a brightness equal to or greater than the second threshold is greater than the number of second judgment areas with a brightness less than the second threshold. The brightness of the first judgment area is preferably normalized by a brightness for the normalization area. The brightness of the second judgment area is preferably normalized by a brightness for the normalization area.

It is preferable that the arrangement shape of the first judgment region group is an inverted triangle obtained by connecting three first judgment regions, and that the arrangement shape of the second judgment region group is a triangle obtained by connecting three second judgment regions. A third judgment region is provided inside the first judgment region group, and the processor determines whether an error has occurred based on information on the third judgment region in addition to information on the first and second judgment regions. It is preferable that the information on the third judgment region is a color tone, and that the processor performs an error determination when the color tone of the third judgment region is outside a specific color tone range.

The endoscope system of the present invention includes the processor device of the present invention described above and a display that displays an endoscope image, and the chart is provided with marks. The display displays a guide area between the first and second determination area groups to guide the position of the marks. The guide area preferably has a double circle.

The operation method of the endoscope system of the present invention includes a display step in which the processor displays an endoscopic image obtained by imaging a chart from the tip of the endoscope on a display, and an error determination step in which the processor obtains, from the endoscopic image, information on a first determination area group including at least three first determination areas and information on a second determination area group that is provided outside the first determination area group and includes at least three second determination areas, and performs an error determination as to whether or not an error has occurred regarding the positional relationship between the chart and the tip based on the information on the first determination area group and the information on the second determination area group.

The information of the three first judgment regions is brightness, and in the error judgment step, it is preferable to judge an error if there is at least one first judgment region among the three first judgment regions whose brightness is less than a first threshold value. The information of the three second judgment regions is brightness, and in the error judgment step, it is preferable to judge an error if there is at least one second judgment region among the three second judgment regions whose brightness is equal to or greater than a second threshold value.

The present invention makes it possible to position the tip of the endoscope at an appropriate position or distance relative to the chart in the adjustment jig, regardless of the structure, size, etc., of the tip of the endoscope.

1 is a schematic diagram of an endoscopy system for the digestive tract. 11A and 11B are explanatory diagrams showing the display state on the display and the extended display in normal mode. 13 is an explanatory diagram showing the display mode on the display and the extended display in the oxygen saturation mode. FIG. 13 is an explanatory diagram showing the display mode of the extended display when the mode is switched to the oxygen saturation mode. FIG. FIG. 13A is an image diagram of an expanded display showing an oxygen saturation image inside the digestive tract, and FIG. 13B is an image diagram of an expanded display showing an oxygen saturation image on the serosal side. 2 is a block diagram showing the functions of the endoscope system according to the first embodiment. FIG. 1 is a graph showing an emission spectrum of white light. 1A is a graph showing the emission spectrum of the first illumination light, FIG. 1B is a graph showing the emission spectrum of the second illumination light, and FIG. 4 is a graph showing the spectral sensitivity of an image sensor. 11 is a table showing illumination and acquired image signals in a normal mode. 11 is a table showing illumination and acquired image signals in an oxygen saturation mode or a correction mode. 11A and 11B are explanatory diagrams showing light emission control and display control in the oxygen saturation mode or the correction mode. 1 is a graph showing the reflectance spectrum of hemoglobin that varies depending on blood concentration. 1 is a graph showing the reflection spectrum of hemoglobin and the absorption spectrum of a yellow dye that vary depending on the concentration of the yellow dye. 11 is a table showing the oxygen saturation dependency, blood concentration dependency, and brightness dependency of a B1 image signal, a G2 image signal, and an R2 image signal in a case where there is no influence of a yellow pigment. 1 is a graph showing contour lines representing oxygen saturation. 13 is a table showing the oxygen saturation dependency, blood concentration dependency, and brightness dependency of the X-axis value indicating the signal ratio ln(R2/G2) and the Y-axis value indicating the signal ratio ln(B1/G2). 13 is a table showing the oxygen saturation dependency, blood concentration dependency, yellow pigment dependency, and brightness dependency of a B1 image signal, a G2 image signal, and an R2 image signal when there is an influence of a yellow pigment. FIG. 13 is an explanatory diagram showing the oxygen saturation with and without a yellow dye when the observation subject has the same oxygen saturation. 13 is a table showing the oxygen saturation dependency, blood concentration dependency, yellow pigment dependency, and brightness dependency of a B1 image signal, a B3 image signal, a G2 image signal, a G3 image signal, an R2 image signal, and a B2 image signal when influenced by yellow pigment. 1 is a graph showing a surface representing oxygen saturation as a function of yellow dye. 1 is an explanatory diagram of an oxygen saturation state expressed in three-dimensional coordinates of X, Y, and Z, expressed in two-dimensional coordinates of X and Y. FIG. 13 is a table showing the oxygen saturation dependency, blood concentration dependency, yellow pigment dependency, and brightness dependency of the X-axis value indicating the signal ratio ln(R2/G2), the Y-axis value indicating the signal ratio ln(B1/G2), and the Z-axis value indicating the signal ratio ln(B3/G3). FIG. 2 is a block diagram showing functions of an image processing unit. FIG. 11 is an explanatory diagram showing a method for calculating oxygen saturation. FIG. 11 is an explanatory diagram showing a method for generating a contour line corresponding to a specific dye concentration. FIG. 11 is an image diagram of a display showing a correction image and a specific region. 11 is a graph showing the relationship between pixel value and reliability. 13 is a graph showing a two-dimensional plane for expressing the relationship between bleeding and reliability. 1 is a graph showing a two-dimensional plane for expressing the relationship between fat, residue, mucus, residual liquid, and reliability. FIG. 13 is an image diagram of a display showing low-confidence and high-confidence regions with different saturation. FIG. 13 is an image diagram of a display showing low-confidence regions with region highlighting lines superimposed thereon. 1A is an image diagram of a display showing a specific area highlighted in a first color, and FIG. 1B is an image diagram of a display showing a specific area highlighted in a second color. 13 is a diagram showing an image on the display when it is determined that the table correction process can be performed appropriately. FIG. 13 is a diagram showing an image on the display when it is determined that the table correction process cannot be performed properly. FIG. 10 is a flowchart showing a series of steps in a correction mode. FIG. 11 is a block diagram showing the functions of an endoscope system according to a second embodiment. FIG. 2 is a plan view of the rotary filter. 11 is an explanatory diagram showing a difference value ΔZ used in the calculation value correction process. FIG. FIG. 11 is an explanatory diagram showing a specific oxygen saturation calculation method. FIG. 1 is a schematic diagram of an endoscope system for laparoscopy. 1 is a graph showing an emission spectrum of mixed light. FIG. 2 is an explanatory diagram showing the function of a camera head having four monochrome image sensors. 13 is a graph showing emission spectra of purple light and a second blue light. 11 is a graph showing an emission spectrum of a first blue light. 1 is a graph showing an emission spectrum of green light. 1 is a graph showing an emission spectrum of red light. 1 is a graph showing a wavelength band Rk in the reflectance spectrum of hemoglobin that varies depending on the concentration of a yellow pigment. 13 is a table showing the oxygen saturation dependency, blood concentration dependency, yellow pigment dependency, and brightness dependency of G2, G3, R2, and Rk image signals when there is an influence of yellow pigment. FIG. 1 is an explanatory diagram of a two-sensor type laparoscopic endoscope having a camera head with a color image sensor and a monochrome image sensor. 13A and 13B are graphs showing the light emission pattern when a two-sensor type laparoscopic endoscope is used, in which (A) is a graph showing the light emission pattern when a white frame is used, and (B) is a graph showing the light emission pattern when a green frame is used. (A) is a graph showing the emission pattern during a white frame, (B) is the reflectance of the dichroic mirror, (C) is the sensitivity of the monochrome image sensor, and (D) is a graph showing the pixel value of the image signal output from the monochrome image sensor during a white frame. 1A is a graph showing the light emission pattern during a white frame, (B) is the transmittance of the dichroic mirror, (C) is the sensitivity of the color image sensor, and (D) is a graph showing the pixel values of the image signal output from the color image sensor during a white frame. 13 is a graph showing the light emission pattern during a green frame, (B) the transmittance of the dichroic mirror, (C) the sensitivity of the color image sensor, (D) the pixel value of the image signal output from the B pixel of the color image sensor during a green frame, and (E) the pixel value of the image signal output from the G pixel of the color image sensor during a green frame. 11 is a table showing image signals used in an oxygen saturation mode or a correction mode among image signals obtained in a white frame or a green frame. FIG. 1 is an explanatory diagram showing FPGA processing or PC processing. FIG. 11 is an explanatory diagram showing a light emission control and image signal set when using a two-sensor type laparoscope endoscope. FIG. 11 is an explanatory diagram illustrating effective pixel data determined as being effective pixels. FIG. 2 is an explanatory diagram showing an ROI. FIG. 11 is an explanatory diagram showing effective pixel data used in PC processing. 11A and 11B are explanatory diagrams showing calculation of reliability, calculation of specific dye concentration, and specific dye concentration correlation determination; FIG. 11 is a perspective view of the adjustment jig as viewed from the lid side. FIG. 4 is a perspective view of the adjustment jig as viewed from the insertion hole side. 13 is a perspective view showing a case where the tip portion of the endoscope is inserted into the insertion hole with a larger inner diameter. FIG. 13 is a perspective view showing a case where the tip portion of the endoscope is inserted into the insertion hole having a smaller inner diameter. FIG. This is a cross-sectional view taken along line II in Figure 65. This is a cross-sectional view taken along line II-II in Figure 65. FIG. 11 is an explanatory diagram illustrating separation of the lid portion. 1A is an explanatory diagram showing the outer surface of the lid portion, and FIG. 1A is an explanatory diagram showing an oblique-viewing mirror, and FIG. 1B is an explanatory diagram showing a direct-viewing mirror. This is a cross-sectional view taken along line II in Figure 65, and is an explanatory diagram showing the inclination angle when a direct viewing mirror is used. This is a cross-sectional view taken along line II in Figure 65, and is an explanatory diagram showing the inclination angle when an oblique mirror is used. 1 is a block diagram showing functions of a processor device including an error determination unit; FIG. 13 is an image showing a chart including marks on a display. 11A and 11B are image diagrams of marks that are displayed shifted in the Y or Z direction from the appropriate position on the display. FIG. 13 is an image showing a mark when the distance between the chart and the tip of the endoscope is too close. FIG. 13 is an image showing a mark when the distance between the chart and the tip of the endoscope is too far. FIG. 1 is an image diagram showing a mark whose color tone falls outside a particular color tone range. 11 is a flowchart showing a sequence of operations in a white balance mode. FIG. 13 is an explanatory diagram showing four first determination regions, eight second determination regions, and a normalization region. FIG. 11 is an explanatory diagram of a case where the brightness of the entire image is outside the reference range. FIG. 13 is an explanatory diagram of a flare captured in an endoscopic image. FIG. 13 is an explanatory diagram of two adjacent marks in an endoscopic image. FIG. 13 is an explanatory diagram of a mark distorted into an oval shape.

[First embodiment]
1, an endoscope system 10 includes an endoscope 12, a light source device 13, a processor device 14, a display 15, a processor-side user interface 16, an extended processor device 17, and an extended display 18. The endoscope 12 is optically or electrically connected to the light source device 13, and is electrically connected to the processor device 14. The extended processor device 17 is electrically connected to the light source device 13 and the processor device 14. Note that the "display" in the claims includes the extended display 18 in addition to the display 15.

The endoscope 12 has an insertion section 12a, an operating section 12b, a bending section 12c, and a tip section 12d. The insertion section 12a is inserted into the subject's body. The operating section 12b is provided at the base end of the insertion section 12a. The bending section 12c and the tip section 12d are provided on the tip side of the insertion section 12a. The bending section 12c is bent by operating the angle knob 12e of the operating section 12b. The tip section 12d is directed in the desired direction by the bending operation of the bending section 12c. A forceps channel (not shown) for inserting a treatment tool and the like is provided from the insertion section 12a to the tip section 12d. The treatment tool is inserted into the forceps channel from the forceps port 12j.

Inside the endoscope 12, an optical system for forming an image of the subject and an optical system for irradiating the subject with illumination light are provided. The operation unit 12b is provided with an angle knob 12e, a mode switch 12f, a still image acquisition instruction switch 12h, and a zoom operation unit 12i. The mode switch 12f is used to switch between observation modes. The still image acquisition instruction switch 12h is used to instruct acquisition of a still image of the subject. The zoom operation unit 12i is used to enlarge or reduce the observation target. In addition to the mode switch 12f and the still image acquisition instruction switch 12h, the operation unit 12b may also be provided with a scope-side user interface 19 for performing various operations on the processor device 14.

The light source device 13 generates illumination light. The processor device 14 performs system control of the endoscope system 10, and further performs image processing on the image signal transmitted from the endoscope 12 to generate an endoscopic image. The display 15 displays the medical image transmitted from the processor device 14. The processor-side user interface 16 has a keyboard, mouse, microphone, tablet, foot switch, touch pen, etc., and accepts input operations such as function settings.

The endoscope system 10 has three modes: normal mode, oxygen saturation mode, and correction mode, and the user can switch between these three modes by operating the mode switch 12f. As shown in FIG. 2, in the normal mode, a white light image with natural coloring obtained by capturing an image of an object to be observed using white light as illumination light is displayed on the display 15, while nothing is displayed on the extended display 18.

As shown in FIG. 3, in the oxygen saturation mode, the oxygen saturation of the object of observation is calculated, and an oxygen saturation image that visualizes the calculated oxygen saturation is displayed on the extended display 18. Also, in the oxygen saturation mode, a white light equivalent image that has fewer short wavelength components than the white light image is displayed on the display 15. In the correction mode, correction processing is performed on the calculation of oxygen saturation based on the concentration of a specific pigment other than hemoglobin in the blood, such as a yellow pigment. When switching to the oxygen saturation mode, a message MS0 stating "Please perform correction processing" is displayed on the extended display 18, as shown in FIG. 4. When the correction processing is completed, the oxygen saturation image is displayed on the extended display 18.

The endoscope system 10 is a flexible endoscope type for the digestive tract, such as the stomach and large intestine, and in the oxygen saturation mode, as shown in FIG. 5(A), displays an internal digestive tract oxygen saturation image that visualizes the state of oxygen saturation inside the digestive tract on the extended display 18. In addition, in the case of a rigid endoscope type for the abdominal cavity, such as the serosa, in the oxygen saturation mode, as shown in FIG. 5(B), the endoscope system described below displays a serosal side oxygen saturation image that visualizes the state of oxygen saturation on the serosal side on the extended display 18. It is preferable to use an image in which the saturation is adjusted with respect to the white light equivalent image for the serosal side oxygen saturation image. It is preferable to adjust the saturation in the correction mode, regardless of whether it is a mucosa, a serosal membrane, a flexible endoscope, or a rigid endoscope.

In the oxygen saturation mode, it is possible to accurately calculate the oxygen saturation in the following cases.
- When observing a predetermined target area (e.g., esophagus, stomach, large intestine) - When not in an external environment with surrounding lighting - When there is no residue, residual liquid, mucus, blood, or fat remaining on the mucosa or serosa - When dye is not sprayed onto the mucosa - When the endoscope 12 is more than 7 mm away from the observation area - When the endoscope is observed at an appropriate distance without being too far away from the observation area - Areas that are sufficiently illuminated by illumination light - When there is little specular reflection from the observation area - Areas within 2/3 of the oxygen saturation image - When the movement of the endoscope is small, or when there is little patient movement such as pulsation or breathing - When blood vessels deep in the gastrointestinal mucosa cannot be observed

As shown in FIG. 6, the light source device 13 includes a light source section 20 and a light source processor 21 that controls the light source section 20. The light source section 20 has, for example, a plurality of semiconductor light sources, which are turned on or off, and when turned on, the amount of light emitted by each semiconductor light source is controlled to emit illumination light that illuminates the observation target. In this embodiment, the light source section 20 has five color LEDs: a V-LED (Violet Light Emitting Diode) 20a, a BS-LED (Blue Short-wavelength Light Emitting Diode) 20b, a BL-LED (Blue Long-wavelength Light Emitting Diode) 20c, a G-LED (Green Light Emitting Diode) 20d, and an R-LED (Red Light Emitting Diode) 20e.

The V-LED 20a emits violet light V of 410 nm ± 10 nm. The BS-LED 20b emits second blue light BS of 450 nm ± 10 nm. The BL-LED 20c emits first blue light BL of 470 nm ± 10 nm. The G-LED 20d emits green light G in the green band. The central wavelength of the green light G is preferably 540 nm. The R-LED 20e emits red light R in the red band. The central wavelength of the red light R is preferably 620 nm. The central wavelength and peak wavelength of each of the LEDs 20a to 20e may be the same or different.

The light source processor 21 independently controls the on/off and light emission amount of each of the LEDs 20a-20e by inputting control signals to each of the LEDs 20a-20e. The on/off control by the light source processor 21 differs depending on the mode, and will be described in detail later.

The light emitted by each of the LEDs 20a to 20e is incident on the light guide 25 via the optical path coupling section 23, which is composed of a mirror, a lens, etc. The light guide 25 is built into the endoscope 12 and the universal cord (a cord that connects the endoscope 12 with the light source device 13 and the processor device 14). The light guide 25 propagates the light from the optical path coupling section 23 to the tip 12d of the endoscope 12.

The tip 12d of the endoscope 12 is provided with an illumination optical system 30 and an imaging optical system 31. The illumination optical system 30 has an illumination lens 32, and the illumination light propagated by the light guide 25 is irradiated onto the observation object via the illumination lens 32. The imaging optical system 31 has an objective lens 35 and an imaging sensor 36. Light from the observation object irradiated with the illumination light is incident on the imaging sensor 36 via the objective lens 35. As a result, an image of the observation object is formed on the imaging sensor 36.

The imaging sensor 36 is a color imaging sensor that captures an image of an object being observed while being illuminated with illumination light. Each pixel of the imaging sensor 36 is provided with either a B pixel (blue pixel) having a B (blue) color filter, a G pixel (green pixel) having a G (green) color filter, or an R pixel (red pixel) having an R (red) color filter. The spectral transmittance of the B color filter, G color filter, and R color filter will be described later. For example, the imaging sensor 36 is preferably a color imaging sensor with a Bayer array in which the ratio of the number of B pixels, G pixels, and R pixels is 1:2:1.

The imaging sensor 36 may be a CCD (Charge Coupled Device) imaging sensor or a CMOS (Complementary Metal-Oxide Semiconductor) imaging sensor. Also, instead of the primary color imaging sensor 36, a complementary color imaging sensor equipped with complementary color filters of C (cyan), M (magenta), Y (yellow) and G (green) may be used. When a complementary color imaging sensor is used, image signals of four colors CMYG are output, and by converting the four color image signals of CMYG into three color image signals of RGB by complementary color-primary color conversion, image signals of each of the RGB colors similar to those of the imaging sensor 36 can be obtained.

The imaging sensor 36 is driven and controlled by the imaging processor 37. The control of each mode in the imaging processor 37 will be described later. The CDS/AGC circuit 40 (Correlated Double Sampling/Automatic Gain Control) performs correlated double sampling (CDS) and automatic gain control (AGC) on the analog image signal obtained from the imaging sensor 36. The image signal that passes through the CDS/AGC circuit 40 is converted into a digital image signal by an A/D converter 41 (Analog/Digital). The digital image signal after A/D conversion is input to the processor device 14.

The processor device 14 includes an image processing unit 50, a display control unit 52, and a central control unit 53. The processor device 14 has programs related to various processes built into a program memory (not shown). The functions of the image processing unit 50, the display control unit 52, and the central control unit 53 are realized by the central control unit 53, which is made up of a processor, executing the programs in the program memory.

The image processing unit 50 performs various signal processing such as defect correction processing, offset processing, gain correction processing, linear matrix processing, gamma conversion processing, demosaic processing, white balance processing, YC conversion processing, and noise reduction processing on the image signal received from the endoscope 12. In the defect correction processing, the signal of the defective pixel of the imaging sensor 36 is corrected. In the offset processing, the dark current component is removed from the image signal that has been subjected to the defect correction processing, and an accurate zero level is set. In the gain correction processing, the signal level of each image signal is adjusted by multiplying the image signal of each color after the offset processing by a specific gain. The image signal of each color after the gain correction processing is subjected to linear matrix processing to improve color reproducibility.

Then, the brightness and saturation of each image signal are adjusted by gamma conversion processing. The image signals after linear matrix processing are subjected to demosaic processing (also called isotropic processing or synchronization processing), and signals of the missing colors of each pixel are generated by interpolation. Through demosaic processing, all pixels have signals of each color of RGB. The image processing unit 50 performs YC conversion processing on each image signal after demosaic processing, and generates a luminance signal Y and color difference signals Cb and Cr. The image processing unit 50 performs noise reduction processing, for example, using the moving average method or median filter method, on the image signals that have been subjected to demosaic processing, etc.

The image processing unit 50 also performs image processing such as color conversion processing such as 3x3 matrix processing, tone conversion processing, and 3D LUT (Look Up Table) processing, color enhancement processing, and structure enhancement processing such as spatial frequency enhancement. The image processing unit 50 performs image processing according to the mode. In the normal mode, the image processing unit 50 performs image processing for the normal mode to generate a white light image. In the oxygen saturation mode, the image processing unit 50 performs image processing for the oxygen saturation to generate a white light equivalent image. In the oxygen saturation mode, the image processing unit 50 transmits an image signal from the endoscope 12 to the extended processor device 17 via the image communication unit 51.

The display control unit 52 performs display control to display the white light image or image information such as the oxygen saturation image from the image processing unit 50, and other information, on the display 15. In accordance with the display control, the white light image or an image equivalent to white light is displayed on the display 15.

The extended processor device 17 receives image signals from the processor device 14 and performs various image processing. In the oxygen saturation mode, the extended processor device 17 calculates the oxygen saturation and generates an oxygen saturation image that visualizes the calculated oxygen saturation. The generated oxygen saturation image is displayed on the extended display 18. In the correction mode, the extended processor device 17 calculates a specific dye concentration according to user operations and performs correction processing related to the calculation of oxygen saturation based on the calculated specific dye concentration. Details of the oxygen saturation mode and correction mode performed by the extended processor device 17 will be described later.

The following describes the on/off control in each mode. In normal mode, the V-LED 20a, BS-LED 20b, G-LED 20d, and R-LED 20e are simultaneously turned on to emit white light containing purple light V with a central wavelength of 410 nm, second blue light BS with a central wavelength of 450 nm, broadband green light G in the green band, and red light R with a central wavelength of 620 nm, as shown in FIG. 7.

In the oxygen saturation mode and the correction mode, three frames of light emission with different light emission patterns are repeated. In the first frame, as shown in FIG. 8(A), the BL-LED 20c, the G-LED 20d, and the R-LED 20e are simultaneously turned on to emit a first broadband illumination light including a first blue light BL with a central wavelength of 470 nm, a broadband green light G in the green band, and a red light R with a central wavelength of 620 nm. In the second frame, as shown in FIG. 8(B), the BS-LED 20b, the G-LED 20d, and the R-LED 20e are simultaneously turned on to emit a second illumination light including a second blue light BS with a central wavelength of 450 nm, a broadband green light G in the green band, and a red light R with a central wavelength of 620 nm. In the third frame, as shown in FIG. 8(C), the G-LED 20d is turned on to emit a broadband green light G in the green band. In addition, in the oxygen saturation mode, the first and second frames are the frames required to obtain the image signal necessary to calculate the oxygen saturation, so light may be emitted only in the first and second frames.

As shown in FIG. 9, the B color filter BF provided in the B pixel of the image sensor 36 transmits mainly light in the blue band, specifically light with a wavelength band of 380 to 560 nm (blue transmission band). The peak wavelength at which the transmittance is maximum is around 460 to 470 nm. The G color filter GF provided in the G pixel of the image sensor 36 transmits mainly light in the green band, specifically light with a wavelength band of 450 to 630 nm (green transmission band). The R color filter RF provided in the R pixel of the image sensor 36 transmits mainly light in the red band, specifically light with a wavelength band of 580 to 760 nm (red transmission band).

As shown in FIG. 10, in normal mode, the imaging processor 37 controls the imaging sensor 36 to capture an image of an object being observed that is illuminated with purple light V, second blue light BS, green light G, and red light R for each frame. This causes the B pixels of the imaging sensor 36 to output Bc image signals, the G pixels to output Gc image signals, and the R pixels to output Rc image signals.

As shown in FIG. 11, in the oxygen saturation mode, when the first illumination light including the first blue light BL, the green light G, and the red light R is illuminated on the observation object in the first frame, the imaging processor 37 outputs a B1 image signal from the B pixel of the imaging sensor 36, a G1 image signal from the G pixel, and an R1 image signal from the R pixel as the first illumination light image. When the second illumination light including the second blue light BS, the green light G, and the red light R is illuminated on the observation object in the second frame, the imaging processor 37 outputs a B2 image signal from the B pixel of the imaging sensor 36, a G2 image signal from the G pixel, and an R2 image signal from the R pixel as the second illumination light image.

In the third frame, when the third illumination light, which is green light G, is illuminated on the object to be observed, the imaging processor 37 outputs a B3 image signal from the B pixel of the imaging sensor 36, a G3 image signal from the G pixel, and an R3 image signal from the R pixel as the third illumination light image.

In the oxygen saturation mode, as shown in FIG. 12, the first illumination light is emitted in the first frame (1stF), the second illumination light is emitted in the second frame (2ndF), and the third illumination light is emitted in the third frame (3rdF), after which the second illumination light is emitted in the second frame and the first illumination light is emitted in the first frame. An image equivalent to white light obtained based on the emission of the second illumination light in the second frame is displayed on the display 15. In addition, an oxygen saturation image obtained based on the emission of the first to third illumination lights in the first to third frames is displayed on the extended display 18.

In the oxygen saturation mode, of the image signals for the above three frames, the B1 image signal included in the first illumination light image and the G2 and R2 image signals included in the second illumination light image are used. In the correction mode, in addition to the B1, G2, and R2 image signals, the B3 and G3 image signals included in the third illumination light image are used to measure the concentration of a specific pigment (such as a yellow pigment) that affects the accuracy of oxygen saturation calculation.

The B1 image signal includes image information on at least the first blue light BL from the light that has passed through the B color filter BF in the first illumination light. The B1 image signal (image signal for oxygen saturation) includes image information on the wavelength band B1, whose reflection spectrum changes with changes in the oxygen saturation of blood hemoglobin, as image information on the first blue light BL. As the wavelength band B1, for example, as shown in FIG. 13, it is preferable to set the wavelength band B1 to a wavelength band of 460 nm to 480 nm, including 470 nm, at which the difference between the reflection spectrum of oxygenated hemoglobin shown by curves 55b and 56b and the reflection spectrum of reduced hemoglobin shown by curves 55a and 56a is maximized.

In FIG. 13, curve 55a represents the reflectance spectrum of reduced hemoglobin when the blood concentration is high, and curve 55b represents the reflectance spectrum of oxygenated hemoglobin when the blood concentration is high. On the other hand, curve 56a represents the reflectance spectrum of reduced hemoglobin when the blood concentration is low, and curve 56b represents the reflectance spectrum of oxygenated hemoglobin when the blood concentration is low.

The G2 image signal contains image information of at least the wavelength band G2 relating to green light G among the light that has passed through the G color filter GF in the first illumination light. The wavelength band G2 is preferably a wavelength band of 500 nm to 580 nm, for example, as shown in FIG. 13. The R2 image signal contains image information of at least the wavelength band R2 relating to red light R among the light that has passed through the R color filter RF in the first illumination light. The wavelength band R2 is preferably a wavelength band of 610 nm to 630 nm, for example, as shown in FIG. 13.

As shown in FIG. 14, the image information of the wavelength band B1 contains image information on the first blue light BL, and the image information of the wavelength band B3 contains image information on the green light G. The image information on the first blue light BL and the green light G is image information in which the absorption spectrum of a specific pigment changes due to a change in the concentration of the specific pigment such as a yellow pigment. The change in the absorption spectrum of the specific pigment also causes a change in the reflection spectrum of hemoglobin. The curve 55a represents the reflection spectrum of reduced hemoglobin when there is no influence of the yellow pigment, and the curve 55c represents the reflection spectrum of reduced hemoglobin when there is an influence of the yellow pigment. As shown in these curves 55a and 55c, the reflection spectrum of reduced hemoglobin changes depending on the presence or absence of the yellow pigment (the same applies to the reflection spectrum of oxygenated hemoglobin). Therefore, the reflection spectrum of the wavelength bands B1 and B3 changes due to a change in the oxygen saturation of blood hemoglobin under the influence of a specific pigment such as a yellow pigment.

In an ideal case where the object of observation using the endoscope 12 is not affected by a specific pigment such as a yellow pigment, as shown in FIG. 15, the B1 image signal (denoted as "B1"), the G2 image signal (denoted as "G2"), and the R2 image signal (denoted as "R2") are each affected by oxygen saturation dependency, blood concentration dependency, or brightness dependency. As described above, the B1 image signal includes the wavelength band B1 in which the difference between the reflectance spectrum of oxygenated hemoglobin and the reflectance spectrum of reduced hemoglobin is maximized, so that the oxygen saturation dependency that changes with oxygen saturation is about "large". In addition, the B1 image signal has a blood concentration dependency that changes with blood concentration of about "medium", as shown in curves 55a, 55b and curves 56a, 56b. In addition, the B1 image signal has a brightness dependency that changes with the brightness of the object of observation. The degrees of dependency are indicated as "large," "medium," and "small," with "large" indicating a high degree of dependency compared to other image signals, "medium" indicating a medium degree of dependency compared to other image signals, and "small" indicating a low degree of dependency compared to other image signals.

The G2 image signal has a low dependency on oxygen saturation because the magnitude relationship between the reflectance spectrum of oxygenated hemoglobin and the reflectance spectrum of reduced hemoglobin is reversed over a wide wavelength band. Also, the G2 image signal has a high degree of dependency on blood concentration, as shown by curves 55a, 55b and curves 56a, 56b. Also, the G2 image signal has a dependency on brightness, similar to the B1 image signal.

The R2 image signal does not change as much with oxygen saturation as the B1 image signal, but it is "medium" dependent on oxygen saturation. Also, the R2 image signal is "low" dependent on blood concentration, as shown by curves 55a, 55b and curves 56a, 56b. Also, the G2 image signal, like the B1 image signal, is "dependent" on brightness.

As described above, the B1, G2, and R2 image signals all have brightness dependency, so by using the G2 image signal as the normalized signal, the oxygen saturation calculation table 83 for calculating oxygen saturation is created using the signal ratio ln(B1/G2) obtained by normalizing the B1 image signal with the G2 image signal and the signal ratio ln(R2/G2) obtained by normalizing the R2 image signal with the G2 image signal. Note that the "ln" in the signal ratio ln(B1/G2) is the natural logarithm (similar to the signal ratio ln(R2/G2)).

When the relationship between the signal ratio ln(B1/G2) and the signal ratio ln(R2/G2) and the oxygen saturation is expressed in two-dimensional coordinates with the signal ratio ln(R2/G2) on the X-axis and the signal ratio ln(B1/G2) on the Y-axis, as shown in FIG. 16, the oxygen saturation is represented by the contour line EL along the Y-axis direction. The contour line ELH represents an oxygen saturation level of "100%", and the contour line ELL represents an oxygen saturation level of "0%". The contour lines are distributed so that the oxygen saturation level gradually decreases from the contour line ELH to the contour line ELL (in FIG. 16, the contour lines of "80%, " "60%, " "40%, " and "20%" are distributed).

The X-axis value (signal ratio ln(R2/G2)) and Y-axis value (signal ratio ln(B1/G2)) are affected by oxygen saturation dependency and blood concentration dependency, respectively. However, as for brightness dependency, as shown in FIG. 17, the X-axis value and the Y-axis value are each normalized by the G2 image signal, and therefore are considered to be "none," meaning that they are not affected. For the X-axis value, the oxygen saturation dependency is considered to be about "medium," and the blood concentration dependency is considered to be about "large." On the other hand, for the Y-axis value, the oxygen saturation dependency is considered to be about "large," and the blood concentration dependency is considered to be about "medium."

On the other hand, in a realistic case where an object observed using the endoscope 12 is affected by a specific dye such as a yellow dye, as shown in FIG. 18, the B1 image signal (denoted as "B1"), G2 image signal (denoted as "G2"), and R2 image signal (denoted as "R2") are each affected by oxygen saturation dependency, blood concentration dependency, yellow dye dependency, or brightness dependency. The B1 image signal contains image information in which the absorption spectrum of a specific dye such as a yellow dye changes depending on the change in concentration, and therefore has a "large" degree of yellow dye dependency, which changes depending on the yellow dye. In contrast, the G2 image signal is less affected by the yellow dye compared to the B1 image signal, and therefore has a "small to medium" degree of yellow dye dependency. The R1 image signal is less affected by the yellow dye, and therefore has a "small" degree of yellow dye dependency.

When the signal ratio ln(R2/G2) is represented on the X-axis and the signal ratio ln(B1/G2) is represented on the Y-axis in two-dimensional coordinates, even if the subject has the same oxygen saturation, the oxygen saturation StO2A without the yellow dye and the oxygen saturation StO2b with the yellow dye are represented differently, as shown in FIG. 19. The oxygen saturation StO2B appears to be shifted higher than the oxygen saturation StO2A due to the presence of the yellow dye.

Therefore, in order to accurately calculate the oxygen saturation even in the case of yellow pigment dependency, the B3 image signal and the G3 image signal included in the third illumination light image are used when calculating the oxygen saturation. The B3 image signal includes image information related to the light transmitted through the B color filter BF in the third illumination light. The B3 image signal (specific pigment image signal) includes image information of the wavelength band B3 that has sensitivity to specific pigments other than hemoglobin, such as yellow pigment (see FIG. 14). Although the B3 image signal is not as sensitive to the specific pigment as the B1 image signal, it has a certain degree of sensitivity to the specific pigment. Therefore, as shown in FIG. 20, the B1 image signal has a "large" dependency on the yellow pigment, while the B3 image signal has a "medium" dependency on the yellow pigment. Note that the B3 image signal has a "small" dependency on oxygen saturation, a "large" dependency on blood concentration, and a "present" dependency on brightness.

The G3 image signal also does not have as much sensitivity to the specific dye as the B3 image signal, but it does contain an image signal in the wavelength band G3 that has some sensitivity to the specific dye (see FIG. 14). Therefore, the yellow dye dependency of the G3 image signal is "small to medium". The G3 image signal has "small" oxygen saturation dependency, "large" blood concentration dependency, and "some" brightness dependency. The B2 image signal also has "large" yellow dye dependency, so the B2 image signal may be used instead of the B3 image signal when calculating oxygen saturation. The B2 image signal has "small" oxygen saturation dependency, "large" blood concentration dependency, and "some" brightness dependency.

When the relationship between the signal ratio ln(B1/G2) and the signal ratio ln(R2/G2), the yellow pigment, and the oxygen saturation is expressed in three-dimensional coordinates with the signal ratio ln(R2/G2) on the X-axis, the signal ratio ln(B1/G2) on the Y-axis, and the signal ratio ln(B3/G3) on the Z-axis, as shown in FIG. 21, the curved surfaces CV0 to CV4 representing the oxygen saturation are distributed in the Z-axis direction according to the pigment concentration of the yellow pigment. The curved surface CV0 represents the oxygen saturation when the yellow pigment has a concentration of "0" (no effect of the yellow pigment). The curved surfaces CV1 to CV4 represent the oxygen saturation when the yellow pigment has concentrations of "1" to "4", respectively. The higher the concentration number, the higher the concentration of the yellow pigment. Note that, as shown in the curved surfaces CV0 to CV4, the higher the concentration of the yellow pigment, the lower the Z-axis value changes.

When the oxygen saturation state expressed in three-dimensional coordinates X, Y, and Z as shown in FIG. 22(A) is expressed in two-dimensional coordinates X and Y, as shown in FIG. 22(B), the areas AR0 to AR4 representing the oxygen saturation state are distributed at different positions according to the concentration of the yellow dye. Areas AR0 to AR4 represent the distribution of oxygen saturation when the concentration of the yellow dye is "0" to "4", respectively. By defining a contour line EL representing oxygen saturation for each of these areas AR0 to AR4, it is possible to obtain the oxygen saturation corresponding to the concentration of the yellow dye (see FIG. 16). Note that, as shown in areas AR0 to AR4, the higher the concentration of the yellow dye, the higher the value on the X axis and the lower the value on the Y axis.

As shown in FIG. 23, the X-axis value (signal ratio ln(R2/G2)), Y-axis value (signal ratio ln(B1/G2)), and Z-axis value (signal ratio ln(B3/G3)) are yellow pigment dependent. The X-axis value has a yellow pigment dependency of "small to medium", the Y-axis value has a yellow pigment dependency of "large", and the Z-axis value has a yellow pigment dependency of "medium". The Z-axis value has an oxygen saturation dependency of "small to medium" and a blood concentration dependency of "small to medium". The Z-axis value has no brightness dependency because it is normalized by the G3 image signal.

24, the extended processor device 17 includes an oxygen saturation image generating unit 61, a specific dye concentration calculating unit 62, a table correcting unit 63, a mode switching unit 64, a display mode control unit 65, a reliability calculating unit 66, a first correction determining unit 67, a second correction determining unit 68, and a determination notifying unit 69. The extended processor device 17 has programs related to various processes built into a program memory (not shown). The functions of the oxygen saturation image generating unit 61, the specific dye concentration calculating unit 62, the table correcting unit 63, the mode switching unit 64, the display mode control unit 65, the reliability calculating unit 66, the first correction determining unit 67, the second correction determining unit 68, and the determination notifying unit 69 are realized by a central control unit (not shown) configured by a processor executing the programs in the program memory.

The oxygen saturation image generating unit 61 includes a base image generating unit 70, a calculation value calculating unit 71, an oxygen saturation calculating unit 72, an oxygen saturation calculation table 73, and a color adjusting unit 74. The base image generating unit 70 generates a base image based on an image signal from the processor device 14. The base image is preferably an image that can grasp morphological information such as the shape of the observation target. The base image is composed of a B2 image signal, a G2 image signal, and an R2 image signal. The base image may be a narrowband light image in which blood vessels or structures (ductal structures) are highlighted by narrowband light or the like.

The calculation value calculation unit 71 calculates the calculation value by calculation processing based on the B1 image signal, G2 image signal, and R2 image signal included in the oxygen saturation image signal. Specifically, the calculation value calculation unit 71 calculates the signal ratio B1/G2 between the B1 image signal and the G2 image signal, and the signal ratio R2/G2 between the R2 image signal and the G2 image signal as calculation values used to calculate the oxygen saturation. Note that it is preferable to logarithmize (ln) the signal ratios B1/G2 and R2/G2, respectively. In addition, the color difference signals Cr and Cb calculated from the B1 image signal, G2 image signal, and R2 image signal, or saturation S, hue H, etc. may be used as the calculation value.

The oxygen saturation calculation unit 72 refers to the oxygen saturation calculation table 73 and calculates the oxygen saturation based on the calculated value. The oxygen saturation calculation table 73 stores the correlation between the signal ratios B1/G2 and R2/G2, which are one of the calculated values, and the oxygen saturation. When the correlation is expressed in two-dimensional coordinates with the signal ratio ln(B1/G2) on the vertical axis and the signal ratio ln(R2/G2) on the horizontal axis, the state of oxygen saturation is expressed by the contour line EL extending in the horizontal axis direction, and when the oxygen saturation differs, the contour line EL is distributed at different positions in the vertical axis direction (see FIG. 16).

The oxygen saturation calculation unit 72 refers to the oxygen saturation calculation table 73 and calculates the oxygen saturation corresponding to the signal ratios B1/G2 and R2/G2 for each pixel. For example, as shown in Fig. 25, when the signal ratios of a specific pixel are ln(B1 ^* /G2 ^* ) and ln(R2 ^* /G2 ^* ), the oxygen saturation corresponding to the signal ratios ln(B1 ^* /G2 ^* ) and ln(R2 ^* /G2 ^* ) is "40%". Therefore, the oxygen saturation calculation unit 72 calculates the oxygen saturation of the specific pixel to be "40%".

The color tone adjustment unit 74 generates an oxygen saturation image by performing composite color processing that changes the color tone of the base image using the oxygen saturation calculated by the oxygen saturation calculation unit 72. The color tone adjustment unit 74 maintains the color tone of areas in the base image where the oxygen saturation exceeds a threshold, and changes the color tone of areas where the oxygen saturation is below the threshold to a color tone that changes according to the oxygen saturation. This maintains the color tone of normal areas where the oxygen saturation exceeds the threshold, while changing only the color tone of abnormal areas where the oxygen saturation is below the threshold and becomes low, making it possible to grasp the oxygen status of abnormal areas under conditions where morphological information of normal areas can be observed.

In addition, the color tone adjustment unit 74 may generate an oxygen saturation image by pseudo-color processing that assigns colors according to the oxygen saturation level, regardless of the level of oxygen saturation. When performing pseudo-color processing, the base image is not necessary.

In the correction mode, the specific dye concentration calculation unit 62 calculates the specific dye concentration based on the specific dye image signal including image information of a wavelength band that has sensitivity to a specific dye other than hemoglobin in blood among the dyes contained in the observation subject. The specific dye includes, for example, a yellow dye such as bilirubin. The specific dye image signal preferably includes at least a B3 image signal. Specifically, the specific dye concentration calculation unit 62 calculates the signal ratios ln(B1/G2), ln(G2/R2), and ln(B3/G3). The specific dye concentration calculation unit 62 then refers to the specific dye concentration calculation table 62a to calculate the specific dye concentrations corresponding to the signal ratios ln(B1/G2), ln(G2/R2), and ln(B3/G3).

The specific dye concentration calculation table 62a stores the correlation between the signal ratios ln(B1/G2), ln(G2/R2), and ln(B3/G3) and the specific dye concentrations. For example, if the ranges of the signal ratios ln(B1/G2), ln(G2/R2), and ln(B3/G3) are divided into five stages, the specific dye concentrations "0" to "4" are associated with the signal ratios ln(B1/G2), ln(G2/R2), and ln(B3/G3) in the five stages and stored in the specific dye concentration calculation table 62a. Note that it is preferable to logarithmize (ln) the signal ratio B3/G3.

The table correction unit 63 performs table correction processing to correct the oxygen saturation calculation table 73 based on the specific dye concentration as a correction processing performed in the correction mode. In the table correction processing, the correlation between the signal ratios B1/G2 and R2/G2 stored in the oxygen saturation calculation table 73 and the oxygen saturation is corrected. Specifically, when the specific dye concentration is "2", the table correction unit 63 generates a contour line EL representing the state of oxygen saturation in the area AR2 corresponding to the specific dye concentration "2" among the areas AR0 to AR4 determined according to the specific dye concentration as shown in FIG. 26. The table correction unit 63 corrects the oxygen saturation calculation table 73 to the generated contour line EL.

In the present embodiment, in the correction mode, as shown in FIG. 27, a correction image 80 is displayed on the extended display 18, and a specific region 81 used to calculate the specific dye concentration is displayed on the correction image 80. The shape of the specific region 81 may be a circle, an ellipse, a rectangle, or the like. The position of the specific region 81 is preferably the center of the screen, but may be other positions. While observing the correction image, the user operates the endoscope 12 so that the region suitable for correction of the oxygen saturation calculation table 73 falls within the specific region 81. Then, when the region suitable for correction falls within the specific region 81, the user performs a correction operation using the processor-side user interface 16 or the scope-side user interface 19. The table correction unit 63 corrects the oxygen saturation calculation table 73 using the specific dye concentration in the specific region at the time the correction operation is performed. The specific dye concentration in the specific region is preferably the average value of the specific dye concentration in each pixel in the specific region, and is preferably a weighted average value obtained by weighting the specific dye concentration according to the reliability calculated by the reliability calculation unit 66 described later.

In this embodiment, correction assistance is provided so that an appropriate correction area that is less affected by disturbances that affect the calculation of oxygen saturation is clearly displayed as an area suitable for correcting the oxygen saturation calculation table 73, and so that the user can select the appropriate correction area by performing a correction operation. The details of the correction assistance will be described later.

The mode switching unit 64 switches between the oxygen saturation mode and the correction mode in accordance with the operation of the mode switching switch 12f by the user. When the mode is switched to the oxygen saturation mode, the oxygen saturation image generating unit 61 generates an oxygen saturation image. When the mode is switched to the correction mode, the correction operation for performing the correction process is made available for acceptance, and the correction process is executed in accordance with the correction operation.

The correction support in the correction mode will be described below. When the correction mode is switched to, a correction image 80 is displayed on the extended display 18, and a specific region 81 used to calculate the specific dye concentration is displayed on the extended display 18 (see FIG. 27). The display mode control unit 65 at least changes the display mode of the correction image so that the user can select the appropriate correction region by a correction operation, or changes the display mode of the specific region according to the reliability of the calculation of oxygen subsaturation. Note that the correction image is preferably a color image consisting of a B1 image signal, a G1 image signal, and an R1 image signal, but may be other images.

In addition, when residue, bleeding, fat, residue, or mucus is thinly layered on the mucosa, or when it is difficult to determine whether correction processing can be performed appropriately because it is difficult to see on the white light equivalent image, it is preferable for the display mode control unit 65 to generate a correction image as follows, regardless of the level of reliability described below. For example, in order to make it easier to see residue, bleeding, fat, residue, and mucus, it is preferable for the display mode control unit 65 to generate a correction image in which the saturation of an image such as a white light equivalent image is emphasized. Also, in order to make dark areas such as the lumen where the signal strength is insufficient darker, it is preferable for the display mode control unit 65 to generate a correction image in which the brightness of dark areas is reduced.

The display mode of the correction image may be changed according to the reliability. Specifically, the display mode control unit 65 changes the display mode of the correction image 80 so as to emphasize the difference between a low reliability region, which has a low reliability for the calculation of oxygen saturation, and a high reliability region, which has a high reliability. The reliability represents the calculation accuracy of the oxygen saturation in each pixel, and the higher the reliability, the better the calculation accuracy of the oxygen saturation. A low reliability region is a region where the reliability is less than a reliability threshold. A high reliability region is a region where the reliability is equal to or greater than the reliability threshold. By emphasizing the difference between the low reliability region and the high reliability region in the correction image, it is possible to avoid the low reliability region and enter the high reliability region inside a specific region.

The reliability is calculated by the reliability calculation unit 66. Specifically, the reliability calculation unit 66 calculates at least one reliability that affects the calculation of oxygen saturation based on the B1 image signal, G1 image signal, and R1 image signal contained in the first illumination light image, or the B2 image signal, G2 image signal, and R2 image signal contained in the second illumination light image. The reliability is expressed, for example, as a decimal between 0 and 1. When multiple types of reliability are calculated by the reliability calculation unit 66, it is preferable to adopt the minimum reliability among the multiple types of reliability as the reliability of each pixel.

For example, as shown in FIG. 28, for brightness values that affect the accuracy of oxygen saturation calculation, the reliability of the brightness value of the G2 image signal outside a certain range Rx is lower than the reliability of the brightness value of the G2 image signal within the certain range Rx. Outside the certain range Rx includes high brightness values such as halation, as well as extremely small brightness values such as dark areas. In this way, outside the certain range Rx, the accuracy of oxygen saturation calculation is low, and the reliability is accordingly low. Note that the reliability may be calculated using the brightness value of the G1 image signal instead of the G2 image signal.

Disturbances that affect the accuracy of oxygen saturation calculation include at least bleeding, fat, residue, mucus, or residual fluid, and these disturbances also affect the reliability. As for bleeding, which is one of the disturbances, as shown in FIG. 29, the reliability is determined according to the distance from the defined line DFX on a two-dimensional plane consisting of the vertical axis ln (B2/G2) and the horizontal axis ln (R2/G2). Here, the reliability decreases as the coordinates plotted on the two-dimensional plane based on the B2 image signal, G2 image signal, and R2 image signal are farther away from the defined line DFX. For example, the reliability decreases as the coordinates plotted on the two-dimensional plane are located to the lower right. In FIG. 29, ln represents the natural logarithm. B2/G2 represents the signal ratio between the B2 image signal and the G2 image signal, and R2/G2 represents the signal ratio between the R2 image signal and the G2 image signal.

As shown in FIG. 30, the reliability of fat, residue, residual liquid, and mucus contained in the disturbance is determined according to the distance from the defined line DFY on a two-dimensional plane consisting of the vertical axis ln (B1/G1) and the horizontal axis ln (R1/G1). Here, the reliability decreases as the coordinates plotted on the two-dimensional plane based on the B1, G1, and R1 image signals are farther away from the defined line DFY. For example, the reliability decreases as the coordinates plotted on the two-dimensional plane are further to the lower left. In FIG. 30, ln represents the natural logarithm. B1/G1 represents the signal ratio between the B1 and G1 image signals, and R1/G1 represents the signal ratio between the R2 and G2 image signals.

As one method for the display mode control unit 65 to emphasize the difference between the low reliability region and the high reliability region, the display mode control unit 65 makes the saturation of the low reliability region 82a higher than that of the high reliability region 82b, as shown in FIG. 31. This makes it easier for the user to select the high reliability region 82b as the specific region 81, avoiding the low reliability region 82a. In addition, the display mode control unit 65 lowers the luminance of the dark area BA of the low reliability region 82a. This makes it easier to avoid the dark area BA when selecting the specific region 81. A dark area is a dark area with a luminance value equal to or lower than a certain value. Note that the low reliability region 82a and the high reliability region 82b may be of opposite colors.

The display mode control unit 65 preferably at least either superimposes area highlighting lines on the low reliability area or displays the low reliability area 82a in monochrome. For example, as shown in FIG. 32, it is preferable to superimpose diagonal lines at regular intervals as area highlighting lines on the low reliability area 82a. This makes it easier for the user to select the high reliability area 82b as the specific area 81, avoiding the low reliability area 82a. Note that although the area highlighting lines are diagonal lines at regular intervals, the interval between the diagonal lines may be varied depending on the reliability.

It is preferable that the display mode control unit 65 changes the display mode of the specific region according to the reliability within the specific region. Before a correction operation is performed in the correction mode, the first correction determination unit 67 determines whether or not the correction process can be performed properly based on the reliability within the specific region. If the number of valid pixels with a reliability equal to or greater than the reliability threshold for pixels within the specific region is equal to or greater than a certain value, the first correction determination unit 67 determines that the correction process can be performed properly. On the other hand, if the number of valid pixels for pixels within the specific region is less than the certain value, it determines that the correction process cannot be performed properly. Note that it is preferable that the first correction determination unit 67 performs a determination each time an image is acquired and a reliability is calculated until a correction operation is performed. The cycle for making the determination may be changed as appropriate.

It is preferable that the display mode control unit 65 differentiates the display mode of the specific region when the first correction determination unit 67 determines that the correction process can be performed properly from the display mode of the specific region when the first correction determination unit 67 determines that the correction process cannot be performed properly. For example, as shown in FIG. 33(A), when the first correction determination unit 67 determines that the correction process can be performed properly, the specific region 81 is highlighted in a first color. On the other hand, as shown in FIG. 33(B), when the first correction determination unit 67 determines that the correction process cannot be performed properly, the specific region 81 is highlighted in a second color different from the first color. It is preferable that the first and second colors are opposite colors to make it easier to understand whether the correction process is proper or not.

On the other hand, when a correction operation is performed in the correction mode, the second correction determination unit 68 determines whether or not the correction process can be performed appropriately based on the reliability within the specific area at the time when the correction operation is performed. The second correction determination unit 68 performs a determination in the same manner as the first correction determination unit 67. The determination notification unit 69 notifies the user of the determination of the second correction determination unit 68.

When the second correction judgment unit 68 judges that the correction process can be performed properly, the judgment notification unit 69 notifies the user that the correction process can be performed properly. For example, as shown in FIG. 34, a message MS1 such as "Correction process will be performed properly" is displayed on the extended display 18. In this case, the table correction unit 63 performs table correction process based on the specific dye concentration in the specific region as the correction process.

On the other hand, when the second correction judgment unit 68 judges that the correction process cannot be performed properly, the judgment notification unit 69 issues a notice that the correction process cannot be performed properly and therefore another correction operation is necessary. For example, as shown in FIG. 35, a message MS2 such as "Saturation correction operation is required" is displayed on the extended display 18. In this case, it is preferable that the judgment notification unit 69 notifies an operation guidance GD for performing a proper table correction process in addition to or instead of the message MS2. For example, it is preferable to display an operation guidance GD such as "Avoid dark areas" on the extended display 18 as the operation guidance GD. Other operation guidance includes an operation guidance such as "Avoid bleeding, residual fluid, fat, etc.".

Next, a series of steps in the correction mode will be explained with reference to the flowchart in FIG. 36. The user operates the mode change switch 12f to switch to the correction mode. When the correction mode is switched to, a correction image 80 is displayed on the extended display 18, and a specific region 81 is also displayed. In addition, based on the specific dye concentration of a specific dye other than blood hemoglobin contained in the specific region 81, a correction process for calculating oxygen saturation becomes executable.

In the correction mode, in order to make it easier for the user to select a specific region, at least one of the following is performed: the display mode of the correction image 80 is changed, or the display mode of the specific region 81 is changed depending on the reliability of the oxygen saturation calculation. When an appropriate correction region that is less affected by external disturbances that affect the oxygen saturation calculation falls within the specific region 81, the user performs a correction operation using the processor-side user interface 16 or the scope-side user interface 19. Correction processing is performed based on the specific dye concentration in the specific region at the time the correction operation is performed. When the correction processing is completed, the mode can be switched to the oxygen saturation mode manually or automatically.

[Second embodiment]
In the second embodiment, instead of the LEDs 20a to 20e shown in the first embodiment, a broadband light source such as a xenon lamp and a rotary filter may be used to illuminate the observation target. In this case, as shown in Fig. 37, in the endoscope system 100, a broadband light source 102, a rotary filter 104, and a filter switching unit 105 are provided in the light source device 13 instead of the LEDs 20a to 20e. Also, in the imaging optical system 31, a monochrome imaging sensor 106 without a color filter is provided instead of the color imaging sensor 36. The rest is the same as the endoscope system 10 described above.

The broadband light source 102 is a xenon lamp, a white LED, or the like, and emits white light with a wavelength range from blue to red. The rotating filter 104 has an inner filter 108 provided on the inside and an outer filter 109 provided on the outside (see FIG. 38). The filter switching unit 105 moves the rotating filter 104 in the radial direction, and when the normal mode is set by the mode switching switch 12f, the inner filter 108 of the rotating filter 104 is inserted into the optical path of the white light, and when the oxygen saturation mode or correction mode is set, the outer filter 109 of the rotating filter 104 is inserted into the optical path of the white light.

As shown in FIG. 38, the inner filter 108 is provided with a B1 filter 108a that transmits the purple light V and the second blue light BS of the white light along the circumferential direction, a G filter 108b that transmits the green light G of the white light, and an R filter 108c that transmits the red light R of the white light. Therefore, in the normal mode, the purple light V, the second blue light BS, the green light G, and the red light R are alternately irradiated onto the observation object by the rotation of the rotating filter 104.

The outer filter 109 is provided with, in the circumferential direction, a B1 filter 109a that transmits the first blue light BL of the white light, a B2 filter 109b that transmits the second blue light BS of the white light, a G filter 109c that transmits the green light G of the white light, an R filter 109d that transmits the red light R of the white light, and a B3 filter 109e that transmits the blue-green light BG of the white light in the wavelength band B3. Therefore, in the oxygen saturation mode, the first blue light BL, the second blue light BS, the green light G, the red light R, and the blue-green light BG are alternately irradiated onto the observation object by rotating the rotating filter 104.

In the endoscope system 100, in normal mode, the object to be observed is imaged by the monochrome image sensor 106 each time the object is illuminated with the purple light V, the second blue light BS, the green light G, and the red light R. This results in a Bc image signal, a Gc image signal, and an Rc image signal. Then, based on these three color image signals, a white light image is generated in the same manner as in the first embodiment.

On the other hand, in the oxygen saturation mode or correction mode, the object to be observed is imaged by the monochrome image sensor 106 each time the object to be observed is illuminated with the first blue light BL, the second blue light BS, the green light G, the red light R, and the blue-green light BG. This results in a B1 image signal, a B2 image signal, a G2 image signal, an R2 image signal, and a B3 image signal. Based on these five color image signals, the oxygen saturation mode or correction mode is performed in the same manner as in the first embodiment. However, in the second embodiment, the signal ratio ln(B3/G2) is used instead of the signal ratio ln(B3/G3).

In the first and second embodiments described above, a table correction process is performed to correct the oxygen saturation calculation table 73 as a correction process for calculating oxygen saturation in the correction mode. However, a calculated value correction process may be performed to add or subtract a correction value obtained from a specific dye concentration to the oxygen saturation calculated based on the oxygen saturation calculation table 73.

Specifically, in the calculation value correction process, a two-dimensional coordinate 90 shown in Fig. 39 is used to calculate a correction value used to correct the oxygen saturation calculated based on the oxygen saturation calculation table 73. The vertical axis of the two-dimensional coordinate is a specific calculation value obtained based on the B1, G2, R2, and B3 image signals, and the horizontal axis is Ln(R2/G2). The specific calculation value is determined by the following formula A).
Formula A) B1/G2×cosφ−B3/G2×sinφ

In the two-dimensional coordinate system 90, a reference line 91 showing the distribution of predetermined reference baseline information and an actual measurement line 92 showing the distribution of actual baseline information obtained by imaging an actual observation target are shown. The difference value ΔZ between the reference line 91 and the actual measurement line 92 is calculated as a correction value. In the calculation value correction process, the correction value is added to or subtracted from the oxygen saturation calculated based on the oxygen saturation calculation table 73. The reference baseline information is defined as information obtained in the absence of a specific dye and independent of oxygen saturation. Specifically, the reference baseline information is a value obtained by adjusting φ so that the above formula A) remains constant even when the oxygen saturation changes.

In addition, in the correction mode, instead of the correction process, a specific oxygen saturation calculation process may be performed in which the oxygen saturation is calculated according to the specific dye concentration based on at least the oxygen saturation image signal and the specific dye image signal. Specifically, the specific oxygen saturation calculation process uses three-dimensional coordinates 93 shown in FIG. 40. In the three-dimensional coordinates 93, the signal ratio ln(R2/G2) is assigned to the X-axis, the signal ratio ln(B1/G2) is assigned to the Y-axis, and ln(B3/G3) is assigned to the Z-axis. The curved surfaces CV0 to CV4 represent the oxygen saturation states corresponding to the specific dye concentrations "0" to "4" in the three-dimensional coordinates 93.

In the specific oxygen saturation calculation process, the signal ratios ln(R1 ^* /G1*), ln(B2*/G1 ^* ), and ln(B3 ^* /G3 ^* ) calculated based on the B1, ^G2 , R2 ^, B3, and G3 image signals are plotted on the three-dimensional coordinate 93 to calculate the oxygen saturation. The calculated oxygen saturation is an accurate value because it is not affected by the specific dye concentration.

In the first and second embodiments, an endoscope 12 is used, which is a flexible endoscope for the digestive tract, but an endoscope that is a rigid endoscope for laparoscopy may also be used. When using a rigid endoscope, an endoscope system 200 shown in FIG. 41 is used. The endoscope system 200 includes an endoscope 201, a light source device 13, a processor device 14, a display 15, a processor-side user interface 16, an extended processor device 17, and an extended display 18. In the following, parts of the endoscope system 200 that are common to the first and second embodiments will be omitted, and only the different parts will be described.

The endoscope 201 is used in laparoscopic surgery and the like, is rigid and elongated, and is inserted into the subject. The endoscope 201 illuminates the observation object with illumination light supplied from the light source device 13 via the light guide 202. The endoscope 201 also receives reflected light from the observation object being illuminated with the illumination light. The camera head 203 is attached to the endoscope 201, and captures an image of the observation object based on the reflected light guided from the endoscope 201. The image signal captured by the camera head 203 is transmitted to the processor device 14.

In the normal mode, the light source device 13 supplies white light including purple light V, second blue light BS, green light G, and red light R to the endoscope 201. In the oxygen saturation mode and correction mode, the light source device 13 supplies mixed light including the first blue light BL, second blue light BS, green light G, and red light R to the endoscope 12, as shown in FIG. 42.

As shown in FIG. 43, the camera head 203 includes dichroic mirrors 205, 206, and 207, and monochrome image sensors 210, 211, 212, and 213. The dichroic mirror 205 reflects the purple light V and the second blue light BS of the reflected mixed light from the endoscope 201, and transmits the first blue light BL, the green light G, and the red light R. As shown in FIG. 44, the purple light V or the second blue light BS reflected by the dichroic mirror 205 is incident on the image sensor 210. In the normal mode, the image sensor 210 outputs a Bc image signal based on the incidence of the purple light V and the second blue light BS, and in the oxygen saturation or correction mode, the image sensor 210 outputs a B2 image signal based on the incidence of the second blue light BS.

The dichroic mirror 206 reflects the first blue light BL and transmits the green light G and red light R out of the light transmitted through the dichroic mirror 205. As shown in FIG. 45, the first blue light BL reflected by the dichroic mirror 206 is incident on the image sensor 211. The image sensor 211 stops outputting image signals in the normal mode, and outputs a B1 image signal based on the incidence of the first blue light BL in the oxygen saturation or correction mode.

The dichroic mirror 207 reflects the green light G and transmits the red light out of the light that has passed through the dichroic mirror 206. As shown in FIG. 46, the green light G reflected by the dichroic mirror 207 is incident on the image sensor 212. In the normal mode, the image sensor 212 outputs a Gc image signal based on the incidence of the green light G, and in the oxygen saturation or correction mode, outputs a G2 image signal based on the incidence of the green light G.

As shown in FIG. 47, the red light R transmitted through the dichroic mirror 207 is incident on the image sensor 213. In the normal mode, the image sensor 213 outputs an Rc image signal based on the incidence of the red light R, and in the oxygen saturation or correction mode, the image sensor 213 outputs an R2 image signal based on the incidence of the red light R.

In the first and second embodiments, the B1 image signal, the G2 image signal, and the R2 image signal, which contain image information of the wavelength band B1 whose reflection spectrum changes with the change in the oxygen saturation of blood hemoglobin, are used to calculate the oxygen saturation. However, other image signals may be used instead of the B1 image signal. For example, as shown in FIG. 48, instead of the B1 image signal, an Rk image signal containing image information of the wavelength band Rx whose reflection spectrum changes with the change in the oxygen saturation of blood hemoglobin may be used. The wavelength band Rx is preferably 680 nm±10 nm. As shown in FIG. 49, the Rk image signal has a "medium to small" oxygen saturation dependency, a "small" blood concentration dependency, and a "small" yellow pigment dependency. Therefore, even in a situation where a yellow pigment is present in the observation subject, the oxygen saturation can be accurately calculated with only three image signals, the G2 image signal, the R2 image signal, and the Rk image signal.

When using an endoscope (see FIG. 41), which is a rigid endoscope for laparoscopy, unlike the endoscope 201 (see FIG. 43) which uses four monochrome image sensors 210-213 to capture an image of the observation target, the endoscope may capture an image of the observation target using another imaging method. As shown in FIG. 50, the endoscope 300 is a two-sensor type endoscope for the abdominal cavity having one color image sensor 301 and one monochrome image sensor 302. In addition to the color image sensor 301 and monochrome image sensor 302, the camera head 303 of the endoscope 300 is provided with a dichroic mirror 305 which transmits a portion of the light incident on the camera head 303 and reflects the remaining light.

When the endoscope 300 is used, the light emission control of the light source device 13 switches between a white frame that simultaneously emits the first blue light BL, the second blue light BS, the green light G, and the red light R (see FIG. 51 (A)) and a green frame that emits only the green light G (see FIG. 51 (B)) according to a specific emission pattern.

As shown in FIG. 52, when the first blue light BL, the second blue light BS, the green light G, and the red light R are simultaneously emitted in a white frame (see FIG. 52 (A)), the first blue light BL of the light incident on the camera head 303 is reflected by the dichroic mirror 305 (see FIG. 52 (B)). The first blue light BL reflected by the dichroic mirror 305 is incident on the monochrome image sensor 302 (see FIG. 52 (C)). The monochrome image sensor 302 outputs a B1 image signal having a pixel value corresponding to the incident first blue light BL (see FIG. 52 (D)).

Also, as shown in FIG. 53, in the white frame, the second blue light BS, green light G, and red light R transmitted by the dichroic mirror 305 enter the color image sensor 301 (see FIG. 53(C)). In the color image sensor 301, the B pixels output B2 image signals having pixel values corresponding to the light of the second blue light transmitted through the B color filter BF. Also, the G pixels output G2 image signals having pixel values corresponding to the light of the green light G transmitted through the G color filter GF. The R pixels output R2 image signals having pixel values corresponding to the light of the red light R transmitted through the R color filter RF.

On the other hand, as shown in FIG. 54, when only green light G is emitted in the green frame (see FIG. 54 (A)), the green light G incident on the camera head 303 is transmitted by the dichroic mirror 305. The green light G transmitted by the dichroic mirror 305 is incident on the color image sensor 301. The B pixels of the color image sensor 301 output a B3 image signal having a pixel value corresponding to the light of the green light G that has transmitted through the B color filter BF. The G pixels output a G3 image signal having a pixel value corresponding to the light of the green light G that has transmitted through the G color filter GF. Note that in the green frame, the image signal output from the monochrome image sensor 302 and the image signal output from the R pixels of the color image sensor 301 are not used in the subsequent processing steps.

As shown in FIG. 55, in a white frame, the monochrome image sensor 302 outputs a B1 image signal, and the color image sensor 301 outputs a B2 image signal, a G2 image signal, and an R2 image signal, and these B1, B2, G2, and R2 image signals are used in the subsequent processing steps. On the other hand, in a green frame, the color image sensor 301 outputs a B3 image signal and a G3 image signal, and these are used in the subsequent processing steps.

As shown in FIG. 56, the image signal output from the camera head 303 is sent to the processor device 14, and the data that has undergone various processes in the processor device 14 is sent to the extended processor device 17. When the endoscope 300 is used, taking into consideration the processing load on the processor device 14, low-load processing is performed in the processor device 14 for the processes performed in the oxygen saturation mode and correction mode, and then high-load processing is performed in the extended processor device 17. Of the processes performed in the oxygen saturation mode and correction mode, the processes performed by the processor device 14 are mainly performed by an FPGA (Field-Programmable Gate Array) and are therefore referred to as FPGA processing. On the other hand, the processes performed by the extended processor device 17 are referred to as PC processing, as the extended processor device is performed by a PC (Personal Computer).

If the endoscope 300 is provided with an FPGA (not shown), the FPGA processing may be performed by the FPGA of the endoscope 300. In the following, the FPGA processing and PC processing in the correction mode will be explained, but it is preferable to share the processing load by dividing into FPGA processing and PC processing in the oxygen saturation mode as well.

When the endoscope 300 is used and the white frame W and the green frame Gr are controlled to emit light according to a specific light emission pattern, as shown in FIG. 57, the specific light emission pattern is to emit two frames of the white frame W, followed by two frames of blank frames BL in which no light is emitted from the light source device 13. Then, two frames of the green frame Gr are emitted, followed by several blank frames of two or more frames (e.g., seven frames). Then, two frames of the white frame W are emitted again. The above specific light emission pattern is repeated. Note that the white frame W and the green frame Gr are emitted in at least the correction mode as in the above specific light emission pattern, and in the oxygen saturation mode, only the white frame W may be emitted without emitting the green frame Gr.

Hereinafter, in order to distinguish between each emission frame that emits light in a specific emission pattern, the first of the first two white frames will be referred to as white frame W1, and the next white frame as white frame W2. The first of the two green frames will be referred to as green frame Gr1, and the next green frame as green frame Gr2. And the first of the last two white frames will be referred to as white frame W3, and the next white frame as white frame W4.

The image signals for the correction mode obtained in the white frame W1 (B1 image signal, B2 image signal, G2 image signal, R2 image signal, B3 image signal, G3 image signal) are referred to as the image signal set W1. Similarly, the image signals for the correction mode obtained in the white frame W2 are referred to as the image signal set W2. The image signals for the correction mode obtained in the green frame Gr1 are referred to as the image signal set Gr1. The image signals for the correction mode obtained in the green frame Gr2 are referred to as the image signal set Gr2. The image signals for the correction mode obtained in the white frame W3 are referred to as the image signal set W3. The image signals for the correction mode obtained in the white frame W4 are referred to as the image signal set W4. The image signals for the oxygen saturation mode are the image signals included in the white frames (B1 image signal, B2 image signal, G2 image signal, R2 image signal).

The reason why the blank frames between the white frame W and the green frame W need only be about two frames is that it is sufficient to turn off all light other than the green light G, whereas the reason why the blank frames between the green frame G and the white frame W need to be two or more frames is because it is necessary to take time to stabilize the light emission state when light other than the green light G starts to be turned on.

In the FPGA processing, as shown in FIG. 58, a valid pixel determination is performed for the pixels of all image signals included in each image signal set W1, W2, Gr1, Gr2, W3, and W4 to determine whether or not they can be processed accurately in the oxygen saturation mode or the correction mode. The valid pixel determination is performed based on the pixel values in 16 regions of interest ROIs provided in the center of the image, as shown in FIG. 59. Specifically, for each pixel in the ROI, if the pixel value falls within the range between the upper threshold and the lower threshold, it is determined to be a valid pixel. The valid pixel determination is performed for the pixels of all image signals included in the image signal set. In addition, the upper threshold or the lower threshold is preset according to the sensitivity of the B pixel, the G pixel, and the R pixel of the color image sensor 301, or the sensitivity of the monochrome image sensor 302.

Based on the above valid pixel determination, the number of valid pixels, the sum of the pixel values of the valid pixels, and the squared sum of the pixel values of the valid pixels are calculated for each ROI. The number of valid pixels, the sum of the pixel values of the valid pixels, and the squared sum of the pixel values of the valid pixels for each ROI are output to the extended processor device 17 as valid pixel data W1, W2, Gr1, Gr2, W3, and W4, respectively. FPGA processing, like valid pixel determination, is a calculation process for image signals of the same frame, and has a lighter processing load than calculation process for inter-frame image signals with different emission frames, like PC processing described later. Note that the valid pixel data W1, W2, Gr1, Gr2, W3, and W4 correspond to the data determined to be valid pixels for all image signals included in the image signal sets W1, W2, Gr1, Gr2, W3, and W4, respectively.

In PC processing, same-frame PC processing is performed on image signals of the same frame from among the effective pixel data W1, W2, Gr1, Gr2, W3, and W4, and inter-frame PC processing is performed on image signals of different frames. In same-frame PC processing, the average pixel value, standard deviation of pixel values, and effective pixel rate within the ROI are calculated for all image signals contained in each effective pixel data. The average pixel value and other values within the ROI obtained by this same-frame PC processing are used in calculations to obtain specific results in oxygen saturation mode or correction mode.

As shown in FIG. 60, in inter-frame PC processing, of the effective pixel data W1, W2, Gr1, Gr2, W3, and W4 obtained by FPGA processing, those with a close time interval between the white frame and the green frame are used, and the rest are not used in inter-frame PC processing. Specifically, the pair of effective pixel data W2 and effective pixel data Gr1, and the pair of effective pixel data Gr2 and effective pixel data W3 are used in inter-frame PC processing. The other effective pixel data W1 and W4 are not used in inter-frame PC processing. Note that by pairing image signals with a close time interval, it is possible to perform accurate inter-frame PC processing with no positional deviation between pixels.

As shown in FIG. 61, in inter-frame PC processing using a pair of valid pixel data W2 and valid pixel data Gr1, reliability calculation and specific dye concentration calculation are performed, and in inter-frame PC processing using a pair of valid pixel data Gr2 and valid pixel data W3, reliability calculation and specific dye concentration calculation are similarly performed. Then, a specific dye concentration correlation determination is performed based on the calculated specific dye concentration.

In calculating the reliability, the reliability is calculated for each of the 16 ROIs. The method of calculating the reliability is the same as the calculation method by the reliability calculation unit 66 described above. For example, it is preferable to lower the reliability when the brightness value of the G2 image signal is outside the certain range Rx and the reliability when the brightness value of the G2 image signal is within the certain range Rx (see FIG. 28). In the case of a pair of effective pixel data W2 and effective pixel data Gr1, a total of 32 reliability is calculated by calculating the reliability for each ROI for the G2 image signal included in each effective pixel data. Similarly, a total of 32 reliability is calculated for a pair of effective pixel data Gr2 and effective pixel data W3. When the reliability is calculated, if there is an ROI with low reliability or if the average reliability value of each ROI does not meet a predetermined value, an error judgment is made regarding the reliability. The result of the error judgment regarding the reliability is notified to the user by displaying it on the extended display 18, etc.

In the specific dye concentration calculation, the specific dye concentration is calculated for each of the 16 ROIs. The method of calculating the specific dye concentration is the same as the calculation method by the specific dye concentration calculation unit 62 described above. For example, the B1 image signal, G2 image signal, R2 image signal, B3 image signal, and G3 image signal contained in the effective pixel data W2 and the effective pixel data Gr1 are used, and the specific dye concentrations corresponding to the signal ratios ln(B1/G2), ln(G2/R2), and ln(B3/G3) are calculated by referring to the specific dye concentration calculation table 62a. In this way, a total of 16 specific dye concentrations PG1 are calculated for each ROI. Note that in the case of a pair of effective pixel data Gr2 and effective pixel data W3, a total of 16 specific dye concentrations PG2 are calculated for each ROI in the same manner.

Once the specific dye concentrations PG1 and PG2 have been calculated, a correlation value between the specific dye concentrations PG1 and PG2 is calculated for each ROI. It is preferable to calculate the correlation value for each ROI in the same position. If there are a certain number or more ROIs with correlation values lower than a predetermined value, it is determined that movement has occurred between frames, and an error determination is made regarding the movement. The result of the error determination regarding the movement is notified to the user by displaying it on the extended display 18, for example.

If no error is detected in the error determination regarding the movement, one specific dye concentration is calculated from a total of 32 specific dye concentrations PG1 and PG2 using a specific estimation method (e.g., a robust estimation method). The calculated specific dye concentration is used in the correction process in the correction mode. The correction process in the correction mode is the same as described above, such as table correction process.

As shown in Figures 62 and 63, the white balance adjustment tool for endoscopes (hereinafter referred to as the adjustment tool) 400 is a tool used for adjusting the white balance of an endoscope. The adjustment tool 400 is box-shaped and has an insertion section 401, a lid section 402, and an outer peripheral section 403 on the outside, and an internal space 406 (see Figure 67) on the inside. The insertion section 401 has two insertion holes 405 into which the tip section 12d (see Figure 64) of the endoscope 12 is inserted. The insertion hole 405 is composed of insertion holes 405a and 405b.

The tip 12d inserted from the insertion hole 405 in the insertion direction X is advanced to an appropriate position in the internal space 406, and an image of the chart 408 (see FIG. 66) provided on the internal space 406 side of the lid 402 is captured. White balance adjustment is performed using the endoscopic image of the chart 408. Note that while the case where the tip 12d of the endoscope 12 is inserted into the adjustment jig 400 will be described, the tip of the endoscope 201 can also be inserted into the adjustment jig 400 in a similar manner to perform white balance adjustment.

The inner diameters of the two insertion holes 405 are different so that they can accommodate the tip 12d of the endoscope 12 having a different outer diameter. It is preferable that the inner diameter of the insertion hole 405 is as small as possible while leaving a suitable gap between the tip 12d and the insertion hole 405 when the tip 12d is inserted through the insertion hole 405, in order to ensure ease of insertion and a range of motion for adjusting the position of the tip 12d when photographing the chart 408. This ensures that the insertion section 401 can guide the insertion of the tip 12d and adjust the position of the tip 12d, and makes it possible to suppress light entering the internal space 406 through the gap between the tip 12d and the insertion hole 405.

It is preferable that the inner diameter of one of the two insertion holes 405, insertion hole 405a, is 12 mm, and the inner diameter of the other insertion hole 405b is 7 mm. As shown in FIG. 64, if the outer diameter of the tip 12d of the endoscope 12 is thick and is in the range of more than 5 mm and not more than 10 mm, the tip 12d is inserted into the insertion hole 405a. As shown in FIG. 65, if the outer diameter of the tip 12d of the endoscope 12 is thin and is not more than 5 mm, the tip 12d is inserted into the insertion hole 405b.

The insertion section 401 may be provided with guide sections 401a and 401b corresponding to the insertion holes 405a and 405b. The guide section 401a has a function of guiding the user when inserting the tip section 12d into the insertion hole 405. The guide section 401a may be cylindrical with the longitudinal direction being the insertion direction X. By forming the cylindrical guide section 401a, when the user inserts the tip section 12d, a part of the bottom, outer surface, etc. of the tip section 12d is naturally inserted while contacting a part of the bottom and inner surface of the cylindrical insertion hole 405a, and the tip section 12d can be more appropriately inserted toward the lid section 402. In addition, the user can more easily place the tip section 12d of the endoscope 12 in an appropriate position in the internal space 406. The guide section 401b is similar to the guide section 401a.

As shown in Figures 66 and 67, the guide portions 401a and 401b may be tapered such that the inner diameter increases slightly from the entrance of the insertion holes 405a and 405b toward the lid portion 402 along the insertion direction X. The inner diameter of the entrance of the insertion hole 405a is 11 mm, but the inner diameter is 12 mm at the exit of the insertion hole 405a, which is the position in the guide portion 401a closest to the lid portion 402 along the insertion direction X. Similarly, the inner diameter of the entrance of the insertion hole 405b is 6 mm, but the inner diameter is 7 mm at the exit of the insertion hole 405b, which is the position in the guide portion 401b closest to the lid portion 402 along the insertion direction X.

The internal space 406 is divided according to the insertion holes 405a and 405b. The partition 403e divides the internal space 406 into two spaces: an internal space 406a connected to the insertion hole 405a, and an internal space 406b connected to the insertion hole 405b. For example, when the tip 12d is inserted into the insertion hole 405a, the light from the insertion hole 405b is blocked by the partition 403e and can be prevented from entering the internal space 406a. Even when two insertion holes 405a and 405b are provided to accommodate various types of endoscopes with different outer diameters of the tip, as in this embodiment, the chart 408 can be photographed under appropriate shooting conditions because light from an unused internal space does not enter the internal space in use where the tip 12d is inserted.

As shown in FIG. 68, the cover 402 is a plate with a uniform thickness, and the surface facing the internal space 406 is the inner surface 402b, and the surface opposite the inner surface 402b is the outer surface 402a. The chart 408 is provided on the inner surface 402b. It is preferable that the cover 402 and the outer peripheral portion 403 are separably connected. By releasing the connection between the cover 402 and the outer peripheral portion 403, only the cover 402 can be removed from the adjustment jig 400. Therefore, it is possible to clean, disinfect, etc. the inside of the adjustment jig 400. In addition, it is possible to clean, replace, etc. only the cover 402, and it is possible to select a cover having a chart 408 corresponding to the type of endoscope 12 for which white balance adjustment is to be performed from among multiple cover parts 402, and replace the cover part 402 that has been used until now.

The chart 408 is preferably made of a material that does not reflect light from the endoscope 12. This can suppress the occurrence of flare FLA (see FIG. 82) on the chart 40, and therefore, in the error judgment (judgment regarding the positional relationship between the chart 408 and the tip 12d of the endoscope) described below, it is possible to avoid misrecognizing the flare as the mark 407a, etc., and also to recognize the color of the mark 407a as the correct color. In addition, since the light from the endoscope 12 is no longer reflected by the chart 408, it is preferable that the chart 408 is made of matte paper such as Yupo material. In addition, it is preferable that the chart 408 has a structure that is less likely to cause localized reflection due to diffuse reflection of light, such as a micropoid structure.

It is preferable that the lid portion 402 and the outer peripheral portion 403 are configured so as to be easily separated and connected. The lid portion 402 and the outer peripheral portion 403 may be connected so as to be separable by magnetic force and/or engagement. That is, the lid portion 402 and the outer peripheral portion 403 may be connected by using a magnet, or at least a portion of each of the lid portion 402 and the outer peripheral portion 403 may be connected by fitting a convex portion into a concave portion, or a magnet may be used and the connection may be made by fitting a convex portion into a concave portion.

In this embodiment, the outer peripheral portion 403 is provided with a magnet insertion portion 403f in which a magnet is fitted inside. The lid portion 402 has a property that at least the convex portion 402c corresponding to the magnet insertion portion 403f is attracted by a magnet to a part of the concave portion 403g of the outer peripheral portion 403 corresponding to the inside of the magnet insertion portion 403f. Therefore, the lid portion 402 and the outer peripheral portion 403 can be easily connected by bringing the lid portion 402 close to the outer peripheral portion 403. In addition, when removing the lid portion 402, the lid portion 402 can be easily removed from the outer peripheral portion 403 with only a force that resists the magnetic force from the magnet insertion portion 403f. In addition, the concave portion 403g of the outer peripheral portion 403 is provided on each of the upper, lower, left, and right outer circumferences of the outer peripheral portion 403, and the convex portion 402c of the lid portion 402 is also provided on each of the upper, lower, left, and right outer circumferences of the lid portion 402. Then, by fitting the protrusion 402c of the lid 402 into the recess 403g of the outer periphery 403, the entire surfaces of the two internal spaces 406a, 406b of the adjustment jig 400 can be covered.

As shown in FIG. 69(a), the outer surface 402a of the lid portion 402 constitutes the outer surface of the adjustment jig 400. As shown in FIG. 69(b), the inner surface 402b of the lid portion 402 faces the internal space 406 and constitutes the inner surface of the adjustment jig 400. Four marks 407a, 407b, 407c, and 407d are provided on the inner surface 402b. These four marks 407a, 407b, 407c, and 407d constitute the chart 408.

The marks 407a to 407d may be formed directly on the inner surface 402b of the lid substrate 402e, or the marks 407a to d may be formed on a substrate constituting the chart 408. The chart 408 having the marks 407a to d may be disposed on the inner surface 402b. In this case, the shape of the chart 408 may be the same as or different from that of the lid 402. In this embodiment, the chart 408 has a different shape from the lid 402, and is rectangular without any convex portion like the convex portion 402c.

The chart 408 has a vertical plane perpendicular to the insertion direction X of the tip 12d of the endoscope, and is positioned so as to be inclined toward the insertion section 401. The vertical plane is a virtual plane. This makes it possible to appropriately photograph the chart 408 and appropriately adjust the white balance when each of the multiple types of endoscopes performs white balance adjustment. Note that the multiple types of endoscopes referred to here are types that are distinguished by the angle of the optical axis direction of the imaging optical system 31 relative to the axial direction of the tip.

The types of endoscopes include a direct-viewing scope in which the optical axis direction of the imaging optical system 31 is parallel to the axial direction of the tip 12d, and an oblique-viewing scope in which the optical axis direction of the imaging optical system 31 is at a predetermined angle to the axial direction of the tip 12d. If the endoscope is a rigid scope that is straight, like the endoscope 201, it is the same as the axial direction of the tip 12d. Even if the endoscope 12 is a flexible scope, if the tip 12d is not curved, the type of endoscope 12 can be distinguished by the optical axis direction of the imaging optical system 31 relative to the axial direction of the tip 12d. Note that an oblique-viewing scope is sometimes called a side-viewing scope, etc.

As shown in FIG. 70(a), in the oblique mirror 411, the inclination angle α, which is the angle between the optical axis direction m of the imaging optical system 31 and the axial direction n of the tip surface 12k of the tip 12d, is, for example, 30°, 45°, etc. As shown in FIG. 70(b), in the direct mirror 412, the optical axis direction m of the imaging optical system 31 and the axial direction n of the tip 12d are parallel. With the direct mirror 412, an endoscopic image of a subject located axially forward of the tip 12d can be obtained, and with the oblique mirror 411, an endoscopic image of a subject located slightly forward and to the side as viewed from the axial direction of the tip 12d can be obtained.

In the adjustment jig 400, the chart 408 on which the marks 407a-d are appropriately arranged is arranged so as to be inclined at an appropriate angle with respect to a vertical plane perpendicular to the insertion direction X. In any of the multiple types of endoscopes 12 with various inclination angles α, when the marks 407a-d are photographed, the distance and angle between the tip 12d of the endoscope and the marks 407a-d will be within a specified range.

The degree of inclination of the chart 408 should be such that, when endoscopic images obtained by each of the multiple endoscopes 12 photographing the chart 408 are used for white balance adjustment, the distance and angle between the tip 12d and the marks 407a-d are within a specified range, and the color, brightness, distortion, etc. of the marks 407a-d caused by the inclination of the chart 408 do not cause problems in white balance adjustment. The degree of inclination of the chart 408 can be indicated by the inclination of the chart 408 with respect to a vertical plane perpendicular to the optical axis of the direct or oblique endoscope.

As shown in FIG. 71, in the direct mirror 412, the surface k including the surface of the chart 408 is arranged so as to have an inclination angle β1 with respect to a vertical plane l perpendicular to the optical axis direction m1 of the imaging optical system of the direct mirror 412. By the direct mirror 412 photographing the mark 407b at the inclination angle β1, it is possible to appropriately perform white balance adjustment based on the endoscopic image obtained by imaging the mark 17b. Also, as shown in FIG. 72, in the oblique mirror 411, the surface k of the chart 408 is arranged so as to have an inclination angle β2 with respect to a vertical plane l perpendicular to the optical axis direction m2 of the imaging optical system of the oblique mirror 411. By the oblique mirror 411 photographing the mark 407a at the inclination angle β2, it is possible to appropriately perform white balance adjustment based on the endoscopic image obtained by imaging the mark 407a.

In white balance mode, in which white balance adjustment is performed using the adjustment jig 400, an error determination is made as to whether or not an error has occurred regarding the positional relationship between the tip 12d of the endoscope 10 and the chart 408. In this embodiment, the error determination is made as to whether or not the tip 12d of the endoscope 10 inserted into the adjustment jig 400 is in an appropriate positional relationship with any of the marks 407a to 407d on the chart 408. When the tip 12d is in an appropriate positional relationship with any of the marks 407a to 407d on the chart 408, the white balance adjustment can be performed appropriately.

In this embodiment, in order to perform error determination, as shown in FIG. 73, the processor device 14 is provided with an image acquisition unit 420 and an error determination unit 421. The processor device 14 has programs related to various processes for the image acquisition unit 420 and the error determination unit 421 built into a memory (not shown). The functions of the image acquisition unit 420 and the error determination unit 421 are realized by the programs being executed by a control unit (not shown) configured by a processor.

The image acquisition unit 420 acquires an endoscopic image obtained by capturing an image of the chart 408 from the tip 12d of the endoscope. Specifically, when the white balance adjustment mode is in operation and the tip 12d of the endoscope is inserted into the adjustment jig 400, the tip 12d of the endoscope captures an image of one of the marks 407a to 407d of the chart 408 in the adjustment jig 400 (see, for example, Figures 71 and 72). The endoscopic image obtained by capturing the image is acquired by the image acquisition unit 420. Note that the image acquisition unit 420 and the error determination unit 421 may be provided in the processor device 14, and the endoscopic image displayed in the white balance mode may be displayed on the extended display 18.

The error determination unit 421 obtains information on a first determination area group including at least three first determination areas and information on a second determination area group that is provided outside the first determination area group and includes at least three second determination areas from the endoscopic image, and performs an error determination as to whether or not an error has occurred regarding the positional relationship between the chart 408 and the tip 12d of the endoscope based on the information on the first area determination group and the information on the second area determination group.

Specifically, as shown in FIG. 74, the first judgment region group has three first judgment regions Jm1 to Jm3. When the tip 12d of the endoscope is in an appropriate position in the white balance mode, the first judgment regions Jm1 to Jm3 are located in the area of the marks 407a to 407d of the chart 408. Brightness is used as information for the three first judgment regions Jm1 to Jm3, and in the endoscopic image, the brightness of the marks 407a, etc. is greater than the brightness of the background part of the chart 408 other than the marks 407a, etc. When the tip 12d of the endoscope is in an appropriate position and the brightness of the first judgment regions Jm1 to Jm3 is acquired from the endoscopic image, the brightness of all three first judgment regions Jm1 to Jm3 is equal to or greater than the first threshold value. Therefore, the error determination unit 421 determines that an error has occurred when there is at least one first judgment region in the three first judgment regions Jm1 to Jm3 whose brightness is less than the first threshold value. In this way, the calculation process for determining brightness is not a high-resource calculation process such as edge extraction, but a low-resource process that has traditionally been included in the processor unit 14, so the calculation load is small and it can be completed in a short time.

The second judgment region group also has three second judgment regions Jn1 to Jn3. In white balance mode, when the tip 12d of the endoscope is in an appropriate position, the second judgment regions Jn1 to Jn3 are located in the background. Brightness is used as information for the three second judgment regions, and in the endoscopic image, the brightness of the background is less than the brightness of the mark 407a and the like. When the brightness of the second judgment regions Jn1 to Jn3 is acquired from the endoscopic image with the tip 12d of the endoscope in an appropriate position, the brightness of all three second judgment regions Jn1 to Jn3 is less than the second threshold. Therefore, the error determination unit 421 determines that an error has occurred when at least one second judgment region among the three second judgment regions Jn1 to Jn3 has a brightness equal to or greater than the second threshold.

In this embodiment, in order for the error determination unit 421 to perform accurate error determination, it is preferable that the chart 408 on the endoscopic image has sufficient contrast between the marks 407a, etc. and the background. For example, it is preferable to configure the chart 408 so that the brightness of the mark 407a minus the sum of the brightness of the noise and the background ((brightness of the marks 407a, etc. - (noise + background))) is equal to or greater than random noise.

The layout shape of the first determination region group in the endoscopic image is an inverted triangle obtained by connecting the three first determination regions Jm1 to Jm3. On the other hand, the layout shape of the second determination region group in the endoscopic image is a triangle obtained by connecting the three second determination regions Jn1 to Jn3. The six first determination regions Jm1 to Jm3 and the six second determination regions Jn1 to Jn3 arranged in this manner make it possible to detect positional deviations of marks 407a, etc. in a 360° direction. For example, if the layout shape of the first determination region group is a triangle, it may be difficult to accurately determine an error if marks 407a, etc. are shifted downward on the chart.

Note that the first judgment regions Jm1-Jm3 and the second judgment regions Jn1-Jn3 are shown in FIG. 74 for the purpose of explaining the regions where endoscopic image information is obtained at those positions, and are not actually displayed on the display 15. Note that the shape of the first judgment regions Jm1-Jm3 or the second judgment regions Jn1-Jn3 is preferably rectangular. Also, taking into account the processing load, it is preferable that the size of the first judgment regions Jm1-Jm3 or the second judgment regions Jn1-Jn3 is as small as possible without affecting the brightness judgment based on the first threshold value or the second threshold value.

The error determination unit 421 determines whether an error has occurred based on the information of the first and second judgment area groups, as well as the information of the third judgment area. The third judgment area Jt is provided inside the first judgment area group and has a rectangular shape (as with the first and second judgment areas, it is not displayed on the display 15). Specifically, it is preferable that the information of the third judgment area Jt is a color tone. It is preferable that the error determination unit 421 performs an error determination when the color tone of the third judgment area Jt is outside a specific color tone range.

The specific color tone range is determined based on the color tone of the mark 407a, etc., and is preferably, for example, gray. Outside the specific color tone range, it is preferable that the saturation of the color tone of the mark 407a, etc. is outside a certain range. For example, if the specific color tone range is a certain saturation range based on gray, it is preferable that saturation away from gray, such as red and blue, is outside the specific color tone range. In reality, blood or the like may adhere to the tip 12d of the endoscope, and in such a case, it is expected that the third determination region Jt will be outside the specific color tone range. Note that an error determination may be made based only on the information of the third determination region Jt.

In addition, in the white balance mode, when an endoscopic image is displayed on the display 15, a guide area GA for guiding the position of any one of the marks 407a to 407d on the chart 408 is superimposed on the endoscopic image. The guide area GA is provided between the first and second determination area groups, and has a double circle consisting of an inner circle GAx and an outer circle GAy. In the white balance mode, the user operates the tip 12d of the endoscope so that the mark on the endoscopic image falls in the area between the inner circle GAx and the outer circle GAy (mark 407a is shown as an example in Figure 74).

Specifically, as shown in FIG. 74, if the brightness of all three first judgment areas is equal to or greater than the first threshold value and the brightness of all three second judgment areas is less than the second threshold value, the error judgment unit 421 does not make an error judgment. In this case, it is preferable to display a guidance message such as "Positioning OK" on the display 15.

On the other hand, if at least one of the three first judgment regions has a brightness less than the first threshold value, or if at least one of the three second judgment regions has a brightness equal to or greater than the second threshold value, the error judgment unit 421 makes an error judgment. It is preferable to display the result of the error judgment on the display 15 (for example, displaying a guidance message such as "The scope is not in an appropriate position").

For example, as shown in FIG. 75, if the position of the mark 407a is shifted in a direction (Y direction or Z direction (see FIG. 62)) that is substantially perpendicular to the insertion direction X of the tip 12d of the endoscope, an error is determined because the brightness of one of the three first determination regions Jm1 to Jm3, Jm1, is less than the first threshold. In this case, the user operates the tip 12d in the Y direction or Z direction so that the mark 407a is between the inner circle GAx and the outer circle GAy of the guide region.

Also, if the distance between the chart 408 and the tip 12 of the endoscope is too close, as shown in FIG. 76, the brightness of the three second judgment areas Jn1 to Jn3 is equal to or greater than the second threshold, and an error is determined. In this case, the user operates the tip 12d in the X direction so that the mark 407a fits between the inner circle GAx and the outer circle GAy of the guide area. On the other hand, if the distance between the chart 408 and the tip 12 of the endoscope is too close, as shown in FIG. 77, the brightness of two of the three first judgment areas Jm1 to Jm3, the second judgment areas Jm1 and Jm2, is less than the first threshold, and an error is determined. In this case, the user operates the tip 12d in the X direction so that the mark 407a fits between the inner circle GAx and the outer circle GAy.

The error determination unit 421 may perform an error determination when the brightness of the three first determination regions is equal to or greater than the first threshold value and the brightness of the three second determination regions is less than the second threshold value, and the color tone of the third determination region Jt is outside a specific color tone range. As shown in FIG. 78, an error determination is performed when the color tone of the third determination region Jt is red or the like, even though the position of the mark 407a is between the inner circle GAx and the outer circle GAy of the guide region. In this case, it is preferable to notify the result of the error determination, but it is preferable to make the notification different from the notification of the result of the error determination based on the information of the first determination region group and the information of the second determination region group. For example, a guidance display such as "The color tone is not suitable for white balance adjustment" is performed. In FIG. 78, the difference in color tone of the third determination region Jt from that of the third determination region Jt in FIG. 74 is expressed by different hatching (with different hatching inclinations).

As shown in Figure 83, when not only mark 407a but also mark 407b is captured in the endoscopic image, the brightness of the first judgment regions Jm1 to Jm4 is less than the first threshold value and the brightness of the second judgment regions Jn7 and Jn8 is equal to or greater than the second threshold value, so it is determined to be an error. Also, as shown in Figure 84, when the shape of mark 407a is deformed into an ellipse and exceeds the outer circle GAy of the guide region, the brightness of the first judgment regions Jm1 to Jm4 is equal to or greater than the first threshold value, but the brightness of the second judgment regions Jn2 to Jn4 is equal to or greater than the second threshold value, so it is determined to be an error.

Next, a series of steps in the white balance mode will be explained with reference to the flowchart in Figure 79. The user inserts the tip 12d (scope tip) of the endoscope into the adjustment jig 400. In this case, it is inserted into one of the insertion holes 405 of the adjustment jig 400 that corresponds to the diameter of the tip 12d of the endoscope. The mode change switch 12f is operated to switch to the white balance mode. As a result, an endoscopic image of the state of the internal space 406 of the adjustment jig 400 is displayed on the display 15. The user inserts the tip 12d of the endoscope in the insertion direction X until it reaches the mark display position where a mark on the chart 408 is displayed on the endoscopic image.

When the tip 12d of the endoscope is inserted to the mark display position, an endoscopic image obtained by imaging the chart 408 including marks 407a to 407d on the chart 408, the position of the insertion hole into which the tip 12d of the endoscope is inserted, and a mark corresponding to the type of endoscope 12, is displayed on the display 15. The error determination unit 421 obtains information on a first determination area group including three first determination areas Jm1 to Jm3 from the endoscopic image, and information on a second determination area group that is provided outside the first determination area group and includes at least three second determination areas Jn1 to Jn3, and performs an error determination as to whether or not an error has occurred regarding the positional relationship between the chart 408 and the tip 12d of the endoscope based on the information on the first area determination group and the information on the second area determination group.

If no error is determined, the white of the processor device 14 is adjusted in the lance processing unit (not shown) based on the endoscopic image when no error is determined. On the other hand, if an error is determined, the user uses the guide area GA as a clue to operate the tip 12d of the endoscope so that the mark is displayed in the correct position. When the tip 12d of the endoscope is displayed in the correct position, the error determination is resolved. The white of the processor device 14 is adjusted in the lance processing unit based on the endoscopic image when the error determination is resolved.

As shown in FIG. 80, the error determination unit 421 may obtain information on the four first determination regions Jm1 to Jm4 included in the first determination region group and information on the eight second determination regions Jn1 to Jn8 included in the second determination region group from the endoscopic image and perform error determination. As with the above embodiment, the information on the first determination regions Jm1 to Jm4 or the information on the second determination regions Jn1 to Jn8 is preferably brightness. The four first determination regions Jm1 to Jm4 are provided at approximately 90° intervals inside the inner circle GAx of the guide region. As for the eight second determination regions Jn1 to Jn8, four second determination regions Jn1 to Jn4 are provided at a predetermined interval in the left half outside the outer circle GAy of the guide region, and four second determination regions Jn5 to Jn8 are provided at a predetermined interval in the right half.

It is also preferable to normalize the brightness of the first judgment regions Jm1 to Jm4 or the brightness of the second judgment regions Jn1 to Jn8 with the brightness of the normalization region Ju. As a result, as shown in FIG. 81, even if the brightness of the entire image changes and falls outside the reference range, such as immediately after automatic exposure control, by using the normalized brightness of the first judgment region or second judgment region, it is possible to accurately perform error determination regardless of changes in the brightness of the entire image.

For normalization, it is preferable to divide the brightness of the first judgment region or the second judgment region by the brightness of the normalization region. It is also preferable to provide the normalization region Ju inside the first judgment region group including the first judgment regions Jm1 to Jm4. Also, in the case of FIG. 74, the brightness of the first judgment regions Jm1 to Jm3 or the brightness of the second judgment regions Jn1 to Jn3 may be normalized using the brightness of the normalization region.

In addition, it is preferable that the error determination unit 421 determines that there is an error when the number of second determination areas whose brightness is equal to or greater than the second threshold is equal to or greater than a predetermined number among the multiple second determination areas. For example, it is preferable that the error determination unit 421 determines that there is an error when the number of second determination areas whose brightness is equal to or greater than the second threshold is greater than the number of second determination areas whose brightness is less than the second threshold. As shown in FIG. 81, when the number of second determination areas is 8, the specified number is set to, for example, 6 when performing error determination by majority vote. By performing error determination by majority vote in this manner, for example, even if the mark 407a is between the inner circle GAx and the outer circle GAy of the guide area, but only the brightness of the second determination area Jm3 is equal to or greater than the second threshold due to the flare FLA reflected in the endoscopic image, it can be determined that there is no error by using error determination by majority vote.

In the above embodiment, the hardware structure of the processing unit that executes various processes such as the oxygen saturation image generating unit 61, the specific dye concentration calculating unit 62, the table correcting unit 63, the mode switching unit 64, the display mode control unit 65, the reliability calculating unit 66, the first correction determining unit 67, the second correction determining unit 68, the determination notifying unit 69, the base image generating unit 70, the calculation value calculating unit 71, the oxygen saturation calculating unit 72, the color tone adjusting unit 74, the image acquiring unit 420, and the error determining unit 421 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphical Processing Unit), a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

A processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Multiple processing units may also be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by computers such as clients and servers, there is a form in which one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

More specifically, the hardware structure of these various processors is an electric circuit (circuitry) that combines circuit elements such as semiconductor elements. The hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or solid state drive (SSD).

10, 100 Endoscope system 12 Endoscope 12a Insertion section 12b Operation section 12c Bending section 12d Tip section 12e Angle knob 12f Mode changeover switch 12h Still image acquisition instruction switch 12i Zoom operation section 12j Forceps port 13 Light source device 14 Processor device 15 Display 16 Processor side user interface 17 Extended processor device 18 Extended display 19 Scope side user interface 20 Light source section 20a V-LED
20b BS-LED
20c BL-ELD
20d G-LED
20e R-LED
21 Light source processor 23 Optical path coupling unit 25 Light guide 30 Illumination optical system 31 Imaging optical system 32 Illumination lens 35 Objective lens 36, 106 Imaging sensor 37 Imaging processor 40 CDS/AGD circuit 41 A/D converter 50 Image processing unit 51 Image communication unit 52 Display control unit 53 Central control unit 55a, 55b, 55c Curve 56a, 55b Curve 61 Oxygen saturation image generation unit 62 Specific dye concentration calculation unit 62a Specific dye concentration calculation table 63 Table correction unit 64 Mode switching unit 65 Display mode control unit 66 Reliability calculation unit 67 First correction judgment unit 68 Second correction judgment unit 69 Judgment notification unit 70 Base image generation unit 71 Calculated value calculation unit 72 Oxygen saturation calculation unit 73 Oxygen saturation calculation table 74 Color tone adjustment unit 75, 76 Contour line 80 Correction image 81 Specific area 82a Low-reliability region 82b High-reliability region 90 Two-dimensional coordinates 91 Reference line 92 Measured line 102 Broadband light source 104 Rotating filter 105 Filter switching unit 108 Inner filter 108a B1 filter 108b G filter 108c R filter 109 Outer filter 109a B1 filter 109b B2 filter 109c G filter 109d R filter 109e B3 filter 200 Endoscope system 201 Endoscope 202 Light guide 203 Camera heads 205 to 207 Dichroic mirrors 210 to 213 Imaging sensor 300 Endoscope 301 Color imaging sensor 302 Monochrome imaging sensor 303 Camera head 305 Dichroic mirror 400 Adjustment jig 401 Insertion section 401a, 401b Guide section 402 Cover section 402a Outer surface 402b Inner surface 402c Convex portion 402e Lid member 403 Outer peripheral portion 403e Partition 403f Magnet insertion portion 403g Convex portion 405 Insertion holes 405a, 405b Insertion hole 406 Internal spaces 406a, 406b Internal spaces 407a to 407d Mark 408 Chart 411 Oblique mirror 412 Direct mirror 420 Image acquisition unit 421 Error determination units AR0 to AR4 Areas DFX, DFY Definition line BF B color filter GD Operation guidance GF G color filter FLA Flare MS0, MS1, MS2 Message RF R color filter CV0 to CV4 Curved surfaces EL, ELL, ELH Contour lines Jm1 to Jm8 First determination region Jn1 to Jn8 Second determination region Jt Third determination region Ju Standardization region GAx Inner circle GAy Outer circle

Claims (15)

A processor is provided.
The processor,
An endoscopic image is obtained by capturing an image of the chart from the tip of the endoscope;
A processor device that acquires information about a first judgment area group including at least three first judgment areas and information about a second judgment area group that is located outside the first judgment area group and includes at least three second judgment areas from the endoscopic image, and performs an error judgment as to whether or not an error has occurred regarding the positional relationship between the chart and the tip portion based on the information about the first judgment area group and the information about the second judgment area group. The information of the three first determination regions is brightness,
2 . The processor device according to claim 1 , wherein the processor determines that an error has occurred when at least one of the three first determination regions has a brightness less than a first threshold value. The information of the three second determination regions is brightness,
2 . The processor device according to claim 1 , wherein the processor determines that an error has occurred when at least one of the second determination areas has a brightness equal to or greater than a second threshold value among the three first determination areas. The information of the second determination region is brightness,
2 . The processor device according to claim 1 , wherein the processor determines that an error has occurred when a predetermined number of the second judgment regions, among the plurality of second judgment regions, are greater than or equal to a second threshold value. The information of the second determination region is brightness,
The processor device according to claim 1 , wherein the processor determines that an error has occurred when the number of the second judgment regions whose brightness is equal to or greater than a second threshold is greater than the number of the second judgment regions whose brightness is less than the second threshold. The processor device according to claim 2, wherein the brightness of the first judgment area is normalized by the brightness for the normalization area. The processor device according to any one of claims 3 to 5, wherein the brightness of the second judgment area is normalized by the brightness for the normalization area. The processor device according to claim 1, wherein the arrangement shape of the first judgment region group is an inverted triangle obtained by connecting the three first judgment regions, and the arrangement shape of the second judgment region group is a triangle obtained by connecting the three second judgment regions. A third determination region is provided inside the first determination region group,
2 . The processor device according to claim 1 , wherein the processor determines whether or not the error has occurred based on information about the third judgment area in addition to information about the first judgment area and information about the second judgment area. the information of the third determination region is a color tone;
The processor device according to claim 9 , wherein the processor performs the error determination when a color tone of the third determination region is outside a specific color tone range. A processor unit including a processor;
a display for displaying the endoscopic image;
The processor,
An endoscopic image is obtained by capturing an image of the chart from the tip of the endoscope;
information on a first judgment area group including at least three first judgment areas and information on a second judgment area group provided outside the first judgment area group and including at least three second judgment areas are obtained from the endoscopic image, and an error judgment is made as to whether or not an error has occurred regarding a positional relationship between the chart and the tip portion based on the information on the first judgment area group and the information on the second judgment area group;
The chart is provided with marks,
An endoscope system in which the display displays a guide area for guiding the position of the mark between the first judgment area group and the second judgment area group. The endoscope system according to claim 11, wherein the guide region has a double circle. The processor:
a display step of displaying an endoscopic image obtained by capturing an image of the chart from the distal end of the endoscope on a display;
An operation method for an endoscopic system comprising: acquiring, from the endoscopic image, information on a first judgment area group including at least three first judgment areas and information on a second judgment area group that is provided outside the first judgment area group and includes at least three second judgment areas; and making an error judgment as to whether or not an error has occurred regarding the positional relationship between the chart and the tip portion based on the information on the first judgment area group and the information on the second judgment area group. The information of the three first determination regions is brightness,
The method for operating an endoscopic system according to claim 13, wherein in the error determination step, if there is at least one first determination area among the three first determination areas whose brightness is less than a first threshold value, the error is determined to be an error. The information of the three second determination regions is brightness,
The method for operating an endoscopic system according to claim 13, wherein in the error determination step, if there is at least one second determination area among the three second determination areas whose brightness is equal to or greater than a second threshold, it is determined that an error has occurred.

Images