JP5816459B2 - Electronic endoscope system, operation method of electronic endoscope system, and software - Google Patents

Description

  The present invention relates to an electronic endoscope system that detects a portion having a predetermined characteristic in an image.

  The electronic endoscope system includes an endoscope scope that is inserted into a subject's body and an endoscope processor that is provided outside the subject's body. The distal end portion of the endoscope scope is provided with a light guide tip portion and an image sensor. Illumination light is irradiated from the distal end of the light guide to the observation object, and the imaging element transmits an observation image obtained by imaging the observation object in the body to the endoscope processor. The endoscope processor displays the observation image on the monitor after image processing.

  Using the luminance signal output from the image sensor, luminance signals indicating red, green, and blue luminance are generated for each pixel. A luminance signal representing red luminance is referred to as an R signal, a luminance signal representing green luminance is referred to as a G signal, and a luminance signal representing blue luminance is referred to as a B signal. In order to image a two-dimensional distribution of the amount of hemoglobin in the surface mucosa of an organ, a logarithm is taken with respect to a value obtained by dividing the G signal or B signal by the R signal (Patent Document 1). The surgeon determines the normal part and the abnormal part on the surface layer of the organ by looking at the image thus created.

JP-A-6-335451

  However, by dividing the G signal by the R signal, if the R and G signals are close to both the normal part and the abnormal part, the pixels having the R and G signals are separated from the normal part and the abnormal part. There is a possibility that it cannot be identified.

  The present invention has been made in view of these problems, and an object thereof is to obtain an electronic endoscope system capable of accurately and clearly separating a normal part and an abnormal part.

  The electronic endoscope system according to the first invention of the present application operates in accordance with a relationship between a calculation unit that calculates a feature amount indicating a feature of a pixel and a predetermined initial threshold value and the feature amount in each pixel of the endoscopic image. A generation unit that generates a dynamic threshold and a determination unit that divides all pixels into two groups using a dynamic threshold, and the generation unit determines a limit threshold using the feature values of pixels belonging to one group In the range from the initial threshold value to the limit threshold value, the temporary threshold value that minimizes the number of pixels having the feature amount included in the vicinity of the temporarily determined temporary threshold value is determined as the dynamic threshold value.

  The generation unit preferably determines the centroid of the feature values belonging to one group as the limit threshold.

  The generation unit preferably temporarily sets a temporary threshold for each movement step obtained by dividing the difference between the limit threshold and the initial threshold by a predetermined value.

  The generation unit calculates a half value obtained by dividing the difference between the limit threshold value and the initial threshold value by a predetermined value as the neighborhood value, and adds the neighborhood value to the provisional threshold value from the value obtained by subtracting the neighborhood value from the provisional threshold value. It is preferable to determine, as the dynamic threshold, a temporary threshold that minimizes the number of feature amounts belonging to the count range up to the value.

  One pixel preferably has a plurality of pieces of luminance information, and the calculation unit preferably calculates the feature amount by normalizing two pieces of luminance information with the other piece of luminance information.

  The generation unit may determine a median value of the feature values belonging to one group as a limit threshold value.

  The generation unit may determine a value having the largest difference from the initial threshold among the feature values belonging to one group as the limit threshold.

  The generation unit calculates a half of the movement step as a neighborhood value, and the provisional threshold that minimizes the number of feature quantities belonging to the count range from a value obtained by subtracting the neighborhood value from the provisional threshold to a value obtained by adding the neighborhood value to the provisional threshold. Is preferably determined as the dynamic threshold.

  The image processing method according to the second invention of this application classifies the pixels into two groups according to the relationship between the step of calculating the feature quantity indicating the feature of the pixel in each pixel of the image and the predetermined initial threshold value and the feature quantity. A step, a step of determining a limit threshold value using a feature amount of pixels belonging to one of the two classified groups, and a temporary threshold value that is temporarily determined in a range from the initial threshold value to the limit threshold value. The method includes a step of determining, as a dynamic threshold, a temporary threshold that minimizes the number of pixels having the included feature amount, and a step of dividing all pixels into two groups using the dynamic threshold.

  The software according to the third aspect of the present application is configured to cause a pixel to be divided into two groups according to a step of causing the calculation unit to calculate a feature amount indicating a feature of the pixel in each pixel of the image and a predetermined initial threshold value and the feature amount. In the range from the initial threshold value to the limit threshold value, the step of causing the generation unit to sort, the step of causing the generation unit to determine the limit threshold value using the feature amount of the pixels belonging to one group of the two classified groups, A step of causing the generation unit to determine, as a dynamic threshold, a temporary threshold that minimizes the number of pixels having a feature amount included in the vicinity of the temporarily determined temporary threshold; and two groups of all pixels using the dynamic threshold And a step of causing the determination unit to classify.

  According to the present invention, an electronic endoscope system capable of accurately and clearly separating a normal part and an abnormal part is obtained.

It is a block diagram of an electronic endoscope system. It is the flowchart which showed the arithmetic processing. It is the figure which showed an example of the picked-up image. It is the figure which showed the normalized luminance information. It is the figure which showed the result of having performed the PCA process on the normalized luminance information. It is the flowchart which showed the dynamic threshold value generation process. It is the figure which showed the concept of the dynamic threshold value generation process on the graph.

  Hereinafter, an embodiment of an electronic endoscope system 100 according to the present invention will be described with reference to the accompanying drawings. First, the configuration of the electronic endoscope system 100 will be described with reference to FIG.

  The electronic endoscope system 100 includes an endoscope scope 110 that is inserted into a subject's body, an endoscope processor 120 that is provided outside the subject's body and performs image processing, and a monitor that is connected to the endoscope processor 120. 140 mainly. The endoscope scope 110 transmits luminance information obtained by capturing a subject image to the endoscope processor 120, and the endoscope processor 120 performs image processing on the received luminance information to create a display image. The display image is displayed on the monitor 140.

  The configuration of the endoscope scope 110 will be described. An imaging lens 111 and a CCD 112 are mainly provided at the distal end portion of the endoscope scope 110. The CCD 112 receives a subject image via the imaging lens 111 and outputs luminance information of three lights of red, blue, and green. A light guide 113 is provided over the entire length of the endoscope scope 110 and carries the illumination light to the distal end.

  A plurality of pixels are provided on the imaging surface of the CCD 112. A filter that transmits light in any one wavelength band of red, blue, or green is attached to the light receiving surface of one pixel. Pixels with a red filter attached output the luminance of light having a red wavelength band as luminance information, pixels with a blue filter attached have blue luminance information, and pixels with a green filter attached have green color The brightness information of is output. Hereinafter, the luminance of light having a red wavelength band is referred to as red luminance information, the luminance of light having a blue wavelength band is referred to as blue luminance information, and the luminance of light having a green wavelength band is referred to as green luminance information. The luminance information output from the CCD 112 includes only luminance information of one color for one pixel. The luminance information output from the CCD 112 is transmitted to the endoscope processor 120.

  The endoscope scope 110 is connected to the endoscope processor 120. The endoscope processor 120 is mainly composed of a main processing unit 121 and a calculation unit 127. The main processing unit 121 receives the luminance signal from the endoscope scope 110, performs image processing to create a captured image, and then outputs the captured image to the monitor 140. The computing unit 127 receives the captured image from the main processing unit 121, classifies the captured image into a normal part and an abnormal part, and outputs information on the abnormal part. The information regarding the abnormal part is the position of the abnormal part in the captured image. The abnormal part in the photographed image corresponds to a lesion part changed by a disease.

  The main processing unit 121 will be described. The main processing unit 121 includes an image input processing unit 123, an image memory 124, a result display unit 125, a light source 126, and a system controller 122.

  The image input processing unit 123 receives luminance information from the CCD 112 and creates a captured image. Processing for creating a captured image will be described. As described above, the luminance information output from the CCD 112 includes only luminance information of one color for one pixel. However, one pixel in the captured image requires luminance information of three colors. Therefore, the image input processing unit 123 uses the luminance information of pixels existing around one pixel of the CCD 112 to create luminance information of a color lacking in one pixel of the CCD 112. For example, blue luminance information and green luminance information are created for a pixel having red luminance information. Thereby, the pixel in the captured image has three pieces of luminance information of red, blue, and green. The image memory 124 stores the captured image created by the image input processing unit 123.

  The system controller 122 transmits the captured image stored in the image memory 124 to the calculation unit 127. As will be described later, the calculation unit 127 calculates the position of the abnormal part included in the captured image and outputs it to the system controller 122. The system controller 122 receives the position of the abnormal part from the calculation unit 127 and transmits the position of the abnormal part to the result display unit 125. The result display unit 125 creates a display image in which the range occupied by the abnormal part in the captured image is marked. The monitor 140 displays the display image created by the result display unit 125.

  The light source 126 supplies illumination light to the light guide 113 in accordance with a signal from the system controller 122.

  Next, the calculation unit 127 will be described. The calculation unit 127 includes an image processing unit 128, a dynamic threshold value generation unit 129, and a lesion determination unit 130. The image processing unit 128 calculates a feature amount to be described later with respect to the captured image. The dynamic threshold value generation unit 129 calculates a dynamic threshold value that distinguishes between a normal part and an abnormal part in the captured image based on the feature amount calculated by the image processing unit 128. The dynamic threshold will be described later. The lesion determination unit 130 uses the threshold value calculated by the dynamic threshold value generation unit 129 to classify the normal part and the abnormal part in the captured image.

  With reference to FIG. 2, the calculation process executed by the calculation unit 127 will be described. The calculation process is executed when the calculation unit 127 receives a captured image. First, in step S21, the image processing unit 128 calculates a feature amount by using a normalization process and a PCA (Principal Component Analysis) process which will be described later. In step S22, the dynamic threshold value generation unit 129 generates a dynamic threshold value using a dynamic threshold value generation process described later. In step S23, the lesion determination unit 130 classifies the normal part and the lesion part in the captured image, that is, determines the normal part and the lesion part. In the last step S24, the determination result is output to the system controller 122, and the process ends.

  Next, the image processing unit 128 will be described in detail with reference to FIG. The image processing unit 128 receives a captured image, that is, red luminance information, blue luminance information, and green luminance information from the system controller 122, and calculates a feature amount. The feature amount is a value representing the characteristic of the pixel, and is obtained for each pixel by normalization processing and PCA processing. Normalization processing and PCA processing are executed by the image processing unit 128. The normalization process is a process for normalizing the other two pieces of luminance information using one piece of luminance information in order to prevent the brightness level from affecting the result of the pixel separation process. The PCA process is a process for obtaining characteristics of each pixel constituting an image. The characteristic of each pixel is generally called a PCA score, and in the present invention is called a feature amount.

  Next, details of the normalization process will be described with reference to FIGS. FIG. 3 is an example of an observation image that is simplified for explanation. The observation image has a pink area A, a vermilion area B, and a skin color area C. Redness increases in the order of region C, region A, and region B.

  The normalization process processes the luminance information of the pixels included in the area A, the area B, and the area C. The blue luminance information and the green luminance information of the pixels included in each of the region A, the region B, and the region C are respectively divided by the red luminance information. Thereby, the blue luminance information and the green luminance information of each pixel are normalized, and each pixel in the observation image has a value obtained by dividing the blue luminance information by the red luminance information (hereinafter referred to as a B / R value) and a green luminance. It has a value (hereinafter referred to as G / R value) obtained by dividing the information by the red luminance information. FIG. 4 shows a graph of normalized blue luminance information and green luminance information. By normalizing, it is possible to eliminate the influence of pixel brightness and perform pixel separation processing.

  Next, the PCA process will be described with reference to FIGS.

  Before executing the PCA process, the following process is executed. An initial observation image that has been captured in advance is prepared. For the sake of explanation, the image shown in FIG. The normal part and the abnormal part in the initial observation image are known, and the pixels of the initial observation image are classified in advance into a normal part and an abnormal part. The normal part is classified into the first group, and the abnormal part is classified into the second group. Here, assuming that the region A and the region B in FIG. 3 are abnormal parts and the region C is a normal part, the regions A and B are classified into the second group and the region C is classified into the first group. Each pixel of the initial observation image has an initial luminance signal composed of initial red luminance information, initial blue luminance information, and initial green luminance information. Then, the initial luminance signal is normalized by a normalization process to obtain an initial normalized signal, that is, an initial B / R value and an initial G / R value.

  In the PCA processing, PCA processing is performed on these initial normalized signals, and a PCA score is calculated. The PCA process is also called a principal component analysis, and is a process for combining (compressing) several factors having a correlation into several components to obtain their total power and characteristics.

First, several specific pixels are acquired from the normal part and the abnormal part, and a variance-covariance matrix is obtained for these initial normalized signals. In other words, a variance covariance matrix is obtained for the initial B / R value and the initial G / R value of a specific pixel. Here, it is assumed that the obtained variance-covariance matrix is the following matrix.

そして、以下のような固有ベクトルが求められる。

そして、大きい方の固有値に対応する固有ベクトルを第１主成分ベクトル、小さい方の固有値に対応する固有ベクトルを第２主成分ベクトルとする。すなわち、第１主成分ベクトルは(−０．５３２３, ０．８４６６)、第２主成分ベクトルは(−０．８４６６, −０．５３２３)となる。特定の画素の正規化信号ベクトル(Ｂ／Ｒ, Ｇ／Ｒ)と第１または第２主成分ベクトルの内積を計算することで、その画素の第１主成分ＰＣ１または第２主成分ＰＣ２が求められる。第１主成分及び第２主成分がＰＣＡスコア、すなわち特徴量を成す。以下、第１主成分ＰＣ１を特徴量１、第２主成分ＰＣ２を特徴量２と呼ぶ。 Next, eigenvalues and eigenvalue vectors of the variance-covariance matrix are obtained. Here, eigenvalues 5.0817 and 4.4406 are obtained as follows.

Then, the following eigenvectors are obtained.

The eigenvector corresponding to the larger eigenvalue is set as the first principal component vector, and the eigenvector corresponding to the smaller eigenvalue is set as the second principal component vector. That is, the first principal component vector is (−0.5323, 0.8466), and the second principal component vector is (−0.8466, −0.5323). By calculating the inner product of the normalized signal vector (B / R, G / R) of the specific pixel and the first or second principal component vector, the first principal component PC1 or the second principal component PC2 of the pixel is obtained. It is done. The first principal component and the second principal component form a PCA score, that is, a feature amount. Hereinafter, the first principal component PC1 is referred to as a feature amount 1 and the second principal component PC2 is referred to as a feature amount 2.

  PCA score is calculated from the eigenvectors (first and second principal component vectors) obtained as described above, and the initial B / R values and the initial G / R values for the pixels included in the regions A, B, and C. Then, the graph shown in FIG. 5 is obtained. This eigenvector is used to calculate the feature amount of the captured image actually input.

  Next, the dynamic threshold value generation unit 129 will be described in detail with reference to FIG. The dynamic threshold value generation unit 129 dynamically calculates a threshold value for classifying the normal part and the abnormal part based on the feature amount 1 calculated by the image processing unit 128. This threshold is called a dynamic threshold. The dynamic threshold is calculated for each captured image by the dynamic threshold determination process. In other words, since this is a threshold that changes for each captured image, it is called a dynamic threshold.

  Next, the lesion determination unit 130 will be described in detail. The lesion determination unit 130 determines an abnormal part from the pixels included in the captured image based on the dynamic threshold. More specifically, a pixel having a feature amount 1 larger than the dynamic threshold is determined as a lesion.

  Next, the dynamic threshold value determination process will be described with reference to FIGS. In this process, an initial threshold value used for provisionally dividing the normal part and the abnormal part is determined in advance. The dynamic threshold value determination process obtains the center of gravity of one group out of the two groups divided by the initial threshold value, and searches for the dynamic threshold value that best separates the normal part and the lesioned part between the initial threshold value and the center of gravity. It is processing. The temporary threshold value is moved from the initial threshold value to the center of gravity by a predetermined movement step, and when the number of pixels located in the vicinity of the temporary threshold value is the smallest, the temporary threshold value is determined as the dynamic threshold value. A flowchart of the dynamic threshold value determination process is shown in FIG.

  Referring to FIG. 7, the normal part and the lesion part are distributed according to the feature amounts 1 and 2. In step S61 of the dynamic threshold value determination process, these are first separated into two groups with an initial threshold value. A group in which the feature amount 1 is larger than the initial threshold is referred to as a provisional lesion group. Then, the center of gravity of the provisional lesion group is obtained. The feature value 1 of the center of gravity is the limit threshold.

  In the next step S62, the difference between the center of gravity and the initial threshold is divided by a predetermined value to calculate a temporary threshold moving step, and the moving value is divided by 2 to calculate a neighborhood value. The predetermined value is a value determined according to the object to be observed, the type of illumination light, the experience value, and the like, for example, 10 is used. The neighborhood value is a value indicating a range in the vicinity of the temporary threshold. Then, the temporary threshold is set to the same value as the initial threshold.

  In step S63, the value moved by the moving step from the current temporary threshold is set as a new temporary threshold. In other words, a value obtained by adding the value of the movement step to the current temporary threshold value is set as a new temporary threshold value.

  In step S64, the number of pixels existing in the counting range from the value obtained by subtracting the neighborhood value from the provisional threshold to the value obtained by adding the neighborhood value to the provisional threshold is counted. This number is called the number of neighboring pixels. The pixels existing in the counting range are shown in FIG.

  In step S65, it is determined whether or not the value obtained by adding the moving step to the temporary threshold is greater than the limit threshold. If so, the process proceeds to step S66. If not, the process returns to step S63. By repeating steps S63 to S65, a plurality of correspondence relationships between the temporary threshold and the number of neighboring pixels are obtained.

  In step S66, the temporary threshold with the smallest number of neighboring pixels is determined as the dynamic threshold. In step S <b> 67, the lesion determination unit 130 determines that a pixel having the feature amount 1 exceeding the dynamic threshold is a lesion. Thereafter, the process ends.

  The dynamic threshold value generation process is executed every time a captured image is input to the calculation unit 127. That is, a dynamic threshold is set for each captured image, and a normal part and a lesioned part are determined for each captured image using the set dynamic threshold. Thereby, it is possible to accurately determine the normal portion and the lesioned portion without being influenced by the imaging conditions that change from moment to moment. In addition, by setting the center of gravity as a limit threshold, it is possible to determine a normal part and a lesion part while minimizing the influence of noise included in the captured image.

  According to the present embodiment, the normal part and the abnormal part can be accurately and clearly separated without being affected by changes in the photographing conditions.

  In normalization, the divisor may not be red luminance information, but may be blue luminance information or green luminance information.

  Further, the NN method or the Bayes identification method may be applied to the normalized signal classified by the Fisher discrimination method instead of the PCA process.

  The imaging unit is not limited to the CCD 112, and may be an imaging element such as a CMOS.

  Not the centroid of the feature quantity 1, but the maximum feature quantity having the largest difference from the initial threshold among the feature quantities 1 belonging to one group, or 90% of the maximum feature quantity or the median (median) is the threshold threshold May be used as The same effect as the center of gravity is obtained.

  The movement step is not limited to a value obtained by dividing the difference between the center of gravity and the initial threshold value by 10, and the neighborhood value is not limited to a value obtained by dividing the movement step by 2.

  The count range may be a value from a value obtained by subtracting the neighborhood value from the provisional threshold to the provisional threshold, or a value from the provisional threshold to a value obtained by adding the neighborhood value to the provisional threshold.

In step S66 of the dynamic threshold value generation process, a temporary threshold value with the smallest density of neighboring pixels may be determined as the dynamic threshold value. The density is obtained by dividing the number of neighboring pixels by the size of the neighboring range.

100 Electronic Endoscope System 110 Endoscope Scope 111 Imaging Lens 112 CCD
113 Light Guide 120 Endoscope Processor 121 Main Processing Unit 122 System Controller 123 Image Input Processing Unit 124 Image Memory 125 Result Display Unit 126 Light Source 127 Calculation Unit 128 Image Processing Unit 129 Dynamic Threshold Generation Unit 130 Lesion Determination Unit 140 Monitor

Claims (9)

For each pixel of the endoscopic image, a calculation unit that calculates a feature amount indicating the feature of the pixel;
A generation unit that generates a dynamic threshold according to a relationship between a predetermined initial threshold and a feature amount;
A determination unit that divides all pixels into two groups using the dynamic threshold,
The generation unit determines a limit threshold value using the feature values of pixels belonging to one group, moves a temporary threshold value that is temporarily determined in a range from the initial threshold value to the limit threshold value, and moves the temporary threshold value. A provisional threshold value that minimizes the number of pixels having a feature amount included in the vicinity of the threshold value or minimizes the density of neighboring pixels is determined as a dynamic threshold value ,
The said production | generation part is an electronic endoscope system which determines a temporary threshold value for every movement step obtained by dividing | segmenting the difference of a limit threshold value and an initial threshold value by a predetermined value .   The electronic endoscope system according to claim 1, wherein the generation unit determines a centroid of feature amounts belonging to one group as a limit threshold. The generation unit calculates a half value obtained by dividing the difference between the limit threshold value and the initial threshold value by a predetermined value as a neighborhood value, and adds the neighborhood value to the provisional threshold value from a value obtained by subtracting the neighborhood value from the provisional threshold value. the electronic endoscope system according to any one of claims 1 to 2 the number of feature amounts belonging to counting range up value determines the tentative threshold becomes minimum as the dynamic threshold was. One of the pixels includes a plurality of luminance information, the calculating unit, in any one of claims 1 to 3 for calculating the feature quantity by normalizing the two other one luminance information in the luminance information The electronic endoscope system described.   The electronic endoscope system according to claim 1, wherein the generation unit determines a median value of feature values belonging to one group as a limit threshold value.   The electronic endoscope system according to claim 1, wherein the generation unit determines a value having the largest difference from the initial threshold among the feature values belonging to one group as a limit threshold. The generation unit calculates half of the movement step as a neighborhood value, and the number of feature quantities belonging to the count range from a value obtained by subtracting the neighborhood value from the provisional threshold to a value obtained by adding the neighborhood value to the provisional threshold is minimized. The electronic endoscope system according to claim 1 , wherein the temporary threshold is determined as a dynamic threshold. A method for operating an electronic endoscope system, comprising:
The calculation unit calculates a feature amount indicating a feature of the pixel in each pixel of the image,
The generation unit sorts the pixels into two groups according to the relationship between the predetermined initial threshold and the feature amount,
The determination unit is a method of determining a limit threshold value using a feature amount of pixels belonging to one group out of two classified groups,
In the range from the initial threshold value to the limit threshold value, the generating unit moves the temporary threshold value temporarily determined by a movement step obtained by dividing the difference between the limit threshold value and the initial threshold value by a predetermined value, and the temporary threshold value. A provisional threshold value that minimizes the number of pixels having the feature amount included in the neighborhood of or the density of neighboring pixels is minimized is determined as the dynamic threshold value,
The operation method of the electronic endoscope system, wherein the determination unit divides all pixels into two groups using the dynamic threshold. In electronic endoscope system,
For each pixel of the image, causing the calculation unit to calculate a feature amount indicating the feature of the pixel;
Causing the generation unit to classify the pixels into two groups according to a relationship between a predetermined initial threshold and a feature amount;
A step of causing the generation unit to determine a limit threshold value using a feature amount of pixels belonging to one group out of the two groups sorted;
In the range from the initial threshold value to the limit threshold value, the temporary threshold value temporarily determined is moved by a movement step obtained by dividing the difference between the limit threshold value and the initial threshold value by a predetermined value, and is included in the vicinity of the temporary threshold value. A step of causing the generation unit to determine, as a dynamic threshold, a temporary threshold that minimizes the number of pixels having a feature amount or the density of neighboring pixels is the smallest;
And causing the determination unit to sort all pixels into two groups using the dynamic threshold.

Images