WO2020224153A1

WO2020224153A1 - Nbi image processing method based on deep learning and image enhancement, and application thereof

Info

Publication number: WO2020224153A1
Application number: PCT/CN2019/106030
Authority: WO
Inventors: 胡珊
Original assignee: 武汉楚精灵医疗科技有限公司
Priority date: 2019-05-07
Filing date: 2019-09-16
Publication date: 2020-11-12
Also published as: CN110189303B; CN110189303A

Abstract

Provided are an NBI image processing method based on deep learning and image enhancement, and an application thereof. A deep learning algorithm and image enhancement technology are used to extract features such as microvessels and microstructures of an NBI image, and the characterized image is presented to an endoscopist, overcoming the bottleneck of the prior art. The use of artificial intelligence enables a doctor to give a more accurate supplemental diagnostic opinion on early cancer under NBI. In an image processed using the present method, a lesion area in an image of early gastric cancer is enhanced, the boundary between the lesion and the normal area is highlighted and becomes clearer. When making a diagnosis, the doctor may refer to the processed image to supplement the determination of whether the patient suffers from early cancer, preventing erroneous identification of a lesion due to the gastroscopy being too rapid or because of fatigue of the doctor when operating.

Description

A NBI image processing method based on deep learning and image enhancement and its application

Technical field

The invention belongs to the field of medical detection assistance, and specifically relates to an artificial intelligence-based early gastric cancer auxiliary diagnosis method.

Background technique

Gastric cancer is one of the common malignant tumors in my country, and its incidence ranks first among digestive system tumors. In 2015, there were 679,000 new cases of gastric cancer and 498,000 deaths in my country, accounting for about 1/5 of the total cancer deaths. The fundamental reason that malignant tumors harm human health is that it is difficult to detect early. If gastrointestinal tumors are diagnosed at an early stage, the 5-year survival rate of patients can be higher than 90%, and if it progresses to advanced stages, the 5-year survival rate of patients is only 5-25%. Therefore, early diagnosis is an important strategy to improve patient survival.

Endoscopy is the most commonly used powerful tool to detect early gastric cancer. Endoscopic ordinary white light plus biopsy is the main method for detecting early gastric cancer. It has the advantages of simplicity and intuitiveness. However, because the changes of early cancer lesions are usually mild and have no specific manifestations under white light, it is difficult to compare with normal mucosa and benign lesions. The distinction between erosions and ulcers has low sensitivity and specificity, which can easily lead to missed diagnosis. In recent years, as some key technologies such as filters and magnification under endoscopy have matured, narrow-band imaging (NBI) and magnifying endoscopy (ME) have developed rapidly. The magnifying endoscope can magnify the object image under the endoscope by tens to hundreds of times, and clearly shows the changes in the microvessels of the digestive tract mucosa, the opening of the glands and other fine structures. Narrowband imaging endoscopy uses filters to filter out the broadband spectrum of the red, blue, and green light waves emitted by the endoscope light source, leaving only the blue (400-430nm) and green (535-565nm) narrowband spectra, while hemoglobin is in the visible light. Under the wave, light waves with wavelengths of 415nm and 540nm are strongly absorbed, so the capillaries and mucosal surface structure of the mucosal surface can be clearly displayed. Narrowband imaging combined with magnifying endoscopy (ME-NBI) can enable endoscopists to observe the surface microvascular morphology and surface microstructure of gastric mucosa more clearly, and greatly improve the accuracy of gastrointestinal endoscopy in the diagnosis of early gastric cancer. However, the diagnostic criteria for early gastric cancer under ME-NBI are very complex, and the appearance of the lesions is different. It requires endoscopists to have a strong knowledge reserve and rich experience in order to make good use of this technology to achieve early cancer diagnosis. Under the current situation of my country's large population base and shortage of medical resources, the complexity of ME-NBI diagnosis greatly restricts its ability to detect early gastric cancer.

In recent years, science and technology have developed rapidly, and artificial intelligence has set off a new wave of technology. With the successful testing of self-driving cars and AlphaGo defeating the world champion of Go, artificial intelligence has gradually entered the public eye in just a few years. In the medical industry, artificial intelligence research is mainly concentrated in the field of static film reading, that is, the machine learns a large number of lesion pictures and normal pictures marked by doctors, summarizes the characteristics of the lesion, and then actively recognizes similar lesions in unfamiliar pictures. The more successful cases include the classification and diagnosis of skin cancer and the detection of lung nodules. However, the application of this method has certain limitations. Firstly, it requires that the target lesions and normal tissues and other lesions have clearly distinguishable characteristics, and secondly, it requires accurate classification of pictures input to the machine for learning and no data pollution. We tried to use this method to train the machine's ability to recognize early gastric cancer. However, because the characteristics of early cancer lesions are complex and changeable, and they have similar characteristics to a variety of benign lesions, it is prone to false positives and misidentifications.

Based on this, we intend to invent a NBI image processing method based on artificial intelligence, and assist in the diagnosis of early gastric cancer through the processed NBI image. This method uses deep learning algorithms and image enhancement technology to extract the microvessels and microstructures of the NBI image. Features: Present the characterized picture to the endoscopist, overcome the bottleneck of the existing technology, and use artificial intelligence to enable the doctor to give more accurate auxiliary diagnosis opinions for early cancer under NBI.

Summary of the invention

The technical problem to be solved by the present invention is to use deep learning algorithms and image enhancement technology to extract the microvessels and microstructures of NBI pictures, present the characterized pictures to the endoscopist, overcome the bottleneck of the prior art, and use artificial intelligence to make Doctors give more accurate auxiliary diagnosis opinions for early cancer under NBI.

In order to achieve the above objective, the present invention adopts an NBI image processing method based on deep learning and image enhancement, which specifically includes the following steps:

Step S1, collecting a large number of NBI early gastric cancer or non-cancerous enlarged images;

Step S2, the white areas and blood vessels in the image are marked by a professional physician, so that the original NBI image with a complex background and structure is transformed into a simple stroke image with clear characteristics to obtain the marked image;

Step S3, the original NBI image and the annotated image are input to the deep convolutional neural network model for training. The deep convolutional neural network model is used to continuously calculate the prominent information features between the original image and the annotated image, including the texture difference L _texture , The content difference L _content , the color difference L _color and the overall difference L _tv are weighted to obtain the total loss function value based on the above difference to complete the mapping relationship from the original NBI image to the annotated image;

Step S4: Obtain the target image of the image to be processed based on the above mapping relationship, and map each pixel into a one-dimensional array composed of numbers.

Step S5: By adjusting the RGB color space of the target image, different numbers in the array are displayed in different colors with different shades, so as to obtain a gastric mucosal image with enhanced blood vessels and surface structure and with the remaining background hidden.

Further, the specific implementation of step S3 is as follows:

In step S31, for the texture difference L _texture , a separate adversarial CNN recognizer is trained. The formula for calculating L _texture is as follows:

Among them, I _S is the original NBI image, I _t is the physician-labeled image, i is the logarithm of I _S and I _t , F _W and F _W (I _S ) refer to the image enhancement function and the enhanced image after function processing, respectively, D Is the recognizer;

Step S32, the content difference L _content is defined according to the activation map generated by the ReLU layer of the pre-trained VGG-19 network;

Among them, C _j H _j W _j represents the number, height and width of I _t and F _W (I _S ) enhanced images respectively, and ψ _j is the feature map after j times of convolution;

Step S33: For the color difference L _color , the Gaussian blur method is used to calculate the Euclidean distance between the doctor-labeled image and the original NBI image, the formula is as follows:

X _b and Y _b are the corresponding values of X and Y (pixel coordinates of the original NBI image) in the labeled image after calculation. The solution process is as follows:

The above formula is the Gaussian filter template, where μ _x is the mean value of X, σ _x is the variance of X, A is the sum of the weights of the pixels, and the result G(k,l) is the filter template value at k and l;

Multiply the original image k, l and surrounding pixels with the filter template value to obtain the Gaussian blur value of X _b ; in the same way, obtain Y _b , and substitute into Equation 3 to obtain L _color ;

In step S34, the spatial smoothness of the image is enhanced by calculating the total variation loss function, the formula is as follows:

Step S35: Finally, the color difference, texture difference, content difference and overall difference are combined to obtain the total loss function value,

L _total ＝L _content +0.4·L _texture +0.1·L _color +400·L _tv (7)

Among them, CHW represents the number, height and width of F _W (I _S ) enhanced images,

Is the Hamiltonian to differentiate between X and Y.

Further, the mapping method in step S4 is to realize rgb channel separation based on the Image method in the PythonPIL package, and then use the reshape method to convert the image into a one-dimensional array composed of numbers.

The present invention also provides an application of an NBI image based on deep learning and image enhancement in the diagnosis of early gastric cancer. The NBI image is obtained through the above technical solution.

The beneficial effects of the present invention are: extracting the microvascular and microstructure characteristics of the lesion under the NBI picture by the present invention, on the one hand, it provides a reference for the endoscopist to independently judge the nature of the lesion; on the other hand, the artificial intelligence model prediction problem is sealed to make it Compared with the prior art, it can give more accurate auxiliary diagnosis opinions for early cancer under NBI.

Description of the drawings

Figure 1 is a flowchart of an embodiment of the present invention.

Figure 2 is a schematic diagram of professional physician labeling (①NBI original image②Doctor labeling image).

Figure 3 is a schematic diagram of the method of the present invention (①NBI original image ②image processed by the method of the present invention).

Detailed ways

The technical scheme of the present invention will be further described below in conjunction with the drawings and embodiments.

As shown in Figure 1, an NBI image processing method based on deep learning and image enhancement provided by the present invention includes the following steps

Step S2: The white area and blood vessels in the image are marked by a professional physician, so that the original NBI image with complex background and structure is transformed into a simple stroke image (annotated image) with clear features, as shown in Figure 2;

Step S3, the original NBI image and the annotated image are input into the deep convolutional neural network model for training, and the deep convolutional neural network model continuously calculates the prominent information features between the original image and the annotated image (such as: texture difference L _texture , content The difference L _content , the color difference L _color, and the overall difference) are very necessary. The final effect is to enable the model to automatically complete the mapping of the original NBI image to the target image (the machine imitates the annotation image processed by the doctor), and the final processing The result is shown in Figure 3.

Among them, I _S is the original NBI image, I _t is the physician-labeled image, i is the logarithm of I _S and I _t , F _W and F _W (I _S ) refer to the image enhancement function and the enhanced image after function processing, respectively. In the embodiment, F _W can be customized according to accuracy requirements, and D is the recognizer;

Step S32, for the content difference L _content , we define it according to the activation map generated by the ReLU layer of the pre-trained VGG-19 network [1];

C _j H _j W _j represents the number, height, and width of I _t and F _W (I _S ) enhanced images, respectively, and ψ _j is the feature map after j times of convolution.

Step S33, for the color difference L _color , we use the Gaussian blur method to calculate the Euclidean distance between the doctor-labeled image and the original NBI image, the formula is as follows:

X _b and Y _b are the corresponding values of X and Y (the pixel coordinates of the original NBI image) in the labeled image after calculation. The solution process is as follows:

The above formula is the Gaussian filter template, where μ _x is the mean value of X, σ _x is the variance of X, and A is the sum of the weights of the pixels, which can be temporarily set to 0.035. The purpose is to make the sum of the weights of the pixels in the area equal to 1, so as to keep the image brightness unchanged. The obtained result G(k,l) is the filter template value at k and l.

The Gaussian blur value of X _b can be obtained by multiplying the source image k, l and the surrounding pixels with the filter template value; as above, Y _b can be obtained, and substituting in Equation 3, L _color can be obtained.

Step S35: Finally, the color, texture, content, and overall difference are combined to obtain the overall loss function value:

L _total ＝L _content +0.4·L _texture +0.1·L _color +400·L _tv (7)

Among them, CHW respectively represents the number, height and width of F _W (I _S ) enhanced images (it should be noted that CHW is only a parameter of enhanced images, and C _j H _j W _j is I _t and F _W (I _S ) enhanced images parameters, in fact, in practice the original image, I _t and F _W (I _S) is enhanced CHW values corresponding to the same image, but distinguished by subscripts mode);

Is the Hamiltonian to differentiate between X and Y.

Step S4: Obtain the target image of the image to be processed based on the above mapping relationship, and map each pixel in the target image to a number. The mapping method is based on the Image method in the PythonPIL package to achieve rgb channel separation, and then the image is converted into a One-dimensional array of numbers;

Step S5: By adjusting the RGB mode of the picture, different numbers in the array are displayed in different shades of different colors to obtain an image of the gastric mucosa with enhanced blood vessels and surface structure and with the remaining background hidden.

The RGB mode uses 8-bit binary numbers to represent colors. The number range is [0,255], which is often referred to as a "gray image". 0 means no brightness, that is, black, and 255 indicates the maximum brightness that can be achieved, that is white. The brightness of the specified pixel can be adjusted by adjusting the size of the number in the array.

As shown in Figure 2 on the left and Figure 3 on the left, the structure of gastric blood vessels and glands is very similar, and it is difficult for us to find or define the lesion area; and after the above image processing, as shown in Figure 2 right and Figure 3 right As shown, the lesion area in the image of early gastric cancer is enhanced, and the boundary between the lesion and the normal area is highlighted and becomes clearer. The white highlighted area is the possible lesion area, and the brighter the color indicates that this area is abnormal The higher the probability. When making a diagnosis, the doctor can refer to the processed images to assist in determining whether the patient has early cancer, so as to avoid missing the recognition of the lesion due to the gastroscopy being too fast or the doctor’s fatigue operation.

The specific embodiments described herein are merely examples to illustrate the spirit of the present invention. Those skilled in the art to which the present invention pertains can make various modifications or additions to the specific embodiments described or use similar alternatives, but they will not deviate from the spirit of the present invention or exceed the definition of the appended claims. Range.

references

[1]Karen Simonyan*&Andrew Zisserman.Very Deep Convolutional Networks for Large-Scale Image Recognition[C].ICLR2015,2015.

Claims

An NBI image processing method based on deep learning and image enhancement, which is characterized in that it includes the following steps:

Step S1, collecting a large number of NBI early gastric cancer or non-cancerous enlarged images;

Step S2, the white areas and blood vessels in the image are marked by a professional physician, so that the original NBI image with a complex background and structure is transformed into a simple stroke image with clear characteristics to obtain the marked image;

Step S3, the original NBI image and the annotated image are input to the deep convolutional neural network model for training. The deep convolutional neural network model is used to continuously calculate the prominent information features between the original image and the annotated image, including the texture difference L texture , The content difference L content , the color difference L color and the overall difference L tv are weighted to obtain the total loss function value based on the above difference to complete the mapping relationship from the original NBI image to the annotated image;

Step S4: Obtain the target image of the image to be processed based on the above mapping relationship, and map each pixel into a one-dimensional array composed of numbers.

Step S5: By adjusting the RGB color space of the target image, different numbers in the array are displayed in different colors with different shades, so as to obtain a gastric mucosal image with enhanced blood vessels and surface structure and with the remaining background hidden.
The NBI image processing method based on deep learning and image enhancement according to claim 1, characterized in that: the specific implementation of step S3 is as follows:

In step S31, for the texture difference L texture , a separate adversarial CNN recognizer is trained. The formula for calculating L texture is as follows:

Among them, I S is the original NBI image, I t is the physician-labeled image, i is the logarithm of I S and I t , F W and F W (I S ) refer to the image enhancement function and the enhanced image after function processing, respectively, D Is the recognizer;

Step S32, the content difference L content is defined according to the activation map generated by the ReLU layer of the pre-trained VGG-19 network;

Among them, C j H j W j represents the number, height and width of I t and F W (I S ) enhanced images respectively, and ψ j is the feature map after j times of convolution;

Step S33: For the color difference L color , the Gaussian blur method is used to calculate the Euclidean distance between the doctor-labeled image and the original NBI image, the formula is as follows:

X b and Y b are the corresponding values of X and Y (pixel coordinates of the original NBI image) in the labeled image after calculation. The solution process is as follows:

The above formula is the Gaussian filter template, where μ x is the mean value of X, σ x is the variance of X, A is the sum of the weights of the pixels, and the result G(k,l) is the filter template value at k and l;

Multiply the original image k, l and surrounding pixels with the filter template value to obtain the Gaussian blur value of X b ; in the same way, obtain Y b , and substitute into Equation 3 to obtain L color ;

In step S34, the spatial smoothness of the image is enhanced by calculating the total variation loss function, the formula is as follows:

Step S35: Finally, the color difference, texture difference, content difference and overall difference are combined to obtain the total loss function value,

L total ＝L content +0.4·L texture +0.1·L color +400·L tv (7)

Among them, CHW represents the number, height and width of F W (I S ) enhanced images, and ▽ x , ▽ y are Hamiltonian operators that differentiate between X and Y.
An NBI image processing method based on deep learning and image enhancement according to claim 1 or 2, characterized in that: the mapping method in step S4 is to realize rgb channel separation based on the Image method in the PythonPIL package, and then use the reshape method Convert the image into a one-dimensional array of numbers.
An application of an NBI image based on deep learning and image enhancement in the diagnosis of early gastric cancer, characterized in that the NBI image is obtained by the method of claim 1 or 2 or 3.