CN112914477B - Capsule endoscope system for fluorescence analysis and control method - Google Patents
Capsule endoscope system for fluorescence analysis and control method Download PDFInfo
- Publication number
- CN112914477B CN112914477B CN202110240597.7A CN202110240597A CN112914477B CN 112914477 B CN112914477 B CN 112914477B CN 202110240597 A CN202110240597 A CN 202110240597A CN 112914477 B CN112914477 B CN 112914477B
- Authority
- CN
- China
- Prior art keywords
- fluorescence
- image
- gray
- module
- fluorescence intensity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000002775 capsule Substances 0.000 title claims abstract description 35
- 238000000034 method Methods 0.000 title claims abstract description 18
- 238000012921 fluorescence analysis Methods 0.000 title claims abstract description 17
- 238000002073 fluorescence micrograph Methods 0.000 claims abstract description 90
- 238000012544 monitoring process Methods 0.000 claims abstract description 37
- 239000000523 sample Substances 0.000 claims abstract description 23
- 230000003902 lesion Effects 0.000 claims description 30
- 239000007850 fluorescent dye Substances 0.000 claims description 28
- 238000000338 in vitro Methods 0.000 claims description 27
- 230000011218 segmentation Effects 0.000 claims description 14
- 230000001186 cumulative effect Effects 0.000 claims description 12
- 230000006698 induction Effects 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000013135 deep learning Methods 0.000 claims description 6
- 238000007781 pre-processing Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000003993 interaction Effects 0.000 claims description 5
- 230000003247 decreasing effect Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 3
- 230000001575 pathological effect Effects 0.000 abstract description 10
- 230000005284 excitation Effects 0.000 description 10
- 238000012549 training Methods 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 210000003238 esophagus Anatomy 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 210000001035 gastrointestinal tract Anatomy 0.000 description 1
- 210000000936 intestine Anatomy 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 210000002784 stomach Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/041—Capsule endoscopes for imaging
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00131—Accessories for endoscopes
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00147—Holding or positioning arrangements
- A61B1/00158—Holding or positioning arrangements using magnetic field
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/043—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances for fluorescence imaging
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/045—Control thereof
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
- A61B1/0661—Endoscope light sources
- A61B1/0684—Endoscope light sources using light emitting diodes [LED]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Surgery (AREA)
- Physics & Mathematics (AREA)
- Optics & Photonics (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Heart & Thoracic Surgery (AREA)
- Biomedical Technology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Animal Behavior & Ethology (AREA)
- Molecular Biology (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Endoscopes (AREA)
Abstract
The application discloses fluorescence analysis's capsule endoscope system and control method, accessible light source module sends the exciting light that arouses fluorescence probe to produce fluorescence signal, detects the fluorescence signal through the image acquisition module after, gathers the fluorescence image data that has fluorescence signal, obtains the fluorescence intensity that fluorescence probe produced through external monitoring unit according to fluorescence image data, still sends the light intensity regulation signal to control module according to fluorescence intensity, thereby adjust the light source module and send the light intensity of exciting light can realize that endoscope light source light intensity adaptability adjusts to improve the efficiency of discernment pathological feature, and reduce the possibility of erroneous judgement or erroneous judgement.
Description
Technical Field
The application relates to the technical field of endoscopes, in particular to a capsule endoscope system for fluorescence analysis and a control method.
Background
The capsule endoscope is a capsule-shaped endoscope, is often used for inspecting intestinal tracts of human bodies, snoops intestines, stomachs and esophagus parts of the human bodies after entering the human bodies, can send shot pictures to the outside of the bodies, can display the pictures in real time after receiving fluorescence image data by an external receiver, and can store the pictures in a memory at the same time, and doctors can read the fluorescence image data through an external monitoring system after the examination is finished and make medical diagnosis according to the pictures.
However, the light source of the current endoscope is single, generally white light, and the light emitting with specific wavelength cannot be controlled, and the conventional endoscope needs to distinguish the obvious changes of biological tissue morphology, color and the like through human eyes, namely, the pathological features can be judged according to the displayed images only by the professional knowledge and experience of doctors. This increases the difficulty of the doctor's work and the efficiency of identifying pathological features is poor, and at the same time, because the judgment is made mainly by the subjectivity of the doctor's experience through the discrimination of human eyes, the possibility of erroneous judgment or erroneous judgment is also greatly increased.
Therefore, endoscopes with light sources of various wavelengths appear in the market at present, and fluorescence can be excited conveniently, so that the efficiency of identifying pathological features is improved, and the possibility of misjudgment or misjudgment of the pathological features and the working difficulty are reduced. However, after imaging through the endoscope, because the light intensity of the endoscope light source is unreasonable, the imaging quality is poor, and the light intensity of the current endoscope light source cannot be automatically adjusted, so that the efficiency of identifying pathological features is poor, and the possibility of misjudgment or misjudgment is greatly improved.
Disclosure of Invention
The application provides a capsule endoscope system for fluorescence analysis and a control method, which are used for solving the technical problems of poor efficiency of pathological feature recognition and high possibility of erroneous judgment or misjudgment caused by the fact that the light intensity of an endoscope light source cannot be adaptively adjusted.
In view of the above, the present application provides, in a first aspect, a fluorescence-analyzed capsule endoscope system, comprising: a capsule endoscope and an in vitro monitoring unit;
a light source module, an image acquisition module and a control module are arranged in the capsule endoscope;
the light source module is used for emitting exciting light for exciting a fluorescent probe to generate a fluorescent signal to the fluorescent probe of target tissue preset in a patient;
the image acquisition module is used for acquiring fluorescence image data of the target tissue after detecting the fluorescence signal and transmitting the fluorescence image data to the control module;
the control module is used for carrying out data interaction with the in-vitro monitoring unit, controlling exciting light emitted by the light source module according to a control instruction transmitted by the in-vitro monitoring unit, and transmitting the fluorescence image data to the in-vitro monitoring unit after receiving the fluorescence image data transmitted by the image acquisition module;
the in-vitro monitoring unit is used for displaying a fluorescence image after receiving the fluorescence image data transmitted by the control module, obtaining the fluorescence intensity generated by the fluorescence probe according to the fluorescence image data, and sending a light intensity adjusting signal to the control module according to the fluorescence intensity so as to adjust the light intensity of the exciting light emitted by the light source module.
Preferably, the light source module includes a plurality of LED lamps with different wavelengths and an LED driver electrically connected to the plurality of LED lamps with different wavelengths, and the LED driver is electrically connected to the control module.
Preferably, the magnetic control capsule endoscope further comprises a magnetic field control table, the capsule endoscope is further provided with a magnetic induction module, and the magnetic field control table is used for carrying out magnetic control on the magnetic induction module.
Preferably, the control module is provided with a data transmission sub-module for transmitting data interacted between the control module and the extracorporeal monitoring unit.
Preferably, the in vitro monitoring unit comprises a display module, an average fluorescence intensity calculation module and a light intensity adjusting module;
the display module is used for displaying the fluorescence image after receiving the fluorescence image data;
the average fluorescence intensity calculating module is used for calculating the average fluorescence intensity generated by the fluorescence probe according to the fluorescence image data;
the light intensity adjusting module is used for sending the light intensity adjusting signal according to a preset fluorescence intensity threshold value and the average fluorescence intensity, and is also used for sending a light intensity reducing adjusting signal to the control module when the average fluorescence intensity is higher than the preset fluorescence intensity threshold value, and is also used for sending a light intensity increasing adjusting signal to the control module when the average fluorescence intensity is lower than the preset fluorescence intensity threshold value.
Preferably, the in-vitro monitoring unit further comprises an image enhancement module, wherein the image enhancement module comprises a deep learning sub-module, a preprocessing sub-module and a segmentation sub-module;
the deep learning sub-module is used for identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network so as to cut the fluorescence image to form the specific lesion partial image;
the preprocessing submodule is used for carrying out graying processing on the specific lesion part image to obtain a gray image and also used for carrying out histogram equalization on the gray image to obtain a new gray image after image enhancement;
the segmentation submodule is used for solving a maximum inter-class variance threshold value of the new gray image based on a preset maximum inter-class variance gray threshold value algorithm and eliminating pixel points lower than the maximum inter-class variance threshold value in the new gray image based on a preset automatic threshold value segmentation algorithm so as to obtain an enhanced gray image;
the average fluorescence intensity calculation module is further used for determining the average fluorescence intensity generated by the fluorescence probe according to the average gray value of the enhanced gray image.
In another aspect, the present invention further provides a method for controlling a capsule endoscope system based on the fluorescence analysis, including the steps of:
emitting exciting light for exciting a fluorescent probe to generate a fluorescent signal to the fluorescent probe of target tissue preset in a patient body through a light source module;
after the fluorescence signal is detected by an image acquisition module, acquiring fluorescence image data of the target tissue, and transmitting the fluorescence image data to a control module;
transmitting the fluorescence image data to an in vitro monitoring unit through the control module;
the in-vitro monitoring unit receives the fluorescence image data and then displays a fluorescence image, meanwhile, the fluorescence intensity generated by the fluorescence probe is obtained according to the fluorescence image data, and a light intensity adjusting signal is sent to the control module according to the fluorescence intensity;
and adjusting the light intensity of exciting light emitted by the light source module through the control module according to the light intensity adjusting signal.
Preferably, the capsule endoscope system for fluorescence analysis comprises a magnetic field control console, the capsule endoscope is provided with a magnetic induction module, and the step of emitting excitation light for exciting a fluorescent probe to generate a fluorescent signal to the fluorescent probe of target tissue preset in a patient body through a light source module comprises the following steps:
and magnetically controlling the magnetic induction module through the magnetic field control console so that the magnetic induction module reaches a position corresponding to the fluorescent probe of a target tissue preset in the body of a patient.
Preferably, the step of obtaining the fluorescence intensity generated by the fluorescence probe according to the fluorescence image data and sending a light intensity adjusting signal to the control module according to the fluorescence intensity specifically includes:
calculating the average fluorescence intensity generated by the fluorescent probe according to the fluorescence image data;
and sending the light intensity adjusting signal according to a preset fluorescence intensity threshold and the average fluorescence intensity, specifically, sending a light intensity decreasing adjusting signal to the control module when the average fluorescence intensity is higher than the preset fluorescence intensity threshold, and sending a light intensity increasing adjusting signal to the control module when the average fluorescence intensity is lower than the preset fluorescence intensity threshold.
Preferably, the step of obtaining the fluorescence intensity generated by the fluorescent probe according to the fluorescence image data specifically includes:
identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network, and cutting the fluorescence image to form the specific lesion partial image;
graying the specific lesion part image into a grayscale image, and then carrying out histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement;
solving a maximum inter-class variance threshold of the new gray image based on a preset maximum inter-class variance gray threshold algorithm, and eliminating pixel points lower than the maximum inter-class variance threshold in the new gray image based on a preset automatic threshold segmentation algorithm so as to obtain an enhanced gray image;
and determining the average fluorescence intensity generated by the fluorescent probe according to the average gray value of the enhanced gray image.
According to the technical scheme, the invention has the following advantages:
the invention provides a capsule endoscope system for fluorescence analysis, which can emit exciting light for exciting a fluorescence probe to generate a fluorescence signal through a light source module, acquire fluorescence image data with the fluorescence signal after detecting the fluorescence signal through an image acquisition module, obtain the fluorescence intensity generated by the fluorescence probe according to the fluorescence image data through an in-vitro monitoring unit, and send a light intensity adjusting signal to a control module according to the fluorescence intensity, so that the light intensity of the exciting light emitted by the light source module is adjusted, the light intensity adaptability adjustment of an endoscope light source can be realized, the efficiency of identifying pathological characteristics is improved, and the possibility of erroneous judgment or erroneous judgment is reduced. The beneficial effects of the control method of the capsule endoscope system for fluorescence analysis provided by the invention are consistent with those described above, and are not repeated herein.
Drawings
FIG. 1 is a schematic structural diagram of a fluorescence analysis capsule endoscope system provided by an embodiment of the present application;
fig. 2 is a schematic structural diagram of an image enhancement module according to an embodiment of the present disclosure;
fig. 3 is a flowchart of a control method of a fluorescence analysis capsule endoscope system according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions of the present application better understood, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
To facilitate understanding, referring to fig. 1, the present application provides a fluorescence analysis capsule endoscope system comprising: a capsule endoscope 100 and an extracorporeal monitoring unit 200;
a light source module 101, an image acquisition module 102 and a control module 103 are arranged in the capsule endoscope 100;
the light source module 101 is used for emitting excitation light for exciting a fluorescent probe to generate a fluorescent signal to the fluorescent probe of a target tissue preset in a patient;
it is understood that the fluorescent probe can be targeted to the target tissue in the patient in advance, the light source module 101 adopts the adjustable light source module 101, and the control module 103 can control the light source module 101 to emit the excitation light with the corresponding wavelength according to the excitation wavelength of the fluorescent probe.
In one embodiment, the light source module 101 includes a plurality of LED lamps with different wavelengths and an LED driver electrically connected to the plurality of LED lamps with different wavelengths, and the LED driver is electrically connected to the control module 103.
The image acquisition module 102 is configured to acquire fluorescence image data of a target tissue after detecting a fluorescence signal, and is further configured to transmit the fluorescence image data to the control module 103;
the control module 103 performs data interaction with the in-vitro monitoring unit 200, and the control module 103 is configured to control excitation light emitted by the light source module 101 according to a control instruction transmitted by the in-vitro monitoring unit 200, and is further configured to transmit fluorescence image data to the in-vitro monitoring unit 200 after receiving the fluorescence image data transmitted by the image acquisition module 102;
it is understood that the data interaction between the control module 103 and the extracorporeal monitoring unit 200 may be wired transmission or wireless transmission.
The in-vitro monitoring unit 200 is configured to display a fluorescence image after receiving the fluorescence image data transmitted by the control module 103, obtain fluorescence intensity generated by the fluorescence probe according to the fluorescence image data, and send a light intensity adjusting signal to the control module 103 according to the fluorescence intensity, so as to adjust light intensity of excitation light emitted by the light source module 101.
It is understood that the fluorescence image data acquired by the image acquisition module 102 is image data with fluorescence signals, and therefore, the fluorescence intensity generated by the fluorescence probe can be obtained from the fluorescence image data.
Further, the system further comprises a magnetic field control console, and the capsule endoscope 100 is further provided with a magnetic induction module, and the magnetic field control console is used for performing magnetic control on the magnetic induction module.
It is understood that the magnetic induction module may be a magnet.
Further, the control module 103 is provided with a data transmission sub-module for transmitting data exchanged between the control module 103 and the extracorporeal monitoring unit 200.
Meanwhile, a power supply module 104 is further arranged in the capsule endoscope 100, and the power supply module 104 is electrically connected with the control module 103 and used for supplying power to each module in the capsule endoscope 100.
Further, the in vitro monitoring unit 200 includes a display module 201, an average fluorescence intensity calculating module 202, and an intensity adjusting module 203;
the display module 201 is configured to display a fluorescence image after receiving fluorescence image data;
the average fluorescence intensity calculating module 202 is used for calculating the average fluorescence intensity generated by the fluorescent probe according to the fluorescence image data;
the light intensity adjusting module 203 is configured to send a light intensity adjusting signal according to a preset fluorescence intensity threshold and an average fluorescence intensity, send a light intensity decreasing adjusting signal to the control module 103 when the average fluorescence intensity is higher than the preset fluorescence intensity threshold, and send a light intensity increasing adjusting signal to the control module 103 when the average fluorescence intensity is lower than the preset fluorescence intensity threshold.
The embodiment takes the average fluorescence intensity as the reference value of light intensity adjustment, and the preset fluorescence intensity can be set according to the experiment, so that the accuracy of light intensity adjustment can be improved.
Further, the in-vitro monitoring unit 200 further includes an image enhancement module 105, please refer to fig. 2, the image enhancement module 105 includes a deep learning sub-module 1051, a preprocessing sub-module 1052 and a segmentation sub-module 1053;
a deep learning sub-module 1051, configured to identify a specific lesion partial image in the fluorescence image based on a preset U-Net network, so as to cut the fluorescence image to form the specific lesion partial image;
the preprocessing submodule 1052 is used for performing graying processing on the specific lesion part image to obtain a grayscale image, and is also used for performing histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement;
the segmentation submodule 1053 is used for solving the maximum inter-class variance threshold of the new gray image based on a preset maximum inter-class variance gray threshold algorithm, and also used for eliminating pixel points lower than the maximum inter-class variance threshold in the new gray image based on a preset automatic threshold segmentation algorithm so as to obtain an enhanced gray image;
the mean fluorescence intensity calculation module 202 is further configured to determine a mean fluorescence intensity generated by the fluorescent probe according to the mean gray scale value of the enhanced gray scale image.
The working process of the image enhancement module 105 is as follows:
1) Identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network, and cutting the fluorescence image to enable the fluorescence image to be cut into the specific lesion partial image, wherein the preset U-Net network is obtained by training a training set of a large number of specific lesion partial images, so that the specific lesion partial image can be identified, and the fluorescence image is cut to obtain the specific lesion partial image;
2) Graying the specific lesion part image into a grayscale image, and then carrying out histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement;
the specific process of histogram equalization on the gray level image is as follows:
taking the gray level image as an original image, and calculating a histogram of the corresponding gray level within the range of 0-255;
the gray distribution probability is calculated by the following formula,
h s (n)＝h(n)/N
in the formula, h s (N) represents a gray distribution probability, h (N) represents the number of pixels of each gray level, N is the total pixels of the image, wherein N = I × w, I represents a gray value, and w represents the number of pixel points corresponding to the gray value;
the cumulative distribution of gray levels is calculated by the following formula,
in the formula, h p (n) represents a cumulative distribution of gray levels, h s (k) A gray distribution probability representing a kth gray level, i representing a gray level;
histogram equalization is performed by the following formula to calculate the gray value of the new gray image after image enhancement,
in the formula, g (n, j) represents a pixel point of a new gray image after image enhancement, f (n, j) represents a pixel point of a gray image before image enhancement, f (n, j) =0 represents that the gray value of the pixel point does not need to be enhanced, and h p (k) A cumulative distribution representing a kth gray level;
the gray value of the new gray image is calculated, so that the gray conversion of the corresponding pixel point is completed, and the image enhancement effect is achieved.
3) Solving the maximum between-class variance threshold value of the new gray image by using a preset maximum between-class variance gray threshold value algorithm, wherein the solving formula is as follows:
ICV＝PA×(MA-M) 2 +PB×(MB-M) 2
the ICV represents a maximum inter-class variance threshold, M represents a preset gray mean value, a set of pixel points with gray values smaller than M is a dark area and is marked as PA, a set of pixel points with gray values larger than M is a bright area and is marked as PB, MA represents the gray mean value of pixel points in the dark area, and MB represents the gray mean value of pixel points in the bright area;
after the maximum interclass variance threshold ICV is obtained through the formula, a preset automatic threshold segmentation algorithm eliminates pixel points lower than the maximum interclass variance threshold in a new gray image so as to obtain an enhanced gray image, and the background interference of irrelevant pixel points is eliminated from the enhanced gray image at the moment, so that the average fluorescence intensity is calculated more accurately.
In one embodiment, the mean fluorescence intensity is solved by the following formula,
wherein Mean represents Mean fluorescence intensity; intDen denotes the sum of fluorescence intensities of the enhanced gray scale images, intDen = I 1 ×Area，I 1 And expressing the average gray value of the enhanced gray image, and expressing the total number of pixel points of the enhanced gray image by Area.
It should be noted that, in this embodiment, the light source module 101 may emit excitation light for exciting the fluorescence probe to generate a fluorescence signal, the image acquisition module 102 acquires fluorescence image data with the fluorescence signal after detecting the fluorescence signal, the extracorporeal monitoring unit 200 obtains fluorescence intensity generated by the fluorescence probe according to the fluorescence image data, and further sends a light intensity adjustment signal to the control module 103 according to the fluorescence intensity, so as to adjust light intensity of the excitation light emitted by the light source module 101, and implement adaptive adjustment of light intensity of the endoscope, thereby improving efficiency of identifying pathological features, and reducing possibility of erroneous judgment or erroneous judgment.
The above is a detailed description of an embodiment of the fluorescence analysis capsule endoscope system provided by the present invention, and the following is a detailed description of an embodiment of a control method of the fluorescence analysis capsule endoscope system provided by the present invention.
For convenience of understanding, referring to fig. 3, the present invention provides a control method of a capsule endoscope system based on fluorescence analysis of the above embodiment, including the steps of:
s1: emitting exciting light for exciting the fluorescent probe to generate a fluorescent signal to the fluorescent probe of the target tissue preset in the patient body through the light source module;
s2: after the fluorescence signal is detected by the image acquisition module, acquiring fluorescence image data of a target tissue, and transmitting the fluorescence image data to the control module;
s3: transmitting the fluorescence image data to an in vitro monitoring unit through a control module;
s4: the in-vitro monitoring unit receives the fluorescence image data and then displays the fluorescence image, meanwhile, the fluorescence intensity generated by the fluorescence probe is obtained according to the fluorescence image data, and a light intensity adjusting signal is sent to the control module according to the fluorescence intensity;
s5: the light intensity of exciting light emitted by the light source module is adjusted through the control module according to the light intensity adjusting signal.
Further, before step S1, the method includes:
and magnetically controlling the magnetic induction module through the magnetic field control table so that the magnetic induction module reaches a position corresponding to a fluorescent probe of a target tissue preset in the body of the patient.
Further, the step S4 of obtaining the fluorescence intensity generated by the fluorescence probe according to the fluorescence image data and sending the light intensity adjusting signal to the control module according to the fluorescence intensity specifically includes:
s401: calculating the average fluorescence intensity generated by the fluorescent probe according to the fluorescence image data;
s402: and sending a light intensity adjusting signal according to a preset fluorescence intensity threshold value and the average fluorescence intensity, specifically, sending a light intensity decreasing adjusting signal to the control module when the average fluorescence intensity is higher than the preset fluorescence intensity threshold value, and sending a light intensity increasing adjusting signal to the control module when the average fluorescence intensity is lower than the preset fluorescence intensity threshold value.
In another embodiment, the step of obtaining the fluorescence intensity generated by the fluorescent probe from the fluorescence image data in step S4 specifically includes:
s411: identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network, and cutting the fluorescence image to enable the fluorescence image to be cut to form the specific lesion partial image;
s412: graying the specific lesion part image into a grayscale image, and then carrying out histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement;
s413: solving a maximum inter-class variance threshold of the new gray image based on a preset maximum inter-class variance gray threshold algorithm, and eliminating pixel points lower than the maximum inter-class variance threshold in the new gray image based on a preset automatic threshold segmentation algorithm so as to obtain an enhanced gray image;
s414: and determining the average fluorescence intensity generated by the fluorescent probe according to the average gray value of the enhanced gray image.
Specifically, the following is the operation procedure of steps S411 to S414:
s421: identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network, and cutting the fluorescence image to enable the fluorescence image to be cut into the specific lesion partial image, wherein the preset U-Net network is obtained by training a training set of a large number of specific lesion partial images, so that the specific lesion partial image can be identified, and the fluorescence image is cut to obtain the specific lesion partial image;
s422: graying the specific lesion part image into a grayscale image, and then carrying out histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement;
the specific process of histogram equalization on the gray level image is as follows:
taking the gray level image as an original image, and calculating a histogram of the corresponding gray level within the range of 0-255;
the gray distribution probability is calculated by the following formula,
h s (n)＝h(n)/N
in the formula, h s (N) represents a gray distribution probability, h (N) represents the number of pixels of each gray level, N is the total pixels of the image, wherein N = I × w, I represents a gray value, and w represents the number of pixel points corresponding to the gray value;
the cumulative distribution of gray levels is calculated by the following formula,
in the formula, h p (n) represents a cumulative distribution of gray levels, h s (k) A gray distribution probability representing a kth gray level, i representing a gray level;
histogram equalization is performed by the following formula to calculate the gray value of the new gray image after image enhancement,
in the formula, g (n, j) represents a pixel point of a new gray image after image enhancement, f (n, j) represents a pixel point of a gray image before image enhancement, f (n, j) =0 represents that the gray value of the pixel point does not need to be enhanced, and h p (k) A cumulative distribution representing a kth gray level;
the gray value of the new gray image is calculated, so that the gray conversion of the corresponding pixel point is completed, and the effect of image enhancement is achieved.
S423: solving the maximum between-class variance threshold value of the new gray image by using a preset maximum between-class variance gray threshold value algorithm, wherein the solving formula is as follows:
ICV＝PA×(MA-M) 2 +PB×(MB-M) 2
the ICV represents a maximum inter-class variance threshold, M represents a preset gray mean value, a set of pixel points with gray values smaller than M is a dark area and is marked as PA, a set of pixel points with gray values larger than M is a bright area and is marked as PB, MA represents the gray mean value of pixel points in the dark area, and MB represents the gray mean value of pixel points in the bright area;
s424: after the maximum inter-class variance threshold ICV is solved through the formula, a preset automatic threshold segmentation algorithm eliminates the pixel points which are lower than the maximum inter-class variance threshold in the new gray level image, so that an enhanced gray level image is obtained, the background interference of irrelevant pixel points is eliminated in the enhanced gray level image at the moment, and the calculation of the average fluorescence intensity is more accurate.
S425: the average fluorescence intensity was solved by the following formula,
wherein Mean represents Mean fluorescence intensity; intDen denotes the sum of fluorescence intensities of the enhanced gray scale images, intDen = I 1 ×Area，I 1 And expressing the average gray value of the enhanced gray image, and expressing the total number of pixel points of the enhanced gray image by Area.
It should be noted that, in this embodiment, the light source module may emit excitation light for exciting the fluorescence probe to generate a fluorescence signal, the image acquisition module acquires fluorescence image data with the fluorescence signal after detecting the fluorescence signal, the extracorporeal monitoring unit obtains fluorescence intensity generated by the fluorescence probe according to the fluorescence image data, and the extracorporeal monitoring unit further sends a light intensity adjusting signal to the control module according to the fluorescence intensity, so as to adjust light intensity of the excitation light emitted by the light source module, and implement adaptive adjustment of light intensity of the endoscope light source, thereby improving efficiency of identifying pathological features, and reducing possibility of erroneous judgment or misjudgment.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.
Claims (5)
1. A fluorescence-analyzed capsule endoscopic system, comprising: a capsule endoscope and an in vitro monitoring unit;
a light source module, an image acquisition module and a control module are arranged in the capsule endoscope;
the light source module is used for emitting exciting light for exciting a fluorescent probe to generate a fluorescent signal to the fluorescent probe of target tissue preset in a patient;
the image acquisition module is used for acquiring fluorescence image data of the target tissue after detecting the fluorescence signal and transmitting the fluorescence image data to the control module;
the control module is used for carrying out data interaction with the in-vitro monitoring unit, controlling exciting light emitted by the light source module according to a control instruction transmitted by the in-vitro monitoring unit, and transmitting the fluorescence image data to the in-vitro monitoring unit after receiving the fluorescence image data transmitted by the image acquisition module;
the in-vitro monitoring unit is used for displaying a fluorescence image after receiving the fluorescence image data transmitted by the control module, obtaining the fluorescence intensity generated by the fluorescence probe according to the fluorescence image data, and sending a light intensity adjusting signal to the control module according to the fluorescence intensity so as to adjust the light intensity of the exciting light emitted by the light source module;
the in-vitro monitoring unit comprises a display module, an average fluorescence intensity calculation module and a light intensity adjusting module;
the display module is used for displaying the fluorescence image after receiving the fluorescence image data;
the average fluorescence intensity calculating module is used for calculating the average fluorescence intensity generated by the fluorescence probe according to the fluorescence image data;
the light intensity adjusting module is used for sending the light intensity adjusting signal according to a preset fluorescence intensity threshold value and the average fluorescence intensity, sending a light intensity reducing adjusting signal to the control module when the average fluorescence intensity is higher than the preset fluorescence intensity threshold value, and sending a light intensity increasing adjusting signal to the control module when the average fluorescence intensity is lower than the preset fluorescence intensity threshold value;
the in-vitro monitoring unit also comprises an image enhancement module, wherein the image enhancement module comprises a deep learning submodule, a preprocessing submodule and a segmentation submodule;
the deep learning sub-module is used for identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network so as to cut the fluorescence image to form the specific lesion partial image;
the preprocessing submodule is used for carrying out graying processing on the specific lesion part image to obtain a grayscale image and also used for carrying out histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement; the specific process of performing histogram equalization on the grayscale image comprises the following steps: taking the gray level image as an original image, and calculating a histogram of the corresponding gray level in the range of 0-255;
the gray distribution probability is calculated by the following formula,
h s (n)＝h(n)/N
in the formula, h s (N) represents a gray distribution probability, h (N) represents the number of pixels of each gray level, N is the total pixels of the image, wherein N = I × w, I represents a gray value, and w represents the number of pixel points corresponding to the gray value;
the cumulative distribution of gray levels is calculated by the following formula,
in the formula, h p (n) represents a cumulative distribution of gray levels, h s (k) A gray distribution probability representing a kth gray level, i representing a gray level;
histogram equalization is performed by the following formula to calculate the gray value of the new gray image after image enhancement,
in the formula, g (n, j) represents a pixel point of a new gray image after image enhancement, f (n, j) represents a pixel point of a gray image before image enhancement, f (n, j) =0 represents that the gray value of the pixel point does not need to be enhanced, and h p (k) A cumulative distribution representing a kth gray level;
the segmentation submodule is used for solving a maximum inter-class variance threshold value of the new gray image based on a preset maximum inter-class variance gray threshold value algorithm and eliminating pixel points lower than the maximum inter-class variance threshold value in the new gray image based on a preset automatic threshold value segmentation algorithm so as to obtain an enhanced gray image; the solving formula is as follows:
ICV＝PA×(MA-M) 2 +PB×(MB-M) 2
the ICV represents a maximum inter-class variance threshold, M represents a preset gray mean value, a pixel point set with a gray value smaller than M is a dark area and is marked as PA, a pixel point set with a gray value larger than M is a bright area and is marked as PB, MA represents a gray mean value of pixel points in the dark area, MB represents a gray mean value of pixel points in the bright area, and different gray mean values M correspond to different inter-class variance threshold values;
the average fluorescence intensity calculation module is further used for determining the average fluorescence intensity generated by the fluorescence probe according to the average gray value of the enhanced gray image; wherein the average fluorescence intensity is solved by the following formula,
wherein Mean represents Mean fluorescence intensity; intDen denotes the sum of fluorescence intensities of the enhanced grayscale images, intDen = I 1 ×Area，I 1 And expressing the average gray value of the enhanced gray image, and expressing the total number of pixel points of the enhanced gray image by Area.
2. The fluorescence analyzing capsule endoscopic system of claim 1, wherein the light source module comprises a plurality of different wavelength LED lights and an LED driver electrically connected to the plurality of different wavelength LED lights, the LED driver being electrically connected to the control module.
3. The fluorescence analysis capsule endoscopic system of claim 1, further comprising a magnetic field console, said capsule endoscope further provided with a magnetic induction module, said magnetic field console for magnetically controlling said magnetic induction module.
4. The fluorescence analyzing capsule endoscopic system of claim 1, wherein the control module is provided with a data transmission sub-module for transmitting data for interaction between the control module and the extracorporeal monitoring unit.
5. A method for controlling a capsule endoscope system based on fluorescence analysis according to claim 1, comprising the steps of:
emitting exciting light for exciting a fluorescent probe to generate a fluorescent signal to the fluorescent probe of target tissue preset in a patient body through a light source module;
after the fluorescence signal is detected by an image acquisition module, acquiring fluorescence image data of the target tissue, and transmitting the fluorescence image data to a control module;
transmitting the fluorescence image data to an in vitro monitoring unit through the control module;
the in-vitro monitoring unit receives the fluorescence image data and then displays a fluorescence image, meanwhile, the fluorescence intensity generated by the fluorescence probe is obtained according to the fluorescence image data, and a light intensity adjusting signal is sent to the control module according to the fluorescence intensity;
adjusting the light intensity of exciting light emitted by the light source module through the control module according to the light intensity adjusting signal;
the step of obtaining the fluorescence intensity generated by the fluorescent probe according to the fluorescence image data specifically includes:
identifying a specific lesion partial image in the fluorescence image based on a preset U-Net network, and cutting the fluorescence image to form the specific lesion partial image;
graying the specific lesion part image into a grayscale image, and then carrying out histogram equalization on the grayscale image to obtain a new grayscale image after image enhancement; the method specifically comprises the following steps: taking the gray level image as an original image, and calculating a histogram of the corresponding gray level in the range of 0-255;
the gray distribution probability is calculated by the following formula,
h s (n)＝h(n)/N
in the formula, h s (N) represents a gray distribution probability, h (N) represents the number of pixels of each gray level, N is the total pixels of the image, wherein N = I × w, I represents a gray value, and w represents the number of pixel points corresponding to the gray value;
the cumulative distribution of gray levels is calculated by the following formula,
in the formula, h p (n) represents a cumulative distribution of gray levels, h s (k) A gray distribution probability representing a kth gray level, i represents a gray level;
histogram equalization is performed by the following formula to calculate the gray value of the new gray image after image enhancement,
in the formula, g (n, j) represents a pixel point of a new gray image after image enhancement, f (n, j) represents a pixel point of a gray image before image enhancement, f (n, j) =0 represents that the gray value of the pixel point does not need to be enhanced, and h p (k) A cumulative distribution representing a kth gray level;
solving a maximum inter-class variance threshold of the new gray image based on a preset maximum inter-class variance gray threshold algorithm, and eliminating pixel points lower than the maximum inter-class variance threshold in the new gray image based on a preset automatic threshold segmentation algorithm so as to obtain an enhanced gray image; the solving formula is as follows:
ICV＝PA×(MA-M) 2 +PB×(MB-M) 2
the ICV represents a maximum inter-class variance threshold, M represents a preset gray mean value, a pixel point set with a gray value smaller than M is a dark area and is marked as PA, a pixel point set with a gray value larger than M is a bright area and is marked as PB, MA represents a gray mean value of pixel points in the dark area, MB represents a gray mean value of pixel points in the bright area, and different gray mean values M correspond to different inter-class variance threshold values;
determining the average fluorescence intensity generated by the fluorescence probe according to the average gray value of the enhanced gray image; wherein the average fluorescence intensity is solved by the following formula,
wherein Mean represents Mean fluorescence intensity; intDen denotes the sum of fluorescence intensities of the enhanced gray scale images, intDen = I 1 ×Area，I 1 Expressing the average gray value of the enhanced gray image, and expressing the total number of pixel points of the enhanced gray image by Area;
the step of obtaining the fluorescence intensity generated by the fluorescence probe according to the fluorescence image data and sending a light intensity adjusting signal to the control module according to the fluorescence intensity specifically comprises the following steps:
calculating the average fluorescence intensity generated by the fluorescent probe according to the fluorescence image data;
and sending the light intensity adjusting signal according to a preset fluorescence intensity threshold and the average fluorescence intensity, specifically, sending a light intensity decreasing adjusting signal to the control module when the average fluorescence intensity is higher than the preset fluorescence intensity threshold, and sending a light intensity increasing adjusting signal to the control module when the average fluorescence intensity is lower than the preset fluorescence intensity threshold.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110240597.7A CN112914477B (en) | 2021-03-04 | 2021-03-04 | Capsule endoscope system for fluorescence analysis and control method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110240597.7A CN112914477B (en) | 2021-03-04 | 2021-03-04 | Capsule endoscope system for fluorescence analysis and control method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112914477A CN112914477A (en) | 2021-06-08 |
| CN112914477B true CN112914477B (en) | 2023-03-14 |
Family
ID=76173330
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110240597.7A Active CN112914477B (en) | 2021-03-04 | 2021-03-04 | Capsule endoscope system for fluorescence analysis and control method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112914477B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113288014A (en) * | 2021-07-05 | 2021-08-24 | 上海交通大学 | Capsule endoscope system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201300068A (en) * | 2011-06-17 | 2013-01-01 | Chung Shan Inst Of Science | Capsule endoscope device with fluorescent target molecular imaging |
| CN106950786A (en) * | 2016-01-07 | 2017-07-14 | 深圳市光峰光电技术有限公司 | Light source module and its light source control method, optical projection system |
| CN107436299A (en) * | 2017-07-25 | 2017-12-05 | 苏州德锐特成像技术有限公司 | A kind of bioluminescence imaging brightness automatically analyzes the method and apparatus of regulation |
| CN108469525A (en) * | 2018-06-30 | 2018-08-31 | 武汉生之源生物科技股份有限公司 | A kind of fluoroimmunoassay device and method |
| CN211155671U (en) * | 2019-10-28 | 2020-08-04 | 重庆金山医疗技术研究院有限公司 | Fluorescent capsule endoscope, wireless signal receiving device, and endoscope system |
| CN112364871A (en) * | 2020-10-27 | 2021-02-12 | 重庆大学 | Part code spraying character segmentation method based on improved projection algorithm |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107561264B (en) * | 2017-09-05 | 2020-03-27 | 齐鲁工业大学 | β amyloid plaque identification and measurement method based on image processing |
| CN111428066B (en) * | 2020-04-24 | 2021-08-24 | 南京图格医疗科技有限公司 | Method for classifying and segmenting lesion image based on convolutional neural network |
-
2021
- 2021-03-04 CN CN202110240597.7A patent/CN112914477B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW201300068A (en) * | 2011-06-17 | 2013-01-01 | Chung Shan Inst Of Science | Capsule endoscope device with fluorescent target molecular imaging |
| CN106950786A (en) * | 2016-01-07 | 2017-07-14 | 深圳市光峰光电技术有限公司 | Light source module and its light source control method, optical projection system |
| CN107436299A (en) * | 2017-07-25 | 2017-12-05 | 苏州德锐特成像技术有限公司 | A kind of bioluminescence imaging brightness automatically analyzes the method and apparatus of regulation |
| CN108469525A (en) * | 2018-06-30 | 2018-08-31 | 武汉生之源生物科技股份有限公司 | A kind of fluoroimmunoassay device and method |
| CN211155671U (en) * | 2019-10-28 | 2020-08-04 | 重庆金山医疗技术研究院有限公司 | Fluorescent capsule endoscope, wireless signal receiving device, and endoscope system |
| CN112364871A (en) * | 2020-10-27 | 2021-02-12 | 重庆大学 | Part code spraying character segmentation method based on improved projection algorithm |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112914477A (en) | 2021-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7068487B2 (en) | Electronic endoscopy system | |
| EP2098160B1 (en) | In-vivo image acquiring apparatus, in-vivo image receiving apparatus, in-vivo image displaying apparatus, and noise eliminating method | |
| CN110461210A (en) | Endoscopic system and its working method | |
| JP5594711B2 (en) | Method for enhancing in-vivo image contrast | |
| EP3332689A1 (en) | Pen-type medical fluorescent imaging device and system for aligning multiple fluorescent images using same | |
| JP2006122502A (en) | Image processing method and capsulated endoscopic device | |
| US20190110716A1 (en) | Method and apparatus for jaundice diagnosis based on an image | |
| US20130064436A1 (en) | Medical image processing apparatus and method of operating medical image processing apparatus | |
| JP2022189900A (en) | Image processing device, endoscope system, and operation method of image processing device | |
| CN112914477B (en) | Capsule endoscope system for fluorescence analysis and control method | |
| CN112351723B (en) | Electronic endoscope system and data processing device | |
| US20220375047A1 (en) | Medical imaging | |
| CN109241963B (en) | Adaboost machine learning-based intelligent identification method for bleeding point in capsule gastroscope image | |
| CN108463157A (en) | Endoscope-use processor | |
| CN105942960B (en) | A kind of endoscopic system and method that may indicate that ureter position | |
| US11464393B2 (en) | Endoscope apparatus and method of operating endoscope apparatus | |
| DE112015002614T5 (en) | Image processing device, image processing method and image processing program | |
| US20230039047A1 (en) | Image processing apparatus, image processing method, navigation method and endoscope system | |
| CN107405050A (en) | Endoscopic system and evaluation value calculation apparatus | |
| JP2007236598A (en) | Processor and electronic endoscope system | |
| US20160210746A1 (en) | Organ imaging device | |
| CN206151422U (en) | System for detect in abdominal cavity operation and show ureter | |
| US11800967B2 (en) | Endoscopic image processing apparatus, endoscopic image processing method, and recording medium recording program | |
| CN115546039A (en) | Image dimming method, device, equipment and storage medium of endoscope system | |
| US11045071B2 (en) | Image processing apparatus for endoscope and endoscope system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |