CN116402711A - Image processing method, device and computer readable storage medium - Google Patents
Image processing method, device and computer readable storage medium Download PDFInfo
- Publication number
- CN116402711A CN116402711A CN202310323697.5A CN202310323697A CN116402711A CN 116402711 A CN116402711 A CN 116402711A CN 202310323697 A CN202310323697 A CN 202310323697A CN 116402711 A CN116402711 A CN 116402711A
- Authority
- CN
- China
- Prior art keywords
- image
- polarization
- polarized
- polarized light
- image processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 45
- 230000010287 polarization Effects 0.000 claims abstract description 241
- 239000013598 vector Substances 0.000 claims description 42
- 230000003287 optical effect Effects 0.000 claims description 32
- 238000007781 pre-processing Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 5
- 210000004369 blood Anatomy 0.000 abstract description 21
- 239000008280 blood Substances 0.000 abstract description 21
- 239000000428 dust Substances 0.000 abstract description 18
- 238000003384 imaging method Methods 0.000 abstract description 13
- 238000000034 method Methods 0.000 abstract description 12
- 210000001519 tissue Anatomy 0.000 description 20
- 230000006870 function Effects 0.000 description 15
- 230000005540 biological transmission Effects 0.000 description 10
- 239000011159 matrix material Substances 0.000 description 7
- 238000004891 communication Methods 0.000 description 6
- 239000003595 mist Substances 0.000 description 6
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 4
- 206010028980 Neoplasm Diseases 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 238000002834 transmittance Methods 0.000 description 2
- 241000023320 Luma <angiosperm> Species 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001839 endoscopy Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000007850 fluorescent dye Substances 0.000 description 1
- 230000008014 freezing Effects 0.000 description 1
- 238000007710 freezing Methods 0.000 description 1
- 210000001035 gastrointestinal tract Anatomy 0.000 description 1
- 210000004907 gland Anatomy 0.000 description 1
- 230000003902 lesion Effects 0.000 description 1
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 1
- 210000004400 mucous membrane Anatomy 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 210000004881 tumor cell Anatomy 0.000 description 1
- 210000001835 viscera Anatomy 0.000 description 1
Images
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
-
- G06T5/70—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10068—Endoscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20172—Image enhancement details
Abstract
The invention discloses an image processing method, an image processing device and a computer readable storage medium, wherein the method comprises the following steps: acquiring to-be-processed polarized images corresponding to a plurality of polarization angles of a polarized light image sensor based on polarized information pixel data acquired by the polarized light image sensor; determining an optimal polarization state image and a worst polarization state image based on the polarization image to be processed; a target sharp image is determined based on the best polarization state image and the worst polarization state image. The invention realizes the function of clearly imaging the polarized light endoscope device in the underwater environment (blood, turbidity, tissue dust, fog) so that when the conditions of blood, tissue dust, fog and the like occur in the operation environment, the problem of unclear output image of the polarized light endoscope device is solved by clearly processing the polarized information pixel data.
Description
Technical Field
The present invention relates to the field of endoscope technology, and in particular, to an image processing method, an image processing device, and a computer readable storage medium.
Background
The endoscope is a medical electronic optical instrument which can be inserted into the cavity of human body and the internal cavity of viscera to directly observe, diagnose and treat, and adopts an optical lens with very small size to optically image the object in the cavity to be observed through a tiny objective imaging system, then the optical imaging is sent to an image processing host, and finally the observed image after the image processing is output on a display screen for observation and diagnosis by doctors.
Currently, endoscopes mainly include ultra-high definition 4K endoscopes, three-wafer endoscopes, fluorescent endoscopes, narrow-band light endoscopes, 3D endoscopes, and the like, and these endoscopes have good functions and performances in respective fields. Wherein, the 4K endoscope has ultra-high definition resolution and good detail reduction; the three-wafer endoscope is provided with three image sensors which respectively collect one monochromatic light signal in RGB, so that RGB color values of each pixel can be obtained, the color reproducibility accuracy is high, and the performance of the three-wafer endoscope is better than that of a common single-wafer endoscope; fluorescent endoscopy is a new imaging technology by utilizing fluorescent molecular imaging in a special spectral environment, and the principle is that tumor cells are marked by fluorescent agents to increase the contrast between lesion tissues and normal tissues, so that cancers and tumors can occur earlier; the narrow-band optical endoscope utilizes a filter to filter out a broadband spectrum in red, blue and green light waves emitted by an endoscope light source, and only the narrow-band spectrum is left for accurately observing the epithelial morphology of the mucous membrane of the digestive tract, such as an epithelial gland concave structure, and can also observe the morphology of an epithelial blood tube network; the shooting scene displayed by the 3D endoscope has depth information, and a doctor can perform operation and treatment on a focus more accurately in operation.
Although the function and performance of the endoscope are good, clear imaging cannot be output under the operation environment with blood, turbid water, tissue scraps and fog, and a doctor is disturbed during operation due to the unclear imaging, and operation can be performed after the disturbance environment (blood, turbid water, tissue scraps and fog) is sometimes cleaned, so that the operation efficiency is lower.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide an image processing method, an image processing device and a computer readable storage medium, and aims to solve the technical problem that a polarized light endoscope device cannot output clear imaging in an underwater environment (blood, turbidity, tissue dust and fog).
In order to achieve the above object, the present invention provides an image processing method applied to a polarized light endoscope apparatus including a camera provided with a polarized light image sensor; the image processing method includes the steps of:
acquiring to-be-processed polarized images corresponding to a plurality of polarization angles of a polarized light image sensor based on polarized information pixel data acquired by the polarized light image sensor;
determining an optimal polarization state image and a worst polarization state image based on the polarization image to be processed;
a target sharp image is determined based on the best polarization state image and the worst polarization state image.
Further, the step of determining the optimal polarization state image and the worst polarization state image based on the polarization image to be processed includes:
determining a total light intensity vector, a linearly polarized light component in the transverse axis direction and a linearly polarized light component in the 45-degree direction based on the polarized image to be processed;
and respectively determining the optimal polarization state image and the worst polarization state image based on the total light intensity vector, the linearly polarized light component in the transverse axis direction and the linearly polarized light component in the 45-degree direction.
Further, the polarization image to be processed comprises a polarization image with a polarization angle of 0 degrees, a polarization image with a polarization angle of 45 degrees, a polarization image with a polarization angle of 90 degrees and a polarization image with a polarization angle of 135 degrees; the step of determining the total light intensity vector, the linearly polarized light component in the transverse axis direction and the linearly polarized light component in the 45-degree direction based on the polarized image to be processed comprises the following steps:
determining a total light intensity vector and a transverse linearly polarized light component based on the polarized image at the 0 ° polarization angle and the polarized image at the 90 ° polarization angle;
the 45 ° direction linearly polarized light component is determined based on the polarized image of the 45 ° polarization angle and the polarized image of the 135 ° polarization angle.
Further, the step of determining the target sharp image based on the best polarization state image and the worst polarization state image includes:
acquiring a polarization degree image corresponding to the polarization image to be processed and a pixel value of the brightest pixel point in the worst polarization state image;
the target sharp image is determined based on the polarization degree image, the pixel value, the optimal polarization state image, and the worst polarization state image.
Further, the step of obtaining the polarization degree image corresponding to the polarization image to be processed includes:
and determining a polarization degree image corresponding to the polarization image to be processed based on the total light intensity vector, the transverse axis direction linear polarized light component and the 45-degree direction linear polarized light component corresponding to the polarization image to be processed.
Further, the step of obtaining the to-be-processed polarized image corresponding to the plurality of polarization angles of the polarized light image sensor based on the polarized information pixel data collected by the polarized light image sensor includes:
acquiring a Bayer format polarized image corresponding to a plurality of polarization angles based on the polarized information pixel data acquired by the polarized light image sensor;
and performing color interpolation processing on the Bayer format polarized image to obtain the polarized image to be processed in an RGB format.
Further, the camera is provided with a Bayer sensor; the image processing method further includes:
acquiring a Bayer image based on pixel data acquired by the Bayer sensor;
preprocessing the Bayer image to obtain an RGB image;
and performing image conversion on the RGB image to obtain a YUV image.
Further, the polarized light endoscope device further comprises a first display and a second display, wherein an optical prism is arranged in the camera and used for dividing one path of light beams output by the optical lens into two paths of light beams; the Bayer sensor is used for collecting the optical signal of one path of light beam output by the optical prism and converting the optical signal into the pixel data; the polarized light image sensor is used for collecting polarized light signals of another path of light beams output by the optical prism and converting the polarized light signals into polarized information pixel data; the image processing method further includes:
and displaying the YUV image through the first display and displaying the clear target image through the second display.
In addition, in order to achieve the above object, the present invention also provides an image processing apparatus including: the image processing device comprises a memory, a processor and an image processing program stored in the memory and capable of running on the processor, wherein the image processing program realizes the steps of the image processing method when being executed by the processor.
In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon an image processing program which, when executed by a processor, implements the steps of the aforementioned image processing method.
According to the invention, polarization information pixel data acquired based on a polarization light image sensor are used for acquiring to-be-processed polarization images corresponding to a plurality of polarization angles of the polarization light image sensor; then, based on the polarization image to be processed, determining an optimal polarization state image and a worst polarization state image; and then, based on the optimal polarized state image and the worst polarized state image, a clear target image is determined, so that the function of clear imaging of the polarized light endoscope device in an underwater environment (blood, turbid, tissue dust, fog) is realized, when the conditions of blood, tissue dust, fog and the like occur in an operation environment, the problem that an output image of the polarized light endoscope device is not clear is solved by clear processing of polarized information pixel data, and the situation that the interference environment (blood, turbid water, tissue dust, fog) is cleared away to directly perform an operation can be avoided, so that the operation efficiency is improved.
Drawings
Fig. 1 is a schematic diagram of the structure of an image processing apparatus of a hardware running environment according to an embodiment of the present invention;
FIG. 2 is a flowchart of a first embodiment of an image processing method according to the present invention;
FIG. 3 is a flowchart illustrating an image processing method according to another embodiment of the present invention;
fig. 4 is a flowchart of an image processing method according to another embodiment of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1, fig. 1 is a schematic structural diagram of an image processing apparatus of a hardware running environment according to an embodiment of the present invention.
The image processing device of the embodiment of the invention can be arranged on the host of the endoscope camera system. As shown in fig. 1, the image processing apparatus may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the terminal structure shown in fig. 1 does not constitute a limitation of the image processing apparatus, and may include more or less components than illustrated, or may combine certain components, or may be a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and an image processing program may be included in the memory 1005, which is one type of computer storage medium.
In the image processing apparatus shown in fig. 1, the network interface 1004 is mainly used for connecting to a background server, and performing data communication with the background server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be used to invoke an image processing program stored in the memory 1005.
In this embodiment, an image processing apparatus includes: the image processing device comprises a memory 1005, a processor 1001 and an image processing program stored in the memory 1005 and capable of running on the processor 1001, wherein the processor 1001 calls the image processing program stored in the memory 1005 and executes the steps of the image processing method in the following embodiments.
The present invention also provides an image processing method, referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the image processing method of the present invention.
In this embodiment, the image processing method is applied to a polarized light endoscope apparatus including a camera provided with a polarized light image sensor.
Specifically, the polarized light endoscope apparatus electrically includes a camera for acquiring pixel data of a target object and an image processing system for processing the pixel data, the camera and the image processing system performing signal transmission through a transmission cable.
The camera comprises an optical lens, an optical prism, a Bayer sensor, a polarized light image sensor, an image sending module and a function selecting module.
The optical lens provides a light path for white light and/or polarized light generated by the light source to the target object, the light source forms return light after being directed to the target object, the light path system simultaneously provides a light path for the return light, the return light can be guided to the optical prism, the optical prism divides the return light into two paths of light, one path of light is transmitted to the light field range of the light signal collected by the Bayer sensor, and the other path of light is transmitted to the light field range of the polarized light signal collected by the polarized light image sensor.
And the Bayer sensor is used for collecting the optical signal of one path of light beam output by the optical prism and converting the optical signal into pixel data. The Bayer sensor in this embodiment is capable of collecting three color light signals of red (R), green (G), and blue (B) simultaneously, and converting the three color light signals into RGB pixel data. The Bayer sensor in the present embodiment may be an image sensor having a resolution of 4K, such as a CCD image sensor and a CMOS image sensor. In addition, the pixel data finally output by the Bayer sensor has a resolution of 4K, and three independent monochromatic image sensors are arranged in the three-wafer endoscope, which respectively collect light signals of three colors of red (R), green (G) and blue (B), respectively convert the light signals into R pixel data, G pixel data and B pixel data, and then synthesize the R pixel data, the G pixel data and the B pixel data into pixel data.
The polarized light image sensor is used for collecting polarized light signals of the other path of light beams output by the optical prism and converting the polarized light signals into polarized information pixel data. In the present embodiment, the polarized light image sensor is capable of collecting a polarized light signal from the above-described return light and converting the polarized light signal into polarized information pixel data, that is, converting the light signal (polarized light) into an electrical signal (polarized information pixel data).
The image transmitting module is used for transmitting the pixel data and the polarization information pixel data to a transmission cable, and the transmission cable comprises transmission lines for transmitting various signal data. In this embodiment, the input end of the image transmission module is connected to the output ends of the Bayer sensor and the polarized light image sensor, and the output end of the image transmission module is connected to the transmission cable.
The function selection module is arranged in the camera and is connected with the image processor, the function selection module comprises a key assembly arranged outside the camera shell, the function selection module is used for outputting at least one function selection signal to the image processor under the triggering of the key assembly, the image processor controls the polarized light endoscope device to complete at least one corresponding function according to the function selection signal, for example, the collected and processed real-time image data can be subjected to functions such as screenshot, freezing and video recording, and each button on the key assembly triggers the polarized light endoscope device to complete the corresponding function when being pressed. The image transmission module in this embodiment transmits the pixel data and the polarization information pixel data to the image receiving module through the serial transmission data line in the transmission cable.
Specifically, the image processing method includes:
step S101, acquiring to-be-processed polarized images corresponding to a plurality of polarized angles of a polarized light image sensor based on polarized information pixel data acquired by the polarized light image sensor;
in this embodiment, the polarized light image sensor may collect a polarized light signal from the above return light, and convert the polarized light signal into polarized information pixel data, and then the image processing system obtains the polarized information pixel data collected by the polarized light image sensor, and obtains to-be-processed polarized images corresponding to a plurality of polarization angles of the polarized light image sensor based on the polarized information pixel data, specifically, extracts image data of a continuous frame number from the polarized information pixel data, where the image data of the continuous frame number includes a plurality of corresponding to-be-processed polarized images, specifically, the polarization angles include 0 °,45 °, 90 ° and 135 °, and further the to-be-processed polarized images include an image of 0 °, an image of 45 °, an image of 90 ° and an image of 135 °.
In this embodiment, the image processing system may perform preprocessing on the polarization information pixel data by using an existing image processing algorithm, for example, linear correction, dead pixel removal, white balance, gamma, automatic exposure control, and further adjust parameters such as brightness, saturation, contrast, sharpness, etc. of the output image data, and then obtain the to-be-processed polarization images corresponding to the plurality of polarization angles according to the preprocessed polarization information pixel data.
Step S102, determining an optimal polarization state image and a worst polarization state image based on the polarization image to be processed;
in this embodiment, a plurality of stokes vectors (stokes vectors) are calculated according to the to-be-processed polarized image of each polarization angle, the polarization degree image corresponding to the to-be-processed polarized image is determined according to each stokes vector, and then the optimal polarization state image and the worst polarization state image are determined according to the polarization degree image.
Step S103, determining a target clear image based on the optimal polarization state image and the worst polarization state image.
In this embodiment, when the optimal polarization state image and the worst polarization state image are obtained, the target clear image is calculated according to the optimal polarization state image and the worst polarization state image, specifically, the underwater transmittance corresponding to the polarization image to be processed is firstly obtained, then the target clear image is calculated by adopting a formula through the underwater transmittance, the optimal polarization state image and the worst polarization state image, and further the clear target polarization image is obtained, so that the function of clearly imaging the polarized light endoscope device in the underwater environment (blood, turbidity, tissue dust, mist) is realized, and when the conditions of blood, tissue dust, mist and the like occur in the operation environment, the problem of unclear output image of the polarized light endoscope device is solved through the clear processing of polarized information pixel data.
According to the image processing method, polarized images to be processed, which correspond to a plurality of polarized angles of a polarized light image sensor, are obtained through pixel data based on polarized information acquired by the polarized light image sensor; then, based on the polarization image to be processed, determining an optimal polarization state image and a worst polarization state image; and then, based on the optimal polarized state image and the worst polarized state image, a clear target image is determined, so that the function of clear imaging of the polarized light endoscope device in an underwater environment (blood, turbid, tissue dust, fog) is realized, when the conditions of blood, tissue dust, fog and the like occur in an operation environment, the problem that an output image of the polarized light endoscope device is not clear is solved by clear processing of polarized information pixel data, and the situation that the interference environment (blood, turbid water, tissue dust, fog) is cleared away to directly perform an operation can be avoided, so that the operation efficiency is improved.
Referring to fig. 3, a second embodiment of the image processing method of the present invention is proposed based on the first embodiment, in this embodiment, as shown in fig. 3, step S102 includes:
step S201, determining a total light intensity vector, a linearly polarized light component in the transverse axis direction and a linearly polarized light component in the 45-degree direction based on the polarized image to be processed;
step S202, determining the best polarization state image and the worst polarization state image based on the total light intensity vector, the linearly polarized light component in the transverse axis direction, and the linearly polarized light component in the 45 ° direction, respectively.
In this embodiment, when the polarization image to be processed of each polarization angle is obtained, a corresponding stokes vector (stokes vector) is calculated from the polarization image to be processed of each polarization angle, where the stokes vector includes four components, specifically includes a polarization image of 0 ° polarization angle, a polarization image of 45 ° polarization angle, a polarization image of 90 ° polarization angle, and a polarization image of 135 ° polarization angle, and the total light intensity vector, the linearly polarized light component in the transverse axis direction, and the linearly polarized light component in the 45 ° direction are determined from the respective components of the stokes vector.
Further, in an embodiment, the polarization image to be processed includes a polarization image with a polarization angle of 0 °, a polarization image with a polarization angle of 45 °, a polarization image with a polarization angle of 90 °, and a polarization image with a polarization angle of 135 °, and step S201 includes:
step a, determining a total light intensity vector and a transverse linear polarized light component based on a polarized image of a 0-degree polarization angle and a polarized image of a 90-degree polarization angle;
and b, determining a 45-degree direction linearly polarized light component based on the polarized image with the 45-degree polarization angle and the polarized image with the 135-degree polarization angle.
In this embodiment, the polarization angles of the polarized light image sensor include 0 °,45 °, 90 °, and 135 °, and the polarized image to be processed includes a polarized image of 0 °, a polarized image of 45 °, a polarized image of 90 °, and a polarized image of 135 °. When the polarized image to be processed is acquired, determining a total light intensity vector and a transverse straight based on the polarized image with the 0-degree polarized angle and the polarized image with the 90-degree polarized angleThe formulas of the linearly polarized light component, specifically, the total light intensity vector I and the transverse linearly polarized light component Q are respectively: i=i 0° +-I 90° ，Q＝I 0° -I 90° 。
Wherein I is the total light intensity vector, Q is the X-axis direction linear polarized light component, i.e. the transverse linear polarized light component, I 0° Three-dimensional matrix parameters of polarized image with 0 degree polarization angle, I 90° The three dimensions of the three-dimensional matrix parameters are R, G, B three dimensions respectively, and the parameters of each column in the three-dimensional matrix parameters are R, G, B values of one pixel point in the polarized image respectively.
Meanwhile, a 45 ° direction linearly polarized light component is determined based on the polarized image of the 45 ° polarization angle and the polarized image of the 135 ° polarization angle. Specifically, the formula of the 45 ° direction linearly polarized light component U is: i=i 45° +I 135° Wherein U is
A directional linear polarized light component, I 45° Three-dimensional matrix parameters of polarized image with 45 DEG polarization angle, I 135° Vector three-dimensional matrix parameters for polarized images at 135 polarization angles.
It should be noted that, the circular polarization component V may also be determined based on a polarization image at a polarization angle of 45 ° and a polarization image at a polarization angle of 135 °, where the formula of the circular polarization component V is: v=i 45° -I 135° 。
Then, an optimal polarization state image and a worst polarization state image are respectively determined based on the total light intensity vector, the horizontal axis direction linear polarized light component, and the 45 ° direction linear polarized light component.
When the polarization degree image is obtained, a total light intensity vector, a transverse axis direction linear polarized light component and a 45 degree direction linear polarized light component corresponding to the polarization image to be processed are obtained, wherein the obtaining manners of the total light intensity vector, the transverse axis direction linear polarized light component and the 45 degree direction linear polarized light component are similar to those of the previous embodiment, and are not repeated herein. And calculating an optimal polarized image Imax and a worst polarized image Imin according to the total light intensity vector, the linearly polarized light component in the transverse axis direction and the linearly polarized light component in the 45-degree direction.
The formulas of the optimal polarization state image Imax and the worst polarization state image Imin are respectively:
wherein Imax is the best polarization state image, imin is the worst polarization state image, I is the total light intensity vector, Q is the transverse linear polarized light component, and U is the 45 ° direction linear polarized light component.
According to the embodiment, the total light intensity vector, the transverse linear polarized light component and the 45-degree linear polarized light component can be accurately obtained according to the Stokes vector, and then the optimal polarized state image and the worst polarized state image can be accurately obtained according to the Stokes vector, so that the accuracy of a clear target image is improved, and the polarized light endoscope device can clearly image in underwater environments (blood, turbidity, tissue dust and fog).
According to the image processing method, based on the polarized image to be processed, a total light intensity vector, a linearly polarized light component in the transverse axis direction and a linearly polarized light component in the 45-degree direction are determined; and then, based on the total light intensity vector, the linearly polarized light component in the transverse axis direction and the linearly polarized light component in the 45-degree direction, respectively determining the optimal polarized state image and the worst polarized state image, and accurately obtaining the optimal polarized state image and the worst polarized state image according to the Stokes vector, so as to improve the definition of the target clear image, thereby being convenient for realizing clear imaging of the polarized light endoscope device in an underwater environment (blood, turbidity, tissue dust and mist), avoiding cleaning the interference environment (blood, turbidity, tissue dust and mist) and directly performing operation, and further improving the operation efficiency.
Referring to fig. 3, based on the first embodiment, a third embodiment of the image processing method of the present invention is proposed, in which, as shown in fig. 3, step S103 includes:
step S301, obtaining a polarization degree image corresponding to the polarization image to be processed and a pixel value of the brightest pixel point in the worst polarization state image;
step S302, determining the target sharp image based on the polarization degree image, the pixel value, the optimal polarization state image and the worst polarization state image.
In this embodiment, when the best polarization state image and the worst polarization state image are obtained, the pixel value of the brightest pixel point in the worst polarization state image is obtained, specifically, the luminance value of each pixel point in the worst polarization state image is calculated by adopting the existing luminance value algorithm according to the pixel value (R/G/B value) of each pixel point in the worst polarization state image, and then the brightest pixel point, which is the pixel point with the largest luminance value in the worst polarization state image, is obtained by comparing the luminance values, and the pixel value of the brightest pixel point in the worst polarization state image is obtained.
Meanwhile, a polarization degree image corresponding to the polarization image to be processed is obtained, specifically, a total light intensity vector, a transverse axis direction linear polarized light component and a 45 degree direction linear polarized light component corresponding to the polarization image to be processed are firstly obtained, and then the polarization degree image is determined, wherein the obtaining mode of the total light intensity vector, the transverse axis direction linear polarized light component and the 45 degree direction linear polarized light component is referred to the second embodiment, and is not repeated herein. Further, in an embodiment, the step S301 includes:
and determining a polarization degree image corresponding to the polarization image to be processed based on the total light intensity vector, the transverse axis direction linear polarized light component and the 45-degree direction linear polarized light component corresponding to the polarization image to be processed.
The polarization degree image is the current underwater polarization degree image, and specifically, the formula of the polarization degree image P is as follows:
wherein P is a polarization degree image, I is a total light intensity vector, Q is a transverse linear polarized light component, and U is a 45-degree linear polarized light component.
In this embodiment, stokes vectors I, Q and U are obtained through the polarized image to be processed, and then the polarized image is calculated through I, Q and U, so that the polarized image corresponding to the polarized image to be processed can be accurately obtained, and the accuracy of the optimal polarized image and the worst polarized image is further improved, so that the polarized light endoscope device can clearly image in underwater environments (blood, turbidity, tissue dust and fog).
It should be noted that, the circular polarization component V may also be calculated from a polarization image at a polarization angle of 45 ° and a polarization image at a polarization angle of 135 °, and the polarization degree image P may be calculated from the total light intensity vector I, the transverse linear polarization component Q, the 45 ° direction linear polarization component U, and the circular polarization component V, where the formula of the polarization degree image P is:
because the polarization effect of the underwater target object and the background light incident is extremely small, the circular polarization component can be set to V=0, and further, the polarization degree image can be calculated only according to I, Q and U, so that the image processing efficiency is improved.
Then, determining a target clear image based on the polarization degree image, the pixel value, the optimal polarization state image and the worst polarization state image, wherein the formula of the target clear image is as follows:
wherein H is OBJ For a clear target image, P is a polarization degree image, A ∞ Is the pixel value of the brightest pixel point in the worst polarization state image,imax is the best polarization image and Imin is the worst polarization image.
According to the image processing method, the pixel value of the brightest pixel point in the worst polarized state image and the polarized degree image corresponding to the polarized image to be processed are obtained; and then determining the target clear image based on the polarization degree image, the pixel value, the optimal polarization state image and the worst polarization state image, and accurately obtaining the clear target polarization image through the polarization degree image, the optimal polarization state image and the worst polarization state image, so that the function of clearly imaging the polarized light endoscope device in an underwater environment (blood, turbidity, tissue dust, mist) is realized, and when the conditions of blood, tissue dust, mist and the like occur in an operation environment, the problem of unclear output image of the polarized light endoscope device is solved through the clear processing of polarized information pixel data, and the operation efficiency is further improved.
Referring to fig. 3, a fourth embodiment of the image processing method of the present invention is proposed based on the first embodiment, in which, as shown in fig. 3, step S101 includes:
step S401, acquiring Bayer format polarized images corresponding to a plurality of polarization angles based on polarized information pixel data acquired by a polarized light image sensor;
step S402, performing color interpolation processing on the Bayer format polarized image to obtain the RGB format polarized image to be processed.
In this embodiment, the polarized light image sensor is a Bayer format image sensor, and when polarization information pixel data acquired by the polarized light image sensor is acquired, bayer format polarized images corresponding to a plurality of polarization angles, that is, a Bayer format image of 0 ° polarization angle, a Bayer format image of 45 ° polarization angle, a Bayer format image of 90 ° polarization angle, and a Bayer format image of 135 ° polarization angle are acquired based on the polarization information pixel data.
Then, performing color interpolation processing on the Bayer format polarized image to obtain the to-be-processed polarized image in an RGB format, namely performing color interpolation processing on the Bayer format image with the polarization angle of 0 degree to obtain the polarized image with the polarization angle of 0 degree, performing color interpolation processing on the Bayer format image with the polarization angle of 45 degrees to obtain the polarized image with the polarization angle of 45 degrees, performing color interpolation processing on the Bayer format image with the polarization angle of 90 degrees to obtain the polarized image with the polarization angle of 90 degrees, and performing color interpolation processing on the Bayer format image with the polarization angle of 135 degrees to obtain the polarized image with the polarization angle of 135 degrees, so that the to-be-processed polarized image in the RGB format can be accurately obtained.
In this embodiment, the image processing system may pre-process the polarization information pixel data through an existing image processing algorithm, for example, linear correction, dead pixel removal, white balance, gamma, automatic exposure control, and adjust parameters such as brightness, saturation, contrast, sharpness, etc. of the output image data, and then obtain a Bayer format polarization image corresponding to a plurality of polarization angles according to the pre-processed polarization information pixel data.
According to the image processing method, polarization images in a Bayer format corresponding to a plurality of polarization angles are obtained based on polarization information pixel data acquired by a polarized light image sensor; and then performing color interpolation processing on the Bayer format polarized image to obtain the RGB format polarized image to be processed, and obtaining the RGB format polarized image to be processed according to the polarized information pixel data so as to facilitate the subsequent processing of the polarized image to be processed, improve the calculation efficiency of the target clear image and further improve the operation efficiency.
Referring to fig. 4, based on the foregoing embodiments, a fifth embodiment of the image processing method of the present invention is provided, as shown in fig. 4, in this embodiment, the image processing method further includes:
step S501, acquiring a Bayer image based on pixel data acquired by the Bayer sensor;
step S502, preprocessing the Bayer image to obtain an RGB image;
step S503, performing image conversion on the RGB image to obtain a YUV image.
In this embodiment, a Bayer image is acquired based on the pixel data acquired by the Bayer sensor, and the image processing system performs preprocessing on the pixel data acquired by the Bayer sensor by using an existing image processing algorithm, such as linear correction, dead pixel removal, white balance, gamma, automatic exposure control, and also adjusts parameters such as brightness, saturation, contrast, sharpness, and the like of the output image data, and then obtains the Bayer image according to the preprocessed pixel data acquired by the Bayer sensor.
And then preprocessing the Bayer image to obtain an RGB image, specifically, performing color interpolation processing on the Bayer image to obtain a first RGB image, performing CCM (Color Correction Matrix, color matrix) color correction on the first RGB image to obtain a second RGB image closest to the true color of the operation scene, and performing Gamma correction on the second RGB image to obtain an RGB image so as to enable the RGB image to be more in line with a human eye vision system.
Then, performing image conversion on the RGB image to obtain a YUV image, namely converting the RGB image in the RGB domain into the YUV image in the YUV domain, and adopting wide dynamic processing to solve the problem that the operation scene image has overexposure in a brightness area and underexposure in a dark area; in order to eliminate the loss of image details in the noise reduction process, the image needs to be sharpened, and the detail information of the image is restored and enhanced. After treatment, the traditional endoscope imaging (any resolution) is realized, the operation scene is clearly restored, and the operation efficiency is further improved. Wherein YUV is divided into three components, "Y" represents brightness (luminence or Luma), that is, gray scale value; "U" and "V" denote Chroma (Chroma) to describe the image color and saturation for the color of the given pixel.
Further, in an embodiment, the polarized light endoscope device further includes a first display and a second display, and an optical prism is disposed in the camera, and the optical prism is used for dividing one path of light beam output by the optical lens into two paths of light beams; the Bayer sensor is used for collecting the optical signal of one path of light beam output by the optical prism and converting the optical signal into the pixel data; the polarized light image sensor is used for collecting polarized light signals of another path of light beams output by the optical prism and converting the polarized light signals into polarized information pixel data; the image processing method further includes: and displaying the YUV image through the first display and displaying the clear target image through the second display. And further, the simultaneous display of the clear target image and the YUV image is realized, and the operation efficiency is further improved.
According to the image processing method provided by the embodiment, a Bayer image is obtained based on pixel data acquired by the Bayer sensor; preprocessing the Bayer image to obtain an RGB image; and then performing image conversion on the RGB image to obtain a YUV image, and further restoring a surgical scene through the YUV image and a target clear image, so that the polarized light endoscope device can clearly image in an underwater environment (blood, turbidity, tissue dust and fog), and further improving the surgical efficiency.
The invention also provides a computer readable storage medium.
The computer-readable storage medium of the present invention stores thereon an image processing program which, when executed by a processor, implements the steps of the image processing method as described above.
The method implemented when the image processing program running on the processor is executed may refer to various embodiments of the image processing method of the present invention, which are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (10)
1. An image processing method is characterized by being applied to a polarized light endoscope device comprising a camera, wherein the camera is provided with a polarized light image sensor; the image processing method includes the steps of:
acquiring to-be-processed polarized images corresponding to a plurality of polarization angles of a polarized light image sensor based on polarized information pixel data acquired by the polarized light image sensor;
determining an optimal polarization state image and a worst polarization state image based on the polarization image to be processed;
a target sharp image is determined based on the best polarization state image and the worst polarization state image.
2. The image processing method according to claim 1, wherein the step of determining the best polarization state image and the worst polarization state image based on the polarization image to be processed includes:
based on the polarized image to be processedDetermining the total light intensity vector, the transverse axis direction linear polarized light component and 45 ° A directional linearly polarized light component;
based on the total light intensity vector, the transverse axis direction linearly polarized light component, and the 45 ° And (3) determining the optimal polarized state image and the worst polarized state image respectively by linearly polarized light components.
3. The image processing method according to claim 2, wherein the polarization image to be processed includes a polarization image of 0 ° polarization angle, a polarization image of 45 ° polarization angle, a polarization image of 90 ° polarization angle, and a polarization image of 135 ° polarization angle;
the total light intensity vector, the transverse axis direction linear polarized light component and 45 are determined based on the polarized image to be processed ° The step of directing the linearly polarized light component comprises:
determining a total light intensity vector and a transverse linearly polarized light component based on the polarized image at the 0 ° polarization angle and the polarized image at the 90 ° polarization angle;
determining 45 based on the polarized image of the 45 ° polarization angle and the polarized image of the 135 ° polarization angle ° The direction is linearly polarized light component.
4. The image processing method of claim 1, wherein the step of determining the target sharp image based on the best polarization state image and the worst polarization state image comprises:
acquiring a polarization degree image corresponding to the polarization image to be processed and a pixel value of the brightest pixel point in the worst polarization state image;
the target sharp image is determined based on the polarization degree image, the pixel value, the optimal polarization state image, and the worst polarization state image.
5. The image processing method according to claim 4, wherein the step of acquiring the polarization degree image corresponding to the polarization image to be processed includes:
based on the total light intensity vector, the horizontal axis direction linear polarized light component and 45 corresponding to the polarized image to be processed ° And determining a polarization degree image corresponding to the polarization image to be processed by linearly polarizing the light component in the direction.
6. The image processing method according to claim 1, wherein the step of acquiring the polarization image to be processed corresponding to the plurality of polarization angles of the polarized light image sensor based on the polarization information pixel data acquired by the polarized light image sensor includes:
acquiring a Bayer format polarized image corresponding to a plurality of polarization angles based on the polarized information pixel data acquired by the polarized light image sensor;
and performing color interpolation processing on the Bayer format polarized image to obtain the polarized image to be processed in an RGB format.
7. The image processing method according to any one of claims 1 to 6, wherein the camera is provided with a Bayer sensor; the image processing method further includes:
acquiring a Bayer image based on pixel data acquired by the Bayer sensor;
preprocessing the Bayer image to obtain an RGB image;
and performing image conversion on the RGB image to obtain a YUV image.
8. The image processing method according to claim 7, wherein the polarized light endoscope device further comprises a first display and a second display, and an optical prism is arranged in the camera and is used for dividing one light beam output by the optical lens into two light beams; the Bayer sensor is used for collecting the optical signal of one path of light beam output by the optical prism and converting the optical signal into the pixel data; the polarized light image sensor is used for collecting polarized light signals of another path of light beams output by the optical prism and converting the polarized light signals into polarized information pixel data; the image processing method further includes:
and displaying the YUV image through the first display and displaying the clear target image through the second display.
9. An image processing apparatus, characterized in that the image processing apparatus comprises: memory, a processor and an image processing program stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 8.
10. A computer-readable storage medium, on which an image processing program is stored, which when executed by a processor implements the steps of the image processing method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310323697.5A CN116402711A (en) | 2023-03-29 | 2023-03-29 | Image processing method, device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310323697.5A CN116402711A (en) | 2023-03-29 | 2023-03-29 | Image processing method, device and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116402711A true CN116402711A (en) | 2023-07-07 |
Family
ID=87015417
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310323697.5A Pending CN116402711A (en) | 2023-03-29 | 2023-03-29 | Image processing method, device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116402711A (en) |
-
2023
- 2023-03-29 CN CN202310323697.5A patent/CN116402711A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9154745B2 (en) | Endscope apparatus and program | |
| US9486123B2 (en) | Endoscope system which enlarges an area of a captured image, and method for operating endoscope system | |
| JP5449816B2 (en) | Image processing apparatus, image processing program, and method of operating image processing apparatus | |
| US20180008134A1 (en) | Endoscope system | |
| US20090213211A1 (en) | Method and Device for Reducing the Fixed Pattern Noise of a Digital Image | |
| WO2023005374A1 (en) | Polarized light endoscope apparatus and data processing method | |
| US10163196B2 (en) | Image processing device and imaging system | |
| JP6329715B1 (en) | Endoscope system and endoscope | |
| US10943333B2 (en) | Method and apparatus of sharpening of gastrointestinal images based on depth information | |
| JP6140100B2 (en) | Endoscope apparatus, image processing apparatus, and operation method of endoscope apparatus | |
| US7534205B2 (en) | Methods and apparatuses for selecting and displaying an image with the best focus | |
| US11354783B2 (en) | Method and apparatus of sharpening of gastrointestinal images based on depth information | |
| JP2012205619A (en) | Image processor, control device, endoscope apparatus, image processing method, and image processing program | |
| US9323978B2 (en) | Image processing device, endoscope apparatus, and image processing method | |
| JP7179837B2 (en) | Endoscope device, endoscope image display method, and operation method of endoscope device | |
| JP2016073572A (en) | Image processing apparatus for electronic endoscope, electronic endoscope system, and image processing method for electronic endoscope | |
| JP2017170157A (en) | Endoscope apparatus and image processing apparatus and method for operating endoscope apparatus | |
| CN116402711A (en) | Image processing method, device and computer readable storage medium | |
| JP6352673B2 (en) | Endoscope apparatus and operation method of endoscope apparatus | |
| CN113744266B (en) | Method and device for displaying focus detection frame, electronic equipment and storage medium | |
| CN114027765B (en) | Fluorescence endoscope system, control method, and storage medium | |
| KR102212964B1 (en) | Deblurring image processing method and apparatus | |
| JP3126996B2 (en) | Endoscope image processing method | |
| CN219940557U (en) | Polarized light endoscope device and endoscope imaging system | |
| US20150003700A1 (en) | Image processing device, endoscope apparatus, and image processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |