CN109389569A - Based on the real-time defogging method of monitor video for improving DehazeNet - Google Patents
Based on the real-time defogging method of monitor video for improving DehazeNet Download PDFInfo
- Publication number
- CN109389569A CN109389569A CN201811261910.XA CN201811261910A CN109389569A CN 109389569 A CN109389569 A CN 109389569A CN 201811261910 A CN201811261910 A CN 201811261910A CN 109389569 A CN109389569 A CN 109389569A
- Authority
- CN
- China
- Prior art keywords
- image
- layer
- video
- defogging
- dehazenet
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000013528 artificial neural network Methods 0.000 claims abstract description 21
- 238000009826 distribution Methods 0.000 claims abstract description 11
- 239000003595 mist Substances 0.000 claims description 25
- 230000006870 function Effects 0.000 claims description 24
- 238000002834 transmittance Methods 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 9
- 238000000605 extraction Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 230000002146 bilateral effect Effects 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 3
- 238000003384 imaging method Methods 0.000 claims description 3
- 235000013399 edible fruits Nutrition 0.000 claims description 2
- 230000015572 biosynthetic process Effects 0.000 claims 1
- 230000008030 elimination Effects 0.000 claims 1
- 238000003379 elimination reaction Methods 0.000 claims 1
- 238000003786 synthesis reaction Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 21
- 238000012549 training Methods 0.000 description 18
- 238000013461 design Methods 0.000 description 5
- 230000006872 improvement Effects 0.000 description 5
- 230000000149 penetrating effect Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 230000001537 neural effect Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 241001269238 Data Species 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012880 independent component analysis Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000035772 mutation Effects 0.000 description 2
- 210000004218 nerve net Anatomy 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- HPTJABJPZMULFH-UHFFFAOYSA-N 12-[(Cyclohexylcarbamoyl)amino]dodecanoic acid Chemical compound OC(=O)CCCCCCCCCCCNC(=O)NC1CCCCC1 HPTJABJPZMULFH-UHFFFAOYSA-N 0.000 description 1
- 241000512668 Eunectes Species 0.000 description 1
- 238000000342 Monte Carlo simulation Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 235000019504 cigarettes Nutrition 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000004132 cross linking Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013479 data entry Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000005284 excitation Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012886 linear function Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 210000005036 nerve Anatomy 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000004445 quantitative analysis Methods 0.000 description 1
- 230000005855 radiation Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30232—Surveillance
Abstract
A kind of based on the monitor video real-time defogging method for improving DehazeNet, step includes: 1) to acquire video by video capture device, then video is cut into single picture frame by frame, is supplied to neural network and is handled；2) trained improved DehazeNet neural network is utilized, known to i.e. each layer weight, the transmitance t (x) and atmosphere light constant that each section is obtained for the picture piecemeal processing of input, finally constitute transmitance distribution map and atmosphere light constant distribution map；3) output for obtaining neural network solves fogless figure according to based on atmospherical scattering model defogging algorithm, then fogless figure is spliced into video again.Compared to traditional single image defogging, the present invention realizes the real-time defogging of video, while ensure that the effect of defogging, solves supersaturation existing for traditional defogging method and skyline fuzzy problem.
Description
Technical field
The invention belongs to computer technology video processing in application technology, it is specifically a kind of based on improve DehazeNet
The real-time defogging method of monitor video.
Background technique
Mist is a kind of common meteor.Ubiquitous dust, cigarette or other particles, can all reduce atmosphere in air
Clarity.Since light is by KPT Scatter in atmosphere, so that object contrast in visual effect reduces when imaging.Therefore, mist
Many problems would generally be brought to photographic imagery.The presence of mist substantially changes atmospheric transmittance, pair of outdoor scene image
It all can therefore be varied so that many features contained in image are capped or thicken than degree and color, video is caused to be supervised
Control product cannot collect clearly live image, seriously affect to the safety precaution generation of important place in city.
The transmissivity of mist is often related with the depth of field in image, and object range image acquisition device is remoter, and transmissivity is lower.
There are many defogging methods for being directed to single image, image restoration and two major class of image enhancement can be divided into.In addition to respectively
According to histogram, the method for contrast and saturation degree, multiple image and image depth under Same Scene difference atmospheric conditions are utilized
The method of information defogging has also been suggested.But in practical applications, several of the depth information of picture and stringent Same Scene
Mist figure is not easy acquisition.
It there has been more reasonably hypothesis and priori knowledge in recent years, the defog effect of single image has length on this basis
The progress of foot, but some problems are still had in picture quality and rapidity, such as:
1. precision is lower.It is higher according to the local contrast ratio image containing mist in fog free images it is assumed that be based on Ma Erke
The maximized defogging method of local contrast of husband's random field (MRF) is although can be realized good defog effect, simultaneously
Easily lead to the supersaturation of image.
2. speed is slower.Independent component analysis method (ICA) based on minimum input is used for defogging and works, but this side
The processing time of method is very long and is consequently not used for handling the picture of mistiness.
Summary of the invention
To solve the problems, such as that precision present in traditional single image algorithm is lower, slow, in conjunction with convolutional Neural net
Network and parallel computation structure have carried out collectivity Scheme Design to new video defogging algorithm.The invention proposes based on convolution mind
Improved defogging network DehazeNet through network effectively increases image using atmospherical scattering model and every priori knowledge
Defogging precision；The design structure and operand for analyzing each section in original DehazeNet are guaranteeing precision and defog effect
Under the premise of network is adjusted and is optimized, thus reduce restore image the processing time；Meanwhile it is pixel recovery process is parallel
Change, accelerate foggy image treatment process, to realize high definition real-time video dehazing function, and has and work well, is adaptive
The features such as answering defogging.
The specific technical solution of the present invention is described as follows:
A kind of real-time defogging method of monitor video based on improvement DehazeNet, step include:
1) video is acquired by video capture device, then video is cut into single picture frame by frame, is supplied to nerve net
Network is handled；
2) using trained neural network, i.e., each layer weight is it is known that the picture piecemeal for input is handled to obtain
The transmitance and atmosphere light constant for taking each section, finally constitute transmitance distribution map and atmosphere light constant distribution map；
3) output for obtaining neural network solves fogless figure according to atmospherical scattering model, then fogless figure is spliced into again
Video.
Specifically:
(1) defogging theory of algorithm model
1, atmospherical scattering model
In order to describe the composition of mist figure, atmospherical scattering model is proposed in the prior art, this is the basic model of image defogging,
This model then has multiple improvement again.
Atmospherical scattering model is stated with following equation (1):
I (x)=J (x) t (x)+α (1-t (x)), (1)
Wherein, I (x) is the mist image primitive observed, J (x) is the true picture pixel after restoring, and t (x) is atmospheric transmission
Rate, α are global atmosphere light constants.In formula (1), there are three unknown parameter J (x), t (x), α, are estimating t (x) and α
Afterwards, real scene figure J (x) can be resumed to obtain.
Atmospheric transmissivity t (x) is used to describe not scattered and reached institute's accounting of the light of camera, fixed by formula (2)
Justice:
T (x)=e-βd(x), (2)
Wherein, d (x) is the corresponding scene point of the pixel to the distance of camera, the as depth of field, and β is atmospheric scattering coefficient,
Show that t (x) tends to 0 when d (x) tends to be infinite.Comprehensive (1) (2) can obtain
α=I (x), d (x) → ∞, (3)
In actual imaging process, depth of field d (x) can not tend to be infinite, but one can be defined in remote situation
The transmissivity t of very little0.In this case, no longer using the method for atmosphere light is obtained in formula (3), according to the following formula (4)
Method estimation atmosphere light can be more acurrate:
Based on discussed above, it can be deduced that conclusion: accurately estimating atmospheric transmissivity is the key that recover clear image.
2, defogging priori knowledge
Based on empiric observation, existing method proposes a variety of hypothesis for calculating feature relevant to mist or priori is known
Know.The defogging of image can use these relevant features and realize.
1. dark channel prior:
Dark channel prior is obtained based on the outdoor fog free images of widely observation.In most fog free images block,
At least one Color Channel has the low-down block of pixels of some intensity values, or even close to 0.Dark is defined as the image block
The minimum value of middle all pixels:
Wherein, IcIt is the RGB color channel of I, Ωr(x) refer to the image block of r × r size centered on x.Dark channel prior
Feature and the concentration of mist in image are closely connected, and can be directly used in estimation atmospheric transmissivity t (x) ∝ 1-D (x).
2. maximum-contrast:
According to atmospherical scattering model, mist influences that the contrast of image can be reduced caused by transmissivity:
Based on this observation as a result, using local contrast (in the image block Ω of s × s sizes(x) each picture in
Difference between element) and in r × r size area ΩrInterior local contrast maximizes, and carries out defogging.It is defined as follows:
Wherein | Ωs(y) | it is the radix of local neighborhood, the relationship between contrast metric and transmissivity t is apparent
, so local contrast defined in formula maximizes the visual effect that can enhance image.
3. color decays:
The saturation degree I of image blocks(x) it will receive the influence of mist to fall sharply, and brightness value I simultaneouslyv(x) it can obviously increase,
Difference between the two is caused to increase.According to color decaying priori, the difference between brightness and saturation degree can be used for estimating
The concentration of mist:
A (x)=Iv(x)-Is(x), (8)
Wherein Iv(x) and Is(x) it can be expressed as in hsv color space
Iv(x)=maxC ∈ { r, g, b }Ic(x) (9)
Color decay characteristics are directly proportional to depth of field d (x), it is easy to be used for transmissivity estimation.
4. tone difference:
Original image I (x) and corresponding half anti-figure Isi(x) the tone difference between can be used for the mist in detection image.Half is anti-
Figure is defined as follows:
Isi(x)=maxC ∈ { r, g, b }[Ic(x), 1-Ic(x)], (11)
For fog free images, the pixel value in three channels in half anti-figure will not be inverted all, lead to Isi(x) and I (x) it
Between generate huge tone difference, this tone difference is defined as:
Wherein footmark ' h ' represents the luminance channel in hsv color space.According to formula (12), atmospheric transmissivity t (x) is anti-
To having traveled to H (x).
(2) DehazeNet Crosslinking Structural
Existing DehazeNet network is made of convolutional layer and pond layer, and in network finally using bilateral linear
Function is rectified as activation primitive.According to the general structure of neural network, network can be divided into feature extraction, multiple dimensioned reflect
It penetrates, four major parts of local extremum and non-linear convergence.The overall structure of network is as shown in Figure 1.
1. feature extraction:
According to above-mentioned defogging priori knowledge, feature relevant to mist includes that the dark of picture, tone are different in image
Cause property, maximum-contrast and color dough softening etc..The extraction of picture feature be substantially using filter appropriate and picture into
Row convolution operation, while being usually associated with Nonlinear Mapping.Therefore the first part of network by convolutional layer conv1 and reshapes layer
Reshape1 is constituted.Wherein convolutional layer for realizing filter function and to reshape layer then be in order to which the pond layer to after mentions
For suitable data entry modality.The present invention is increasing pond layer pool1 for further feature extraction later.
RGB mist figure (each pixel of colored mist figure is all indicated with the different proportion of three kinds of colors of RGB RGB)
Effective information is not often evenly distributed in tri- channels R, G, B, because the MAX function of pond layer is utilized in the invention,
Purpose mainly more fully extracts the feature of image and reasonably simplifies data.While in order to which real-time Penetrating Fog institute is better achieved
It is required that rapidity target, the design of pond layer of the invention be not single step sliding, but once sliding pond layer itself greatly
Small step-length is with the feature of more promptly extraction image.Further, in order to reach the processing requirement of next part, integral net
Network is provided with one layer again after the layer of pond and reshapes a layer reshape2.
2. multiple dimensioned mapping:
The Feature Mapping of the different scale of existing DehazeNet network has played important work in defogging treatment process
With.Meanwhile the mapping of different scale facilitates further compressed data structure, reduces the computational burden of network, reaches rapidity
Requirement.Second part main body in order to realize multiple dimensioned mapping, the invention proposes parallel convolutional coding structure as neural network.
According to the universal experience of image procossing, the convolution kernel size that the present invention selects is 1 × 1,3 × 3,5 × 5 and 7 × 7.Pass through four
A convolutional layer conv2/1 × 1 of different sizes, conv2/3 × 3, conv2/5 × 5, conv2/7 × 7, four scales in image
Corresponding feature is extracted.Splice layer conv2/output by a convolution later to realize between four input datas
Splicing.Splicing layer input dimension has used default setting, i.e., is spliced with all channel datas.
3. local extremum:
In the evaluation criterion of image restoration, space-invariance is very important a Xiang Zhibiao.Space-invariance feature
It can use a series of pondization operation to realize.In convolutional neural networks, local extremum is then for overcoming local sensitivity to make
At the inconsistent classical way in space.Simultaneously as the target of image acquisition equipment is that outdoor highway monitoring and security protection are supervised
Control, the transmitance of air dielectric should not generate huge mutation in a small range, and the operation of local extremum can be eliminated effectively
White noise that may be present.Therefore the overall network Part III pond layer pool2 that is set as MAX by one is constituted, it and the
The distinguishing pond layer of a part is often caused by single pixel in view of local sensitivity, therefore, pond layer setting herein
Sliding step be 1, to preferably keep the space-invariance of entire image.
4. non-linear convergence:
The activation primitive that the deep learning of earliest period uses is S type activation primitive, such as following formula (13)
Later in order to improve convergence rate, it is contemplated that gradient attenuation problem existing for S type activation primitive, most nerve
Start when network training using line rectification unit as activation primitive, such as following formula (14).
But the problem of line rectification function be its output value be unlike S type activation primitive from 0 to 1,
But from 0 to infinity.Such output obviously can be met with transmitance beyond the 1 physics fact.Therefore in order to make most
Output is between 0 and 1 eventually, and the present invention has selected bilateral line rectification unit as activation primitive, such as following formula (15).
In caffe frame (Convolutional Architecture for Fast Feature Embedding, volume
Product neural network framework) under, traditional excitation layer include line rectification unit (ReLU), S type function (Sigmoid), hyperbolic just
Cut function (TanH), ABS function (AbsVal), weighting function (Power) and binomial natural logrithm likelihood score function
(BNLL).Among these and do not include bilateral line rectification unit, therefore needs to utilize full articulamentum and linear whole in caffe frame
Stream unit matching is just able to achieve the function of bilateral line rectification, the transmitance that final output invention needs.
(3) image restoration accelerates
In order to realize real-time defogging, accelerate image processing speed, since Python can call kernel function to realize GPU figure
Processor concurrent operation, the present invention call library function to carry out multi-threaded parallel operation using Python, and from following two side
Face, which is realized, to be accelerated:
(1) estimate global atmosphere light constant a.During obtaining atmospheric parameter, tentatively improves and atmosphere light can not be trained
After the DehazeNet network of constant obtains transmissivity distribution, need to solve global atmosphere light constant α.The single block of the present invention
Merger by Sharing Memory Realization transmittance minimum solves, and intermediate result is stored in shared drive, reduces to main memory
Access, and then realize optimization；And it is same for the most value between row and row realized using single block, asked by merger
The most value of all rows is obtained, finally using the intensity value of the smallest point of transmittance values as global atmosphere light constant α.
(2) image restoration.For the Multitask network of transmitance Yu atmosphere light constant can be trained after improving simultaneously, this
Corresponding J (x) is calculated by parallel computation with peak efficiency in invention.Clear image J (x) can be according to t (x) and α by following public affairs
Formula is restored:
When realizing parallel computation on GPU, parallel thread number is pixel number, and per thread can calculate pixel
The corresponding clear image J (x) of x realizes parallel accelerate.
(4) evaluation criterion determines
1, neural network convergence
Mean square error (Mean Square Error, abbreviation MSE) is to evaluate the performance function of network.For example, there is n to defeated
Enter output data, network is denoted as y by the output after trainingi.The statistical parameter is that prediction data and initial data corresponding points are missed
The mean value of the quadratic sum of difference, calculation formula is,
Wherein n is the number of sample, yiIt is truthful data,It is the data of fitting, wi> 0.By the definition of MSE it is found that its
Value illustrates that model selection and fitting are better closer to 0, and data estimation is relatively more successful, and the result of network output has more
Convincingness.
2, image restoration effect assessment
Comentropy: the richness of color can be quantified with probability.If the probability that each pixel value occurs in picture
It is all not zero, that is, may be regarded as the gorgeous picture of color.The final quantized result of this probability, exactly comentropy are fitted
Value.Entropy is bigger, and mist figure treatment effect is better.
Average gradient: nearby gray scale has notable difference on the boundary or hachure two sides of image, i.e. rate of gray level is big.This change
The size of rate can be used to indicate image definition.It reflects the rate of image minor detail contrast variation, characterizes image
Opposite readability.
Detailed description of the invention
Fig. 1 is DehazeNet network structure；
Fig. 2 is the flow chart of this defogging method；
Fig. 3 is the improvement DehazeNet structure chart of this defogging method；
Fig. 4 is that have mist tile data collection schematic diagram；
Fig. 5-1 is certain city's haze image (not using present invention processing)；
Fig. 5-2 is certain city's haze image (using present invention processing).
Specific embodiment
This case is further illustrated with specific embodiment with reference to the accompanying drawing:
(1) defogging method master-plan
Based on atmospherical scattering model, video defogging is exactly that transmitance and atmosphere light constant are estimated in the case where known mist figure
To solve original image.Entire defogging process in real time is divided into three parts (the general frame is as shown in Figure 2) by the present invention:
First part is to acquire video by video capture device, then video is cut into single picture frame by frame, provided
It is handled to neural network.
Second part is using trained neural network, i.e., each layer weight is it is known that picture piecemeal for input
It handles to obtain the transmitance and atmosphere light constant of each section, finally constitutes transmitance distribution map and atmosphere light constant distribution map.
Part III is the output for obtaining neural network, solves fogless figure according to atmospherical scattering model, then by fogless figure weight
Newly it is spliced into video.
(2) using the hardware design and environmental structure of the defogging system of this method
1, image acquisition equipment calls
Using the defogging system of this method, select web camera as image capture device.
Traditional analog video camera generally uses CMOS as sensor, and resolution ratio and illumination are relatively low, is suitable for one
The videophone and conference telephone not high to image quality requirements a bit；And analog video camera can only carry out unidirectional signal biography
Defeated, transmission is vision signal, and needing to connect hard disk video recorder either monitor when checking could read.In contrast, net
Network video camera increases compression and handles the module of video, while having the function of video camera and video server.Therefore, network
Video camera, which only needs an IP network interface, can be carried out the processing of data, meet the requirement monitored in real time.
In view of needing to be handled in real time video, web camera can not be passed by video recorder, directly by video
It is defeated on computer, speed is faster；Meanwhile it can be by using the standard agreement RTSP on-link mode (OLM) of video camera for third party device
It carries out taking stream, the code stream message on real-time reception network is decoded into video.
During realizing the installation of the end PC with routine call, camera is connected by the present invention with cable with computer.Make
Used time first looks at the IP address of the machine computer and the IP address of IP Camera whether in same network segment, changes net as one sees fit
The IP address of network camera, subnet mask, default gateway allow the machine computer to identify IP Camera, on computers may be used
The standard successfully configured using real-time display monitor video as camera.The present invention accesses IP address by Opencv library function,
Obtain the data flow of IP Camera transmission.An IP address is set by IP Camera, the ground is accessed by Opencv
Location, so that it may realize and call camera indirectly, the data acquired are converted to image and show in computer end.Draw in python
Enter cv2 function library, by calling the VideoCapture function of Opencv, obtains the real-time view transmitted from IP Camera
Frequency evidence is handled and is shown in real time to video image.
2, host computer running environment configures
The hard-wired master system of this system is ubuntu16.04, memory headroom 7.7GB, video memory 4GB.On
The Intel i7 core that position machine central processing unit is 2.80GHz, the graphics processing unit used is then Geforce GTX
1050Ti.Present invention uses the classical Integrated Development Environment Pycharm Community Editions of Python.In order to more easily carry out square
Battle array arithmetic operation, the interpreter of Pycharm are set as Anaconda3, to be easy to use the numpy mould carried in Anaconda
Block.In order to realize a series of processing for video and image, the present invention has invoked the function library in OpenCV3.4.0, accordingly
Ground introduces cv2 module in Python.In order to be trained and examine to neural network, the present invention has been compiled containing CUDA9.0
The caffe module of interface and cuDNN network function, and basic calling is completed in Python.
(3) DehazeNet building is improved
1, network structure regulation and improvement
The present invention is based primarily upon two thinkings for the adjustment of network structure:
First is that in order to realize that rapidity, the present invention carry out the structure and parameter of four parts of existing DehazeNet
It improves；
Second is that in order to reach better defog effect, it is contemplated that shade that may be present causes atmosphere light to shine in safety monitoring
The problem of mutation, the invention proposes the multitask structures under caffe frame, at the same to image transmitance and atmosphere light constant into
Study is gone.
Optimized network structure is as shown in Figure 3.
It can be seen that the present invention is provided with to reduce the calculation amount when processing of final picture in multiple dimensioned demapping section
Parallel convolutional layer conv2/1 × 1 that one size is 1 × 1.Correspondingly, in reshaping layer, the present invention has marked off one newly
Dimension reshape_a, and in convolution articulamentum later, the present invention has also accepted a new bottom structure, for will
This smaller convolution kernel conv2/1x1 is included in network overall architecture.
In addition, it is contemplated that the size of existing training set is limited and existing network weight is it is known that present invention uses fine tunings
Coaching method (fine-tuning).The present invention uses Ze Weier (Xaiver) initial method, passes through this initial method, mind
It is as equal as possible through each layer of output variance in network, it ensure that information preferably flows in network.
And it is then similar with transmitance network design for the learning network mentality of designing of atmosphere light radiation, that is, utilize nerve net
Network layers imitate traditional algorithm.The present invention still first with convolutional layer conv_a, reshape layer reshape_a for picture
Essential characteristic is extracted.The atmosphere light constant of image is obtained using a maximum pond layer pool_a later.Finally, originally
Invention, which is still reached using bilateral line rectification unit, accelerates convergent effect.
2, training set production and retraining
The training set of neural network is used increases haze effect based on atmospherical scattering model by the fogless figure of standard at random
Fruit and generate have mist figure.
This training mode is proved to be feasible in the prior art.Fogless figure source of the invention is then
The data set in 2014 that Middlebury Stereo database increases newly.By picture segmentation at 16x16's when present invention training
Image block adds haze effect to these pictures according to formula (1).The transmitance of each picture be then by Monte Carlo Method with
What machine generated, while recording as label.The jpg format of training set is as shown in figure 4, be converted later for lmdb lattice
Formula:
In neural metwork training, the present invention has selected to be finely adjusted mode using newly-increased training set under existing weight
Training, i.e., existing weight path, and newly-increased training set are added using-weights order in training, so that weight is more
It adds kind.The present invention takes the scale of 50 picture of an iteration to be trained, and it is 4 that validation interval, which is arranged,.Finally, network
There is certain progress for the effect of image procossing, all increases in selected three evaluation criterions of the present invention.
3, treatment effect is analyzed
After the present invention makes fine tuning to network structure and carries out retraining using new data set, the mean square error of network
0.0089 is reduced to from initial 0.0127.As can be seen that the training mode of bigger data set and fine tuning compensates for net
The not convergence problem that network structural change may cause.In order to learn transmitance and atmosphere light constant simultaneously, in a multi-tasking mode,
The present invention has re-started training, and with the further complication of network structure, mean square error is increased, but is mentioned herein
Under the standard of comentropy and average gradient out, still improved for the treatment effect of single image, specific effect is shown in Table
1。
Table 1 single width figure DehazeNet, DehazeNet fine tuning are compared with Multitask defog effect
Other than quantitative analysis, defog effect is also that can feel visible, by taking certain city's haze image as an example.Such as Fig. 5-1
With Fig. 5-2, it can be clearly seen that, the clear many of the tower crane of Fig. 5-2.
Later using the different size single images of network checks function treatment, and the processing time is had recorded, is listed in simultaneously
It in table 2, and is compared with the processing time required for dark defogging, it can be seen that utilize the DehazeNet after training
Defogging is carried out, rapidity is significantly improved, and is added to the improvement DehazeNet defogging rapidity of atmosphere light constant learning structure
Although being declined slightly, but obtain better defog effect (as shown in table 1).
Different size picture DCP, DehazeNet and Multitask the method defogging times of table 2
| Dimension of picture | Dark channel prior | DehazeNet | Multitask |
| 800x600 | 0.154s | 0.067s | 0.089s |
| 1280x720 | 0.298s | 0.138s | 0.177s |
| 1600x900 | 0.815s | 0.421s | 0.589s |
| 1920x1080 | 0.926s | 0.679s | 0.790s |
(4) image restoration parallelization
Call GPU Efficient Solution overall situation atmosphere light constant α.When calculating global atmosphere light constant, need every in movement images
The transmitance t (x) of a bit, and therefrom obtain the smallest point of transmissivity.Comparison procedure can be accelerated with parallel.
Firstly, defining kernel function relatively obtains the smallest point of transmissivity in image.The image to be processed of camera acquisition
It is 1920 × 1080 size, image shares 1080 rows.36 block are distributed, a block is enabled to handle the image of 30 rows,
30 thread are distributed in one block, a thread handles a line image.Each thread executes identical kernel function,
Compare to obtain that the smallest point of transmissivity in each row of data.
Due to the processing between row and row be it is parallel, to obtain the minimum transmittance of entire image, it is necessary to do again row with
Comparison between row thus needs temporarily to store the minimum value that every a line obtains.The present invention is in view of calling memory that can consume
A large amount of resource, and waste time, therefore shared drive is used, reduce expense, while accelerating arithmetic speed.
Apply for 33 × 33 shared drive, in total 1089 bank, the minimum transmittance for 1080 rows that can be stored
Data.The capable comparison result between row, the present invention are assigned with a block in order to obtain after comparison in current row,
30 thread are set in this block, and each thread is responsible for relatively and obtains the most value in 36 data；Similarly, Shen
Bank that please be one 3 × 10 utilizes shared drive to record intermediate result；After this is calculated, 30 data are only left.This
When, data processing amount has been greatly reduced, and only operation need to can be rapidly completed with a thread.
In this way, being substantially reduced by the solution of merger three times more just completed to piece image minimum transmittance
Operand accelerates arithmetic speed.Transmissivity t (x) and global atmosphere light constant α have been obtained, recovery can be calculated
Image.
Serial computing is carried out if it is with CPU, i.e., to carry out 1920 × 1080=2073600 times circulation, do so extremely
It wasting time and resource, and the processing of each pixel and uncomplicated, the processing between pixel and pixel does not couple,
Therefore arithmetic speed can be accelerated by GPU acceleration.
By formula (16) it is found that due to only needing once-through operation to can be obtained by output image, thus do not need for
Thread distributes additional memory, directly exports.8 block are distributed, each block needs to handle 259200 pixels,
12 × 20=240 thread is distributed to each block, each thread need to handle 1080 pixels.Each thread is executed
Identical kernel function kernel, equipment end dispatch warp and corresponding output image J (x) are calculated with peak efficiency.Through excessive
Match, arithmetic speed can be accelerated, realizes processing in real time.
But since the process for solving transmissivity is excessively complicated, i.e., improving operational speed is also difficult to using GPU acceleration, it is contemplated that
Under no wind condition, the variation of transmissivity in a short time is almost nil, so, the atmospheric transmission of calculating in 10 seconds is set
Rate reduces unnecessary operation in this way, realizes the real-time processing of defogging.
Technological difficulties of the invention are mainly reflected in following three aspect: first is that the selection of defogging algorithm；Second is that neural network
Training and building；Third is that system framework is built.Monitor video defogging system of the present invention is in conjunction with traditional defogging algorithm and newly
Type defogging network, and it is only limited with recovery effect of the data processing algorithm to thick fog, once mist overrich, the result of Penetrating Fog algorithm
It is just no longer credible.By the comparison of a large amount of pictures, a suitable transmitance is provided using " two classification " in neural network
Threshold value, as judging the whether believable final foundation of Penetrating Fog algorithm final result.
The utility model has the advantages that
The present invention improves simultaneously retraining based on the DehazeNet system of neural convolutional network to obtain artwork transmission rate
Figure, and real-time video defogging is realized using parallelization calculating.In building with training for neural network, present invention uses double
Sideline property amending unit as activation primitive, use Ze Weier distribution as weights initialisation distribution to reduce search space and mention
High convergence rate.The present invention is proposed using comentropy and average gradient as defog effect index, on this basis, using caffe
Multi job mode under frame devises while learning the neural network of transmitance and atmosphere light constant, so that at single image
Reason effect obtains further promotion under appraisement system of the invention.After obtaining transmitance figure, the present invention passes through
CUDA calls kernel function, realizes that parallel computation restores speed with speeding up picture on GPU, has reached wanting for real-time video defogging
It asks.
Compared to traditional single image defogging, the present invention realizes the real-time defogging of video, while ensure that defogging
Effect solves supersaturation existing for traditional defogging method and skyline fuzzy problem.The present invention is realized direct using Python
The processing of software defogging is carried out after calling the video data of camera, compared to traditional optics Penetrating Fog, without setting to image acquisition
It is standby to carry out additional update and upgrading, dramatically save capital investment.
Claims (5)
1. it is a kind of based on the real-time defogging method of monitor video for improving DehazeNet, it is characterized in that the step of defogging method
Include:
1) by video capture device acquire video, video is then cut into single picture frame by frame, be supplied to neural network into
Row processing；
2) trained improved DehazeNet neural network is utilized, i.e., each layer weight is it is known that the picture for input divides
Block is handled to obtain the transmitance t (x) of each section and atmosphere light constant, finally constitutes transmitance distribution map and atmosphere light constant point
Butut；
3) output for obtaining neural network solves fogless figure according to based on atmospherical scattering model defogging algorithm, then by fogless figure weight
Newly it is spliced into video；
The step of step 2) includes:
2.1) feature extraction is carried out to the picture that step 1) obtains:
Improved DehazeNet network is to increase new convolutional layer conv_a in DehazeNet network and reshape layer
reshape_a；Convolutional layer conv_a is parallel with the convolutional layer conv1 in DehazeNet network；Reshaping a layer reshape_a is
Reshaping in DehazeNet network marks off in layer reshape1；
It by convolutional layer conv_a, reshapes layer reshape_a the essential characteristic of picture is extracted, then by maximum pond layer
Pool_a extracts the atmosphere light constants of sampled images；
By convolutional layer conv1, reshape layer reshape1 the essential characteristic of picture extracted, then by pond layer pool_1 Lai
Further extract the atmospheric transmissivity of image；It is set after the layer pool_1 of pond and reshapes a layer reshape2；
2.2) multiple dimensioned mapping:
Output to layer reshape2 is reshaped uses parallel convolutional coding structure, and the convolution kernel size selected is 1 × 1,3 × 3,5
× 5 and 7 × 7, pass through four convolutional layer conv2/1 × 1 of different sizes, conv2/3 × 3, conv2/5 × 5, conv2/7
× 7, four corresponding features of scale will be extracted in image；Then layer conv2/output is spliced by convolution and realizes four
Splicing between a input data；
2.3) local extremum:
For step 2.2) as a result, being set as the pond layer pool2 extraction of maximum MAX, the cunning of pond layer pool2 by one again
Dynamic step-length is 1；
2.4) non-linear convergence:
For step 2.3) as a result, using bilateral line rectification unit as activation primitive, such as following formula:
In the step 3):
Global atmosphere light constant α: the transmitance t (x) of each pixel x in movement images is first solved, and therefrom obtains transmissivity
The smallest point of numerical value, the intensity value of the point is as global atmosphere light constant α；
Then fogless figure is solved according to based on atmospherical scattering model defogging algorithm, clear image J (x) is according to t (x) and α by formulaIt restores, wherein I (x) is the mist image primitive observed.
2. the real-time defogging method of monitor video according to claim 1, it is characterized in that in the step 3), fogless nomography
Selection process be:
Atmospherical scattering model is stated with following equation (1):
I (x)=J (x) t (x)+α (1-t (x)), (1)
Wherein, I (x) is the mist image primitive observed, J (x) is the mist elimination image pixel after recovering, and t (x) is atmospheric transmission
Rate, α are global atmosphere light constants；In formula (1), there are three unknown parameter J (x), t (x), α, are estimating t (x) and α
Afterwards, real scene figure J (x) is resumed to obtain；
Atmospheric transmissivity t (x) is used to describe not scattered and reached institute's accounting of the light of camera, is defined by formula (2):
T (x)=e-βd(x), (2)
Wherein, d (x) is distance of the corresponding scene point of the pixel to camera；
It can be obtained by (1), (2) synthesis
α=I (x), d (x) → ∞, (3)
In actual imaging process, depth of field d (x) can not tend to be infinite, but a very little can be defined in remote situation
Transmissivity t0；In this case, it no longer uses in formula (3) and obtains the method for atmosphere light, according to the following formula the side of (4)
Method estimates that atmosphere light can be more acurrate:
Based on above-mentioned analysis, it can be deduced that conclusion: accurately estimating atmospheric transmissivity is the key that recover clear image.
3. the real-time defogging method of monitor video according to claim 1, it is characterized in that being solved global big in the step 3)
The method of gas light constant α is: being solved with single block by the merger of Sharing Memory Realization transmittance minimum, centre is tied
Fruit is stored in shared drive；Most value between row and row is also realized using single block, is acquired by merger all
Capable most value, specific as follows finally using the intensity value of the smallest point of transmittance values as global atmosphere light constant α:
When calculating global atmosphere light constant, the transmitance t (x) of every bit in movement images is needed, and therefrom obtains transmissivity most
Small point, comparison procedure are accelerated with parallel:
Firstly, defining kernel function relatively obtains the smallest point of transmissivity in image, the image to be processed of camera acquisition is
The size of 1920*1080, image share 1080 rows；
36 block are distributed, a block is enabled to handle the image of 30 rows, 30 thread of distribution in a block, one
Thread handles a line image；Each thread executes identical kernel function, and it is the smallest to compare to obtain transmissivity in each row of data
That point；
Due to the processing between row and row be it is parallel, to obtain the minimum transmittance of entire image, it is necessary to do again capable with row
Between comparison, thus need temporarily to store the minimum value that every a line obtains；
Apply for the shared drive of 33*33, in total 1089 bank, the minimum transmittance data for 1080 rows stored；Current row
After interior comparison, the capable comparison result between row, is assigned with a block, is arranged in this block in order to obtain
30 thread, each thread are responsible for relatively and obtain the most value in 36 data；Similarly, apply for a 3*10's
Bank records intermediate result using shared drive；After this is calculated, 30 data are only left；
Transmissivity t (x) and global atmosphere light constant α have been obtained, then has carried out restored image.
4. the real-time defogging method of monitor video according to claim 3, it is characterized in that in the step 3), restored image
When, corresponding J (x) is obtained by parallel computation；When realizing parallel computation on GPU, parallel thread number is pixel number, often
A thread can calculate the corresponding clear image J (x) of pixel, realize parallel accelerate；
ByKnown to, it is only necessary to once-through operation can be obtained by output image, so do not need for
Thread distributes additional memory, directly exports；
8 block are distributed, each block needs to handle 259200 pixels, distributes 12 × 20=240 to each block
Thread, each thread need to handle 1080 pixels；Each thread executes identical kernel function kernel, equipment end scheduling
Corresponding output image J (x) is calculated with peak efficiency in warp.
5. the real-time defogging method of monitor video according to claim 3, it is characterized in that being arranged 10 seconds in the step 3)
Calculate an atmospheric transmissivity.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811261910.XA CN109389569B (en) | 2018-10-26 | 2018-10-26 | Monitoring video real-time defogging method based on improved DehazeNet |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811261910.XA CN109389569B (en) | 2018-10-26 | 2018-10-26 | Monitoring video real-time defogging method based on improved DehazeNet |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109389569A true CN109389569A (en) | 2019-02-26 |
| CN109389569B CN109389569B (en) | 2021-06-04 |
Family
ID=65427927
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811261910.XA Expired - Fee Related CN109389569B (en) | 2018-10-26 | 2018-10-26 | Monitoring video real-time defogging method based on improved DehazeNet |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109389569B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109961070A (en) * | 2019-03-22 | 2019-07-02 | 国网河北省电力有限公司电力科学研究院 | The method of mist body concentration is distinguished in a kind of power transmission line intelligent image monitoring |
| CN110059699A (en) * | 2019-03-18 | 2019-07-26 | 中南大学 | Skyline automatic testing method in a kind of image based on convolutional neural networks |
| CN110807744A (en) * | 2019-10-25 | 2020-02-18 | 山东工商学院 | Image defogging method based on convolutional neural network |
| CN113632091A (en) * | 2019-03-22 | 2021-11-09 | 辉达公司 | Iterative spatial graph generation |
| CN114387187A (en) * | 2022-01-13 | 2022-04-22 | 河海大学 | Real-time visual enhancement embedded system and visual enhancement method for rainy, snowy and foggy days |
| CN115032817A (en) * | 2022-05-27 | 2022-09-09 | 北京理工大学 | Real-time video defogging intelligent glasses for use in severe weather and control method |
| CN114387187B (en) * | 2022-01-13 | 2024-04-26 | 河海大学 | Real-time visual enhancement embedded system and visual enhancement method for rainy, snowy and foggy days |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105741246A (en) * | 2016-01-29 | 2016-07-06 | 南京航空航天大学 | Method for eliminating influence on optical picture by dust and mist |
| WO2017086296A1 (en) * | 2015-11-16 | 2017-05-26 | デクセリアルズ株式会社 | Optical body, master, and method for manufacturing optical body |
| WO2017175231A1 (en) * | 2016-04-07 | 2017-10-12 | Carmel Haifa University Economic Corporation Ltd. | Image dehazing and restoration |
| CN107749052A (en) * | 2017-10-24 | 2018-03-02 | 中国科学院长春光学精密机械与物理研究所 | Image defogging method and system based on deep learning neutral net |
| CN107967671A (en) * | 2017-10-30 | 2018-04-27 | 大连理工大学 | With reference to data study and the image defogging method of physics priori |
| CN108564535A (en) * | 2017-12-15 | 2018-09-21 | 四川大学 | A kind of image defogging method based on deep learning |
| CN108564077A (en) * | 2018-04-03 | 2018-09-21 | 哈尔滨哈船智控科技有限责任公司 | It is a kind of based on deep learning to detection and recognition methods digital in video or picture |
| CN108665432A (en) * | 2018-05-18 | 2018-10-16 | 百年金海科技有限公司 | A kind of single image to the fog method based on generation confrontation network |
-
2018
- 2018-10-26 CN CN201811261910.XA patent/CN109389569B/en not_active Expired - Fee Related
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017086296A1 (en) * | 2015-11-16 | 2017-05-26 | デクセリアルズ株式会社 | Optical body, master, and method for manufacturing optical body |
| CN105741246A (en) * | 2016-01-29 | 2016-07-06 | 南京航空航天大学 | Method for eliminating influence on optical picture by dust and mist |
| WO2017175231A1 (en) * | 2016-04-07 | 2017-10-12 | Carmel Haifa University Economic Corporation Ltd. | Image dehazing and restoration |
| CN107749052A (en) * | 2017-10-24 | 2018-03-02 | 中国科学院长春光学精密机械与物理研究所 | Image defogging method and system based on deep learning neutral net |
| CN107967671A (en) * | 2017-10-30 | 2018-04-27 | 大连理工大学 | With reference to data study and the image defogging method of physics priori |
| CN108564535A (en) * | 2017-12-15 | 2018-09-21 | 四川大学 | A kind of image defogging method based on deep learning |
| CN108564077A (en) * | 2018-04-03 | 2018-09-21 | 哈尔滨哈船智控科技有限责任公司 | It is a kind of based on deep learning to detection and recognition methods digital in video or picture |
| CN108665432A (en) * | 2018-05-18 | 2018-10-16 | 百年金海科技有限公司 | A kind of single image to the fog method based on generation confrontation network |
Non-Patent Citations (4)
| Title |
|---|
| WANG W: "Fast image dehazing method based on linear transformation", 《JOURNAL OF TECHNOLOGY & SCIENCE》 * |
| ZHU QINGSONG: "A fast single image haze removal algorithm using color attenuation prior", 《IEEE TRANSACTIONS ON IMAGE PROCESSING》 * |
| 徐岩: "基于多特征融合的卷积神经网络图像去雾算法", 《激光与光电子学进展》 * |
| 李策: "生成对抗映射网络下的图像多层感知去雾算法", 《计算机辅助设计与图形学学报》 * |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110059699A (en) * | 2019-03-18 | 2019-07-26 | 中南大学 | Skyline automatic testing method in a kind of image based on convolutional neural networks |
| CN110059699B (en) * | 2019-03-18 | 2021-01-29 | 中南大学 | Automatic detection method for skyline in image based on convolutional neural network |
| CN109961070A (en) * | 2019-03-22 | 2019-07-02 | 国网河北省电力有限公司电力科学研究院 | The method of mist body concentration is distinguished in a kind of power transmission line intelligent image monitoring |
| CN113632091A (en) * | 2019-03-22 | 2021-11-09 | 辉达公司 | Iterative spatial graph generation |
| CN110807744A (en) * | 2019-10-25 | 2020-02-18 | 山东工商学院 | Image defogging method based on convolutional neural network |
| CN110807744B (en) * | 2019-10-25 | 2023-09-08 | 山东工商学院 | Image defogging method based on convolutional neural network |
| CN114387187A (en) * | 2022-01-13 | 2022-04-22 | 河海大学 | Real-time visual enhancement embedded system and visual enhancement method for rainy, snowy and foggy days |
| CN114387187B (en) * | 2022-01-13 | 2024-04-26 | 河海大学 | Real-time visual enhancement embedded system and visual enhancement method for rainy, snowy and foggy days |
| CN115032817A (en) * | 2022-05-27 | 2022-09-09 | 北京理工大学 | Real-time video defogging intelligent glasses for use in severe weather and control method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109389569B (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Image dehazing using residual-based deep CNN | |
| CN109389569A (en) | Based on the real-time defogging method of monitor video for improving DehazeNet | |
| Ma et al. | Perceptual quality assessment for multi-exposure image fusion | |
| Wang et al. | Biologically inspired image enhancement based on Retinex | |
| Wang et al. | A fast single-image dehazing method based on a physical model and gray projection | |
| CN111127416A (en) | Computer vision-based automatic detection method for surface defects of concrete structure | |
| CN111292264A (en) | Image high dynamic range reconstruction method based on deep learning | |
| CN111160202A (en) | AR equipment-based identity verification method, AR equipment-based identity verification device, AR equipment-based identity verification equipment and storage medium | |
| Li et al. | Insulator defect detection for power grid based on light correction enhancement and YOLOv5 model | |
| Wang et al. | Exploiting local degradation characteristics and global statistical properties for blind quality assessment of tone-mapped HDR images | |
| CN111445496B (en) | Underwater image recognition tracking system and method | |
| CN114627269A (en) | Virtual reality security protection monitoring platform based on degree of depth learning target detection | |
| Fu et al. | Multi-feature-based bilinear CNN for single image dehazing | |
| Feng et al. | Low-light image enhancement algorithm based on an atmospheric physical model | |
| Khan et al. | Recent advancement in haze removal approaches | |
| Zhao et al. | A multi-scale U-shaped attention network-based GAN method for single image dehazing | |
| CN117152182B (en) | Ultralow-illumination network camera image processing method and device and electronic equipment | |
| CN113538304B (en) | Training method and device for image enhancement model, and image enhancement method and device | |
| Kim | Image enhancement using patch-based principal energy analysis | |
| Liu et al. | Enhanced image no‐reference quality assessment based on colour space distribution | |
| CN113705380A (en) | Target detection method and device in foggy days, electronic equipment and storage medium | |
| CN113705381A (en) | Target detection method and device in foggy days, electronic equipment and storage medium | |
| Huang et al. | Image dehazing based on robust sparse representation | |
| CN110135274A (en) | A kind of people flow rate statistical method based on recognition of face | |
| Guo et al. | Single Image Dehazing Using Adaptive Sky Segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210604 Termination date: 20211026 |