CN110443759A - A kind of image defogging method based on deep learning - Google Patents

A kind of image defogging method based on deep learning Download PDF

Info

Publication number
CN110443759A
CN110443759A CN201910614615.6A CN201910614615A CN110443759A CN 110443759 A CN110443759 A CN 110443759A CN 201910614615 A CN201910614615 A CN 201910614615A CN 110443759 A CN110443759 A CN 110443759A
Authority
CN
China
Prior art keywords
mist
layer
defogging
concentration map
neural networks
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910614615.6A
Other languages
Chinese (zh)
Inventor
邱斌
苏卓
周凡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sun Yat Sen University
Original Assignee
Sun Yat Sen University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sun Yat Sen University filed Critical Sun Yat Sen University
Priority to CN201910614615.6A priority Critical patent/CN110443759A/en
Publication of CN110443759A publication Critical patent/CN110443759A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/73Deblurring; Sharpening
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/90Dynamic range modification of images or parts thereof
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10004Still image; Photographic image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20081Training; Learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/20Special algorithmic details
    • G06T2207/20084Artificial neural networks [ANN]

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a kind of image defogging method based on deep learning.Then the present invention generates the transmissivity and atmosphere light of different values, and manually have mist figure as training set by the synthesis of atmosphere photon diffusion models by collecting nature picture as fogless figure at random in the different zones of picture;Construct the convolutional neural networks for predicting mist concentration map;The training set training convolutional neural networks are utilized later;Recently enter it is to be processed have mist figure, mist concentration map is calculated using the convolutional neural networks, obtains final defogging figure with there is mist figure to subtract mist concentration map.The present invention is a kind of adaptive defogging method, the defogging result of generation compare naturally, and strong robustness, it is applied widely, can be applied to simultaneously in indoor and natural scene defogging;This method is a kind of full-automatic, defogging method end to end, does not need post-processing step.

Description

A kind of image defogging method based on deep learning
Technical field
The present invention relates to deep learnings and computer vision field, and in particular to a kind of image defogging based on deep learning Method.
Background technique
Image defogging method is the important research content of computer vision field, is widely used in video monitoring, automatically The fields such as driving.In recent years, serious with haze, the visibility and ornamental value of picture are seriously affected, to image defogging skill More stringent requirements are proposed for the effect and versatility of art.Existing image defogging method can be roughly divided into two kinds: be based on image The defogging method of priori and defogging method based on deep learning.
Based on the defogging method of image prior by the contrast of image, atmosphere light scattering is acquired the features such as distribution of color Transmittance figure in model, and then air light value is gone out by the smallest point estimation of a part of transmissivity, mould is scattered using atmosphere light Type recovers defogging figure.Method based on priori is limited to the applicability of priori, cannot be effective in the scene for not meeting priori Defogging.Such method is poor there is also operation efficiency and transmissivity estimates the problems such as inaccurate, and defog effect is difficult to reach each The requirement of kind computer vision application.
With the development of depth learning technology, there are many methods for carrying out defogging using neural network.It is most of this Class method is based on atmosphere photon diffusion models, by estimating that transmissivity and air light value restore to obtain fogless figure in mist figure from having.By It is accurate to estimate that transmittance figure and air light value are extremely difficult in the diversification and complexity of defogging scene.Recent method passes through The effect of defogging is improved using fairly large network structure or by post-processing step, increase network structure can be brought general Change ability is poor, the low problem of operational efficiency.Meanwhile by the insufficient influence of training set, the model that complicated network training goes out is past It is poor toward the extensive effect on other pictures.And increase post-processing step then and be since the capability of fitting of network is insufficient, it cannot Realize adaptive end-to-end defogging, the overhead of post-processing equally reduces the efficiency of defogging.
The way for estimating parameter according to atmosphere photon diffusion models has been abandoned there are also certain methods, then has directly been indicated with network From there is mapping relations of the mist figure to fogless figure.These methods can improve the quality of image to a certain extent, but due to discontented Sufficient physical model, there are larger differences for the result and true picture of these methods, be easy to cause color to deflect, it cannot be guaranteed that defogging The quality of figure.
To sum up, although the defogging method based on deep learning is achieved better than traditional effect based on transcendental method, but is gone There are also very big rooms for promotion for fog effect and efficiency.
An existing method is the defogging method of Combined estimator atmosphere light and transmissivity, comes from paper AOD-Net:All- in-One Dehazing Network.The technology deforms atmosphere photon diffusion models, converts parameter for defogging problem Then forecasting problem constructs 7 with multiple short circuit connections first according to atmosphere photon diffusion models compound training data Layer network structure, training network obtain defogging model.
The shortcomings that this method, is:
First, training data only has more than 1000 in above-mentioned defogging method, and all it is indoor scene, trains Model is poor in the defog effect of various outdoor scenes.
Second, the number of plies of network only has 7 layers, and capability of fitting is poor in above-mentioned defogging method, the defogging when mistiness degree is higher As a result middle fog residual is obvious.
Summary of the invention
The purpose of the present invention is overcoming the shortcomings of existing methods, a kind of image defogging side based on deep learning is proposed Method.There are two problems solved by the invention is main: lacking training data under true outdoor scene for existing method, lead to model The defect poor to real scene effect, the present invention utilize high definition outdoor images, according to atmosphere photon diffusion models in image difference Each region of scale adds mist respectively, constructs the data set including more than 54000 outdoor foggy images.For existing The defect of light-duty defogging network capability of fitting difference, the present invention utilize short-cut structure, and the residual error for devising similar U-Net is gone The output of mist network, network is mist concentration map, has mist figure to subtract mist concentration map and obtains defogging result.
To solve the above-mentioned problems, the invention proposes a kind of the image defogging method based on deep learning, the method Include:
Collect nature picture as fogless figure, then the different zones of picture generate at random different values transmissivity and Atmosphere light, and manually have mist figure as training set by the synthesis of atmosphere photon diffusion models;
Construct the convolutional neural networks for predicting mist concentration map;
It is described for predicting the convolutional neural networks of mist concentration map using training set training;
Input it is to be processed have mist figure, mistiness degree is calculated using the convolutional neural networks for predicting mist concentration map Figure, obtains final defogging figure with there is mist figure to subtract mist concentration map.
Preferably, the synthesis manually has mist figure to specifically include as training set:
Collect all Outdoor Scene pictures in SUN data set;
The Outdoor Scene picture is divided into the rectangular area of 128*128 size;
It is random to generate atmosphere light and transmissivity for each rectangular area, to form the Outdoor Scene picture It is corresponding to have mist figure;
By Outdoor Scene picture and corresponding there is mist figure to zoom to 128*128 size.What is generated has the fogless figure of mist figure to structure At one group of training data, all data constitute whole training sets.
It is preferably, described for predicting the convolutional neural networks of mist concentration map, specifically:
The network is divided into feature extraction layer, Fusion Features layer, Feature Mapping layer, shares 10 layers, is a full convolutional network;
In characteristic extraction part, sufficiently being extracted using the convolutional layer that 4 convolution kernel sizes are 3 has the feature in mist figure to believe Breath, and the convolutional layer for being 1 by two convolution kernels carries out down-sampling twice;
In Fusion Features part, characteristic pattern is amplified to original image size using the warp lamination that two step-lengths are 2, is made simultaneously The shallow-layer information in fusion the deep information and characteristic extraction part is connected with short circuit;
In Feature Mapping part, the dimension of characteristic pattern is reduced using the convolutional layer that two convolution kernel sizes are 1, is finally obtained Mist concentration map of the output of 128*128*3 as prediction;
The network architecture reference structure of residual error neural network, in network third layer and layer 7, the second layer and the 8th layer Between be provided with two short-circuit structures, third, two layers of characteristic pattern are incorporated to the seven, the eight layers.
A kind of image defogging method based on deep learning proposed by the present invention, this method are a kind of adaptive defogging sides The defogging result of method, generation compare naturally, and strong robustness, it is applied widely, can be applied to simultaneously indoor and natural scene In defogging;This method is a kind of full-automatic, defogging method end to end, does not need post-processing step.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with Other attached drawings are obtained according to these attached drawings.
Fig. 1 is the image defogging method overview flow chart of the embodiment of the present invention;
Fig. 2 is the network structure of the embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other Embodiment shall fall within the protection scope of the present invention.
Fig. 1 is the image defogging method overview flow chart of the embodiment of the present invention, as shown in Figure 1, this method comprises:
S1 collects nature picture as fogless figure, then generates the transmission of different values at random in the different zones of picture Rate and atmosphere light, and manually have mist figure as training set by the synthesis of atmosphere photon diffusion models;
S2 constructs the convolutional neural networks for predicting mist concentration map;
S3, it is described for predicting the convolutional neural networks of mist concentration map using training set training;
S4, input is to be processed mist figure, and mist is calculated using the convolutional neural networks for predicting mist concentration map Concentration map obtains final defogging figure with there is mist figure to subtract mist concentration map.
Step S1, specific as follows:
S1-1: all Outdoor Scene pictures in collection SUN data set, more than totally 54000.
S1-2: the picture in S1-1 is divided into the rectangular area of 128*128 size.
S1-3: random to generate atmosphere light and transmissivity for each rectangular area in S1-2.By statistical analysis, greatly The main value range of gas light is between [0.7,1], and the main value range of transmissivity is between [0.3,0.7], therefore this hair It is bright that the value range of the two is fixed in above-mentioned section when randomly selecting.According to the characteristic of transmissivity local continuous, this hair It is bright to keep the atmosphere light in each rectangular area identical with transmissivity.It is produced by the formula of following atmosphere photon diffusion models by original image J There is mist figure I.X in formula represents the position of each pixel in picture, and t (x) represents transmissivity, and A represents air light value.
I (x)=J (x) t (x)+A (1-t (x))
S1-4: the picture in S1-1 and S1-3 is zoomed into 128*128 size.The set for having the fogless figure pair of mist figure generated One group of training data is constituted, all 54000 groups of data constitute whole training sets.Not due to the image size in SUN data set It unanimously, the rectangular area for the 128*128 size chosen in S1-2 can be big with different proportion in representative image after zooming in and out Small region, so that the data diversity in training set is more abundant.Therefore it is trained using training set proposed by the present invention Defogging network more meets the requirement of the image defogging in complicated natural scene.
Data set construction method proposed by the present invention, it is contemplated that the diversity of mist in natural scene, in obtained data set, Area size, regional location, atmosphere light and the transmissivity of every picture fog distribution be all it is random, it is multiple to meet defogging scene The requirement of polygamy also increases the robustness of network.The atmosphere optical range being arranged in S1-3 is by dark channel prior method Estimation obtains from a large amount of pictures, can satisfy the versatility of model.In addition, since the data set is based in SUN data set Outdoor Scene picture building, the quality of original image is preferable, avoids model and is interfered by noise etc..
Step S2, network structure is as shown in Fig. 2, specific as follows:
What is proposed in the present invention is used to predict that the convolutional neural networks of mist concentration map are divided into feature extraction layer, Fusion Features Three layer, Feature Mapping layer part, share 10 layers, are a full convolutional networks.In characteristic extraction part, 4 convolution kernels are used The convolutional layer that size is 3 sufficiently extracts the characteristic information having in mist figure, and the convolutional layer for being 1 by two convolution kernels carries out twice Down-sampling.In Fusion Features part, characteristic pattern is amplified to original image size using the warp lamination that two step-lengths are 2, is made simultaneously The shallow-layer information in fusion the deep information and characteristic extraction part is connected with short circuit.In Feature Mapping part, two convolution are used The convolutional layer that core size is 1 reduces the dimension of characteristic pattern, finally obtains mist concentration map of the output of 128*128*3 as prediction.
The details of above-mentioned network structure is as shown in the table, in addition, being added to batch normalization layer behind each convolutional layer (batch normalization) and Relu activation primitive layer, is omitted in table:
Network structure scale of the invention is smaller, and the number of convolution kernel is fewer, and having only used size is 3,2 and 1 convolution kernel saves the cost of training and prediction and the quantity of parameter, while keeping the generalization ability of model more preferable.Network knot Structure is provided with two articles with reference to the structure of residual error neural network between network third layer and layer 7, the second layer and the 8th layer Third, two layers of characteristic pattern are incorporated to the seven, the eight layers by short-circuit structure.This short-circuit structure helps to merge the figure of low middle higher-dimension As feature, network-evaluated mist concentration distribution is made to be more nearly legitimate reading.
Step S3, specific as follows:
Defogging method in the present invention is needed using caffe frame training network by following steps:
S3-1: convolutional neural networks structure described in S2 is realized using caffe frame.
S3-2: trained loss function is set are as follows:
Loss=∑ | | J- (I-Iconv)||2
Wherein IconvIt is the mist concentration map predicted for the output of convolutional neural networks.
S3-3: setting learning rate is 0.01, and with Adam gradient optimizing method undated parameter, momentum is set as 0.9, is passed through Final model is obtained after 95000 iteration.
Step S4, specific as follows:
S4-1: input has mist picture I.
S4-2: there is mist picture by the model treatment generated in S3, the image of output is the mist concentration map predicted Iconv, final defogging figure is arrived with there is mist figure to subtract mist concentration mapThis process may be expressed as:
A kind of image defogging method based on deep learning that the embodiment of the present invention proposes, this method are a kind of adaptive The defogging result of defogging method, generation compare naturally, and strong robustness, it is applied widely, can be applied to simultaneously indoor and natural In the defogging of scene;This method is a kind of full-automatic, defogging method end to end, does not need post-processing step.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of above-described embodiment is can It is completed with instructing relevant hardware by program, which can be stored in a computer readable storage medium, storage Medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or CD etc..
It has been carried out in detail in addition, being provided for the embodiments of the invention a kind of image defogging method based on deep learning above Thin to introduce, used herein a specific example illustrates the principle and implementation of the invention, and above embodiments are said It is bright to be merely used to help understand method and its core concept of the invention;At the same time, for those skilled in the art, foundation Thought of the invention, there will be changes in the specific implementation manner and application range, in conclusion the content of the present specification is not It is interpreted as limitation of the present invention.

Claims (3)

1. a kind of image defogging method based on deep learning, which is characterized in that the described method includes:
Nature picture is collected as fogless figure, then generates the transmissivity and atmosphere of different values at random in the different zones of picture Light, and manually have mist figure as training set by the synthesis of atmosphere photon diffusion models;
Construct the convolutional neural networks for predicting mist concentration map;
It is described for predicting the convolutional neural networks of mist concentration map using training set training;
Input it is to be processed have mist figure, mist concentration map is calculated using the convolutional neural networks for predicting mist concentration map, Final defogging figure is obtained with there is mist figure to subtract mist concentration map.
2. a kind of image defogging method based on deep learning as described in claim 1, which is characterized in that the synthesis is artificial There is the step of mist figure is as training set, specifically include:
Collect all Outdoor Scene pictures in SUN data set;
The Outdoor Scene picture is divided into the rectangular area of 128*128 size;
It is random to generate atmosphere light and transmissivity for each rectangular area, so that it is corresponding to form the Outdoor Scene picture Have mist figure;
By Outdoor Scene picture and corresponding there is mist figure to zoom to 128*128 size.What is generated has the fogless figure of mist figure to composition one Group training data, all data constitute whole training sets.
3. a kind of image defogging method based on deep learning as described in claim 1, which is characterized in that described for predicting The convolutional neural networks of mist concentration map, specifically:
The network is divided into feature extraction layer, Fusion Features layer, Feature Mapping layer, shares 10 layers, is a full convolutional network;
In characteristic extraction part, the characteristic information having in mist figure is sufficiently extracted using the convolutional layer that 4 convolution kernel sizes are 3, and Down-sampling twice is carried out by the convolutional layer that two convolution kernels are 1;
In Fusion Features part, characteristic pattern is amplified to original image size by the warp lamination for the use of two step-lengths being 2, while using short Shallow-layer information in road connection fusion the deep information and characteristic extraction part;
In Feature Mapping part, the dimension of characteristic pattern is reduced using the convolutional layer that two convolution kernel sizes are 1, finally obtains 128* Mist concentration map of the output of 128*3 as prediction;
The network architecture reference structure of residual error neural network, between network third layer and layer 7, the second layer and the 8th layer Provided with two short-circuit structures, third, two layers of characteristic pattern are incorporated to the seven, the eight layers.
CN201910614615.6A 2019-07-09 2019-07-09 A kind of image defogging method based on deep learning Pending CN110443759A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910614615.6A CN110443759A (en) 2019-07-09 2019-07-09 A kind of image defogging method based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910614615.6A CN110443759A (en) 2019-07-09 2019-07-09 A kind of image defogging method based on deep learning

Publications (1)

Publication Number Publication Date
CN110443759A true CN110443759A (en) 2019-11-12

Family

ID=68429957

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910614615.6A Pending CN110443759A (en) 2019-07-09 2019-07-09 A kind of image defogging method based on deep learning

Country Status (1)

Country Link
CN (1) CN110443759A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111091516A (en) * 2019-12-24 2020-05-01 广州柏视医疗科技有限公司 Anti-scattering grating method and device based on artificial intelligence
CN111539250A (en) * 2020-03-12 2020-08-14 上海交通大学 Image fog concentration estimation method, system and terminal based on neural network
CN111681180A (en) * 2020-05-25 2020-09-18 厦门大学 Priori-driven deep learning image defogging method
CN111861923A (en) * 2020-07-21 2020-10-30 济南大学 Target identification method and system based on lightweight residual error network image defogging
CN113052124A (en) * 2021-04-09 2021-06-29 济南博观智能科技有限公司 Identification method and device for fogging scene and computer-readable storage medium
CN114648467A (en) * 2022-05-18 2022-06-21 中山大学深圳研究院 Image defogging method and device, terminal equipment and computer readable storage medium

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
YINGHONG LIAO 等: ""HDP-Net: Haze Density Prediction Network for Nighttime Dehazing"", 《PCM 2018: ADVANCES IN MULTIMEDIA INFORMATION PROCESSING – PCM 2018》 *

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111091516A (en) * 2019-12-24 2020-05-01 广州柏视医疗科技有限公司 Anti-scattering grating method and device based on artificial intelligence
CN111539250A (en) * 2020-03-12 2020-08-14 上海交通大学 Image fog concentration estimation method, system and terminal based on neural network
CN111539250B (en) * 2020-03-12 2024-02-27 上海交通大学 Image fog concentration estimation method, system and terminal based on neural network
CN111681180A (en) * 2020-05-25 2020-09-18 厦门大学 Priori-driven deep learning image defogging method
CN111681180B (en) * 2020-05-25 2022-04-26 厦门大学 Priori-driven deep learning image defogging method
CN111861923A (en) * 2020-07-21 2020-10-30 济南大学 Target identification method and system based on lightweight residual error network image defogging
CN113052124A (en) * 2021-04-09 2021-06-29 济南博观智能科技有限公司 Identification method and device for fogging scene and computer-readable storage medium
CN113052124B (en) * 2021-04-09 2023-02-10 济南博观智能科技有限公司 Identification method and device for fogging scene and computer readable storage medium
CN114648467A (en) * 2022-05-18 2022-06-21 中山大学深圳研究院 Image defogging method and device, terminal equipment and computer readable storage medium

Similar Documents

Publication Publication Date Title
CN110443759A (en) A kind of image defogging method based on deep learning
CN106910175B (en) Single image defogging algorithm based on deep learning
CN109447907B (en) Single image enhancement method based on full convolution neural network
CN111539887B (en) Channel attention mechanism and layered learning neural network image defogging method based on mixed convolution
CN111489372B (en) Video foreground and background separation method based on cascade convolution neural network
Chen et al. Haze removal using radial basis function networks for visibility restoration applications
CN111161277A (en) Natural image matting method based on deep learning
CN111739082B (en) Stereo vision unsupervised depth estimation method based on convolutional neural network
CN112070690B (en) Single image rain removing method based on convolution neural network double-branch attention generation
CN110956094A (en) RGB-D multi-mode fusion personnel detection method based on asymmetric double-current network
CN109146810A (en) A kind of image defogging method based on end-to-end deep learning
CN102741879A (en) Method for generating depth maps from monocular images and systems using the same
CN109584188B (en) Image defogging method based on convolutional neural network
CN111079507B (en) Behavior recognition method and device, computer device and readable storage medium
CN109509156A (en) A kind of image defogging processing method based on generation confrontation model
CN107749048B (en) Image correction system and method, and color blindness image correction system and method
CN113077505A (en) Optimization method of monocular depth estimation network based on contrast learning
CN110705366A (en) Real-time human head detection method based on stair scene
CN111598793A (en) Method and system for defogging image of power transmission line and storage medium
Qian et al. CIASM-Net: a novel convolutional neural network for dehazing image
CN109685735B (en) Single picture defogging method based on fog layer smoothing prior
Zhao et al. Image dehazing based on haze degree classification
CN118334365A (en) Novel RGB-D image saliency target detection method
Kerim et al. NOVA: Rendering virtual worlds with humans for computer vision tasks
CN117456449A (en) Efficient cross-modal crowd counting method based on specific information

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20191112