CN112906832A - Unbalanced data sampling method and system based on category activation mapping - Google Patents
Unbalanced data sampling method and system based on category activation mapping Download PDFInfo
- Publication number
- CN112906832A CN112906832A CN202110479005.7A CN202110479005A CN112906832A CN 112906832 A CN112906832 A CN 112906832A CN 202110479005 A CN202110479005 A CN 202110479005A CN 112906832 A CN112906832 A CN 112906832A
- Authority
- CN
- China
- Prior art keywords
- data set
- sampling
- picture
- category
- activation mapping
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention discloses an unbalanced data sampling method and system based on category activation mapping, wherein the method comprises the following steps: by random oversampling on the unbalanced data set; circularly traversing the sampling data set to generate a category activation mapping corresponding to each picture; classifying the picture into foreground information and background information by using category activation mapping; keeping the background information of the picture unchanged, and converting the foreground information of the picture; replacing the picture in the original sampling data set with the transformed picture, and continuously traversing the rest pictures until the traversal of the sampling data set is finished; and forming a balanced data set by the transformed data set and the original data set, and further finely adjusting the model by combining a resampling method to obtain a final model. The invention utilizes the class activation mapping of the picture to segment the foreground and background information of the picture, and the model is trained on the enhanced data set to obtain further feature learning and representing capability.
Description
Technical Field
The invention belongs to the field of category unbalanced image identification, and particularly relates to an unbalanced data sampling method and system based on category feature mapping.
Background
The unbalanced data distribution is a real phenomenon existing in natural data, in the related research of image classification, unbalanced data, especially long-tail data, is one of the popular directions of the current research, and related research contents and achievements also meet the requirements of practical application. Unbalanced image data is that in a data set, the number of images occupied by different classes is unequal, in a long tail data distribution, few classes (head classes) occupy most pictures in the data set, and most classes (tail classes) occupy only little image data.
The neural network model performs poorly on the tail class after being trained on an unbalanced data set, mainly because the number of pictures of the tail class is small. In the neural network model training process, most training data are occupied by the head classes, and tail class pictures used in the model training are less than the head classes, so that the classification performance of the model on the tail class data is poor. For the phenomenon that the neural network model is poor in performance of the tail category, researchers put forward a resampling (re-sampling) method on a data level, and in the training process, training data of the tail category are increased through a resampling technology. The main process of the resampling technology is that before each training, unlike random sampling, the resampling technology increases the probability of being sampled for the tail data, so that the image data of the tail category is increased in the training data after sampling. By the resampling technology, the tail data utilized by the neural network model in the training process is increased, so the classification performance of the model on the tail category is enhanced.
Although the resampling technique has a good classification effect on unbalanced data, only original image data information is utilized, and new information with discriminability is not added in the training data after resampling. Although some existing resampling methods utilize data interpolation, model gradient information, etc. to generate new samples to add discriminative information for model training, these methods have the following problems: 1) the complexity is high, the required auxiliary information such as gradient and the like has large acquisition cost; 2) the method cannot well control the generated semantic information of the new sample, and the generated new sample contains large noise information.
Disclosure of Invention
The invention aims to provide an unbalanced data sampling method and system based on class activation mapping.
The technical scheme for realizing the purpose of the invention is as follows:
an unbalanced data sampling method based on class activation mapping, comprising:
random oversampling is carried out on the unbalanced data set, and sampling pictures form a sampling data set;
circularly traversing the sampling data set, and generating a corresponding class activation mapping for each traversed picture by using the trained neural network model;
segmenting foreground information and background information of the picture by utilizing category activation mapping, only carrying out image enhancement on the foreground information of the picture, and keeping the background information unchanged;
replacing the enhanced picture with the picture in the original sampling data set, and continuously traversing the rest pictures until the traversal of the sampling data set is finished to obtain an enhanced sampling data set;
and combining the enhanced sampling data set and the original data set to form a balanced data set, and combining a resampling technology on the balanced data set to adjust the model.
Further, random oversampling is performed on the unbalanced data set, and the sampled pictures form a sampled data set, wherein the random oversampling technology increases the setting of a sampling threshold, and if the number of pictures included in a certain category in the unbalanced data set is greater than or equal to the sampling threshold, the category of pictures does not need to be sampled; for classes in the data set for which the number of class pictures is less than the sampling threshold, random oversampling is performed such that the number of class pictures in the new balanced sampled data set is the difference between the sampling threshold and the original number of pictures.
Further, the segmenting of the foreground information and the background information of the picture by using the category activation mapping specifically includes:
for a given picture, let
Representing the neurons in the last convolutional layer of the neural network
In spatial position
For the neuron, then
Applying global average pooling yields:
thus for a given picture category
The corresponding softmax layer input may be expressed as:
wherein
I.e. the mapping is activated for the category,
parameters of the last full connection layer of the neural network;
obtaining corresponding class activation mapping of the sampling picture through a neural network model, and then taking the average value of the class activation mapping as a foreground and background segmentation threshold; in the category activation mapping, a value greater than or equal to the segmentation threshold is taken as a foreground, and a value smaller than the segmentation threshold is taken as a background; and after obtaining the foreground and the background after segmentation, keeping the background pixel value unchanged, and only carrying out image enhancement operation on the foreground.
Further, image enhancement comprises horizontal turning, rotation, scaling and translation, and for each picture, one image enhancement is randomly selected to be applied to the foreground in three image enhancement modes to obtain an enhanced image.
Further, the rotation angle range is set to [ -45 ]。, 45。]The zoom range is [80%, 120%](ii) a For the enhancement of the translation image, ensuring that the translation range of the foreground does not exceed the range of the original image; the blank pixels after the image transformation are filled by the pixels at the corresponding positions of the original image.
Further, a sampling data set formed by the enhanced image and original image data form a balance data set, and a resampling technology is applied to the new balance data set to finely adjust the model; wherein the sampling weights of the resampling technique are calculated from the raw data set class information.
An unbalanced data sampling system based on class activation mapping, comprising:
the random oversampling module is used for performing random oversampling on the unbalanced data set, and sampling pictures form a sampling data set;
the category activation mapping module is used for segmenting foreground information and background information of the picture by using category activation mapping, only carrying out image enhancement on the foreground information of the picture and keeping the background information of the picture unchanged;
the circulating traversal module generates corresponding category activation mapping for each traversed picture by using the trained neural network model when randomly oversampling; after image enhancement processing, replacing the enhanced picture with the picture in the original sampling data set, and continuously traversing the rest pictures until the traversal of the sampling data set is finished to obtain an enhanced sampling data set;
and the resampling module is used for combining the sampling data set formed by the enhanced pictures and the original data set to form a balanced data set, and combining the resampling technology on the balanced data set to finely adjust the model.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the unbalanced data sampling method described above when executing the program.
A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the above-mentioned unbalanced data sampling method.
Compared with the prior art, the invention has the following remarkable advantages: (1) the method and the device divide the foreground information and the background information of the picture by utilizing the category activation mapping, and have the advantages of insensitive parameters and accurate foreground positioning, so that the generated semantic information of the picture is controllable, and noise information with large influence is not brought. (2) The image foreground and background information are segmented by category activated mapping, and only the foreground is subjected to image enhancement while the background information is kept unchanged. And a new sample with discriminability is generated for the resampling technology, so that the model feature representation capability and performance in the neural network model training process are further improved. (3) The sample of the generated new balanced data set is composed of the sampling data set and the original data set which are subjected to image enhancement transformation, so that the performance of the model can be improved from the enhanced sampling picture in the training process of the neural network model, and meanwhile, the learning stability of the neural network model is ensured in the fine adjustment process due to the retention of the original data set.
Drawings
FIG. 1 is a flow chart of an unbalanced data sampling method based on class activation mapping according to the present invention.
FIG. 2 is a flow visualization diagram of an unbalanced data sampling method based on class activation mapping according to the present invention.
FIG. 3 is a data set composition process diagram of an unbalanced data sampling method based on class activation mapping according to the present invention.
Detailed Description
With reference to fig. 1, fig. 2, and fig. 3, a method for sampling unbalanced data based on class activation mapping specifically includes the following steps:
step 1, random oversampling is carried out on an unbalanced data set, and sampling pictures form a sampling data set;
for unbalanced data sets, the invention first uses a random over-sampling technique (random over-sampling) to perform balanced sampling. For the random sampling technology, the invention adds the setting of a sampling threshold value on the basis of the random sampling technology, and the sampling threshold value has the effect that if the number of pictures contained in a certain category in the unbalanced data set exceeds the sampling threshold value, the pictures in the category do not need to be sampled; for classes in the data set for which the number of class pictures is less than the sampling threshold, random oversampling is performed such that the number of class pictures in the new balanced sample data set is the sampling threshold.
Step 2, segmenting foreground information and background information of the picture by utilizing category activation mapping, only carrying out image enhancement on the foreground information of the picture, and keeping the background information unchanged;
step 2, distinguishing foreground and background information of the sampling picture based on class activation maps (class) technology.
For a given picture, let
Representing the neurons in the last convolutional layer of the neural network
In spatial position
For the neuron, then
Global average pooling (global average pooling) was applied to obtain:
thus for a given picture category
,The corresponding softmax layer input may be expressed as:
wherein
I.e. class activation maps (class activation maps),
the parameters of the last fully connected layer of the neural network.
According to the method, the corresponding class activation mapping of the sampling picture is obtained through the neural network model, and then the average value of the class activation mapping is used as the foreground and background segmentation threshold. In the class activation mapping, a value greater than or equal to the segmentation threshold is a foreground, and a value less than the segmentation threshold is a background. And after obtaining the foreground and the background after segmentation, keeping the background pixel value unchanged, and only carrying out image enhancement operation on the foreground.
The image enhancement comprises horizontal turning, rotation, scaling and translation, and for each picture, one image enhancement is randomly selected to be applied to the foreground in three image enhancement modes to obtain an enhanced image. For rotation and zoom image enhancement, the present invention sets its rotation angle range to [ -45 ]。, 45。]And the zoom range is [80%, 120%](ii) a And for the enhancement of the translation image, the foreground translation range is ensured not to exceed the range of the original image. The blank pixels after the image transformation are filled by the pixels at the corresponding positions of the original image.
And 3, combining a sampling data set consisting of the images after image enhancement and an original data set to form a balanced data set, and finely adjusting the neural network model on the balanced data set by utilizing a resampling technology (re-sampling).
The sample data set composed of the enhanced image obtained through step 2 and the original image data are composed into a balanced data set, as shown in fig. 3. A resampling technique is applied to the new balanced dataset to fine tune the model.
The sampling weight of the resampling technology is calculated by original data set category information, and is not a balanced data set formed after sampling enhancement, the sampling probability of the enhanced picture data can be increased by the mode, and meanwhile, the original data set is reserved to enable the learning stability of the neural network model in the fine tuning process.
The invention also provides an unbalanced data sampling system based on class activation mapping, which comprises:
the random oversampling module is used for performing random oversampling on the unbalanced data set, and sampling pictures form a sampling data set;
the category activation mapping module is used for segmenting foreground information and background information of the picture by using category activation mapping, only carrying out image enhancement on the foreground information of the picture and keeping the background information of the picture unchanged;
the circulating traversal module generates corresponding category activation mapping for each traversed picture by using the trained neural network model when randomly oversampling; after image enhancement processing, replacing the enhanced picture with the picture in the original sampling data set, and continuously traversing the rest pictures until the traversal of the sampling data set is finished to obtain an enhanced sampling data set;
and the resampling module is used for combining the sampling data set formed by the enhanced pictures and the original data set to form a balanced data set, and combining the resampling technology on the balanced data set to finely adjust the model.
It should be noted that, the implementation method of each module in the system is described in detail in the unbalanced data sampling method based on the class activation mapping, and the description of the present invention is not repeated.
The present invention also provides an electronic device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the above-described unbalanced data sampling method when executing the program.
Further, the present invention also provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described unbalanced data sampling method.
In the embodiments provided in the present application, it should be understood that the disclosed method, apparatus, and device may be implemented in other ways. For example, the above-described system embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An unbalanced data sampling method based on class activation mapping, comprising:
random oversampling is carried out on the unbalanced data set, and sampling pictures form a sampling data set;
circularly traversing the sampling data set, and generating a corresponding class activation mapping for each traversed picture by using the trained neural network model;
segmenting foreground information and background information of the picture by utilizing category activation mapping, only carrying out image enhancement on the foreground information of the picture, and keeping the background information unchanged;
replacing the enhanced picture with the picture in the original sampling data set, and continuously traversing the rest pictures until the traversal of the sampling data set is finished to obtain an enhanced sampling data set;
and combining the enhanced sampling data set and the original data set to form a balanced data set, and combining a resampling technology on the balanced data set to adjust the model.
2. The unbalanced data sampling method based on the category-activated mapping according to claim 1, wherein the unbalanced data set is randomly oversampled, and the sampled pictures form a sampled data set, wherein the random oversampling technology increases a sampling threshold setting, and if the number of pictures included in a category in the unbalanced data set is greater than or equal to the sampling threshold, the picture in the category does not need to be sampled; for classes in the data set for which the number of class pictures is less than the sampling threshold, random oversampling is performed such that the number of class pictures in the new balanced sampled data set is the difference between the sampling threshold and the original number of pictures.
3. The unbalanced data sampling method based on class activation mapping according to claim 1, wherein the segmentation of the foreground information and the background information of the picture using the class activation mapping is as follows:
for a given picture, let
Representing the neurons in the last convolutional layer of the neural network
In spatial position
For the neuron, then
Applying global average pooling yields:
thus for a given picture category
The corresponding softmax layer input may be expressed as:
wherein
I.e. the mapping is activated for the category,
parameters of the last full connection layer of the neural network;
obtaining corresponding class activation mapping of the sampling picture through a neural network model, and then taking the average value of the class activation mapping as a foreground and background segmentation threshold; in the category activation mapping, a value greater than or equal to the segmentation threshold is taken as a foreground, and a value smaller than the segmentation threshold is taken as a background; and after obtaining the foreground and the background after segmentation, keeping the background pixel value unchanged, and only carrying out image enhancement operation on the foreground.
4. The unbalanced data sampling method based on class-activated mapping according to claim 3, wherein the image enhancement comprises horizontal flipping, rotation and scaling, and translation, and for each picture, one of the three image enhancement modes is randomly selected for foreground application to obtain an enhanced image.
5. The unbalanced data sampling method based on class activation mapping according to claim 4, wherein the rotation angle range is set to
The zoom range is [80%, 120%]。
6. The unbalanced data sampling method based on category-activated mapping of claim 4, wherein the foreground translation range is guaranteed not to exceed the original image range during translation; the blank pixels after image enhancement are filled by the pixels at the corresponding positions of the original image.
7. The unbalanced data sampling method based on class activation mapping according to claim 3, wherein the sampling data set composed of enhanced images and the original image data are composed into a balanced data set, and a resampling technique is applied to the new balanced data set to adjust the model, wherein the sampling weight of the resampling technique is calculated from the original data set class information.
8. An unbalanced data sampling system based on class activation mapping, comprising:
the random oversampling module is used for performing random oversampling on the unbalanced data set, and sampling pictures form a sampling data set;
the category activation mapping module is used for segmenting foreground information and background information of the picture by using category activation mapping, only carrying out image enhancement on the foreground information of the picture and keeping the background information of the picture unchanged;
the circulating traversal module generates corresponding category activation mapping for each traversed picture by using the trained neural network model when randomly oversampling; after image enhancement processing, replacing the enhanced picture with the picture in the original sampling data set, and continuously traversing the rest pictures until the traversal of the sampling data set is finished to obtain an enhanced sampling data set;
and the resampling module is used for combining the sampling data set formed by the enhanced pictures and the original data set to form a balanced data set, and adjusting the model on the balanced data set by combining the resampling technology.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the unbalanced data sampling method as claimed in any one of claims 1 to 7 when executing the program.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the unbalanced data sampling method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110479005.7A CN112906832A (en) | 2021-04-30 | 2021-04-30 | Unbalanced data sampling method and system based on category activation mapping |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110479005.7A CN112906832A (en) | 2021-04-30 | 2021-04-30 | Unbalanced data sampling method and system based on category activation mapping |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112906832A true CN112906832A (en) | 2021-06-04 |
Family
ID=76108945
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110479005.7A Pending CN112906832A (en) | 2021-04-30 | 2021-04-30 | Unbalanced data sampling method and system based on category activation mapping |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112906832A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113780463A (en) * | 2021-09-24 | 2021-12-10 | 北京航空航天大学 | Multi-head normalization long tail classification method based on deep neural network |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109767440A (en) * | 2019-01-11 | 2019-05-17 | 南京信息工程大学 | A kind of imaged image data extending method towards deep learning model training and study |
| CN110674846A (en) * | 2019-08-29 | 2020-01-10 | 南京理工大学 | Genetic algorithm and k-means clustering-based unbalanced data set oversampling method |
| CN112053366A (en) * | 2019-06-06 | 2020-12-08 | 阿里巴巴集团控股有限公司 | Model training method, sample generating method, electronic device and storage medium |
-
2021
- 2021-04-30 CN CN202110479005.7A patent/CN112906832A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109767440A (en) * | 2019-01-11 | 2019-05-17 | 南京信息工程大学 | A kind of imaged image data extending method towards deep learning model training and study |
| CN112053366A (en) * | 2019-06-06 | 2020-12-08 | 阿里巴巴集团控股有限公司 | Model training method, sample generating method, electronic device and storage medium |
| CN110674846A (en) * | 2019-08-29 | 2020-01-10 | 南京理工大学 | Genetic algorithm and k-means clustering-based unbalanced data set oversampling method |
Non-Patent Citations (2)
| Title |
|---|
| THANOS TAGARIS 等: "High-Resolution Class Activation Mapping", 《2019 IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING》 * |
| 刘有用: "基于生成对抗网络的水下多目标识别", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113780463A (en) * | 2021-09-24 | 2021-12-10 | 北京航空航天大学 | Multi-head normalization long tail classification method based on deep neural network |
| CN113780463B (en) * | 2021-09-24 | 2023-09-05 | 北京航空航天大学 | Multi-head normalization long-tail classification method based on deep neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | A closed-form solution to photorealistic image stylization | |
| US11593615B2 (en) | Image stylization based on learning network | |
| CN108764195B (en) | Handwriting model training method, handwritten character recognition method, device, equipment and medium | |
| CN111709406B (en) | Text line identification method and device, readable storage medium and electronic equipment | |
| DE102021105249A1 (en) | MICROTRAINING FOR ITERATIVE REFINEMENT OF A NEURAL NETWORK WITH FEW ADAPTATIONS | |
| CN114240735B (en) | Arbitrary style migration method, system, storage medium, computer equipment and terminal | |
| WO2023212997A1 (en) | Knowledge distillation based neural network training method, device, and storage medium | |
| US20220083808A1 (en) | Method and apparatus for processing images, device and storage medium | |
| EP3821402A1 (en) | Method for book recognition and book reading device | |
| WO2023151529A1 (en) | Facial image processing method and related device | |
| CN112906832A (en) | Unbalanced data sampling method and system based on category activation mapping | |
| Cong et al. | CAN: Contextual aggregating network for semantic segmentation | |
| Guo et al. | Attribute-controlled face photo synthesis from simple line drawing | |
| CN114386482A (en) | Image classification system and method based on semi-supervised incremental learning | |
| CN113837290A (en) | Unsupervised unpaired image translation method based on attention generator network | |
| CN114913588B (en) | Face image restoration and recognition method applied to complex scene | |
| CN113344771B (en) | Multifunctional image style migration method based on deep learning | |
| CN112419249B (en) | Special clothing picture conversion method, terminal device and storage medium | |
| CN114913382A (en) | Aerial photography scene classification method based on CBAM-AlexNet convolutional neural network | |
| CN114092494A (en) | Brain MR image segmentation method based on superpixel and full convolution neural network | |
| Li et al. | FreePIH: Training-Free Painterly Image Harmonization with Diffusion Model | |
| CN112991236A (en) | Image enhancement method and device based on template | |
| DE102021105291A1 (en) | WEIGHT DEMODULATION FOR A GENERATING NEURAL NETWORK | |
| Dy et al. | MCGAN: mask controlled generative adversarial network for image retargeting | |
| CN112446345A (en) | Low-quality three-dimensional face recognition method, system, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210604 |