CN117893624B - Color image lossless compression and decompression method based on quaternion neural network - Google Patents
Color image lossless compression and decompression method based on quaternion neural network Download PDFInfo
- Publication number
- CN117893624B CN117893624B CN202410006795.0A CN202410006795A CN117893624B CN 117893624 B CN117893624 B CN 117893624B CN 202410006795 A CN202410006795 A CN 202410006795A CN 117893624 B CN117893624 B CN 117893624B
- Authority
- CN
- China
- Prior art keywords
- quaternion
- network
- layer
- lossless compression
- decompression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000006835 compression Effects 0.000 title claims abstract description 67
- 238000007906 compression Methods 0.000 title claims abstract description 67
- 230000006837 decompression Effects 0.000 title claims abstract description 50
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 48
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000012549 training Methods 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 33
- 238000009826 distribution Methods 0.000 claims description 30
- 238000013139 quantization Methods 0.000 claims description 22
- 239000011159 matrix material Substances 0.000 claims description 17
- 230000009467 reduction Effects 0.000 claims description 11
- 238000011156 evaluation Methods 0.000 claims description 10
- 238000011084 recovery Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 5
- 238000009827 uniform distribution Methods 0.000 claims description 5
- 238000013527 convolutional neural network Methods 0.000 claims description 4
- 238000000926 separation method Methods 0.000 claims description 4
- 238000005315 distribution function Methods 0.000 claims description 2
- 238000010801 machine learning Methods 0.000 abstract description 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000004913 activation Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention provides a color image lossless compression and decompression method based on a quaternion neural network, and belongs to the technical field of image processing and machine learning. The method comprises the following steps: constructing a network model; training a network model; lossless compression of images; the image is decompressed losslessly. The method utilizes the quaternion neural network to extract the multi-level quaternion hidden variable of the color image, and takes the extracted multi-level quaternion hidden variable as a condition to realize accurate conditional probability estimation of the color image, thereby further realizing high-performance lossless compression and decompression of the color image. The invention has excellent compression performance, short compression time and good application value.
Description
Technical Field
The invention belongs to the technical field of image processing and machine learning, and particularly relates to a color image lossless compression and decompression method based on a quaternion neural network.
Background
With the rapid development of medical, remote sensing, internet and multimedia technologies, a large number of color images are being generated, transferred and stored by people at a rapid rate. The image information contains a large amount of visually useless or redundant information, and in order to better transmit and store the image data, image compression is an important technical means. At present, the traditional image compression method basically reaches the performance ceiling, and is difficult to realize large improvement. Like the JPEG algorithm, it was proposed in 1992, and it became the standard of ISO in 1994, and JPEG2000 was the standard established in 2000. In recent years, with the rapid development of depth computing theory and GPU hardware resources, an image compression method based on depth computing has become a research hotspot. A large number of high-performance depth computing image compression algorithms have emerged, the performance of which has exceeded that of conventional image compression methods. The lossless compression can reduce the data storage amount on the premise of not losing information, so that the lossless compression is more focused and applied in the fields of remote sensing and medical images.
In the deep learning method, both forward coding and entropy model can be implemented using neural networks. The neural network introduces an activation function (activation function) so as to have strong nonlinear feature learning capability, so that unlike the traditional image compression method, deep features can be learned through the deep neural network in forward coding, and the deep features are more compact, contain more abundant information and have good distribution characteristics.
However, there is a significant problem in feature learning of color images using convolutional neural networks, namely: the multiple channels of the color image are independently operated with convolution kernels, the results of which are then simply added up, and this mode of operation results in the loss of rich information between the multiple channels of the color image. The image features learned using convolutional neural networks are not compact enough.
Disclosure of Invention
In order to solve the problems, the invention provides a color image lossless compression and decompression method based on a quaternion neural network, which aims to better extract quaternion hidden variable characteristic information from a color image by using the quaternion neural network, construct probability distribution of the color image by using the extracted multi-level quaternion hidden variable characteristic information, and realize lossless compression of the color image by using arithmetic coding.
In order to achieve the above purpose, the invention adopts the following technical scheme:
A color image lossless compression and decompression method based on a quaternion neural network comprises the following steps:
Step one, constructing a network model; constructing a lossless compression and decompression network model by using a quaternion neural network, and constructing a loss function for network learning;
Training a network model; constructing a training image data set, inputting image data in the image data set into the lossless compression and decompression network model constructed in the first step in batches, accumulating and calculating the output of a loss function in batches, updating parameters of the lossless compression and decompression network model by using the derivative relation between the loss function and the lossless compression and decompression network model, repeating the training process, and continuously updating the parameters of the lossless compression and decompression network model until the lossless compression and decompression network model converges;
step three, image lossless compression; the image X is converted into a matrix of quaternions, Wherein X r,Xg,Xb is the matrix representation of the red, green, blue channels of the color image,For the quaternion matrix representation of a color image, a, f, c represent 3 imaginary numbers, and the quaternion matrix is used for the color imageInputting the four-element hidden variable sequence into a converged lossless compression and decompression network model, and outputting the four-element hidden variable sequence of the M layer through the operation of the lossless compression and decompression network modelConditional probability distribution of layers Wherein M is a natural number,Generating lossless compression coding data bits of the quaternion hidden variables from the Mth layer to the 1 st layer in sequence by using arithmetic coding, and finally generating lossless compression coding data bits of the image X;
step four, lossless decompression of the image; recovery of quaternion hidden variables for M-th layer using a priori uniform distribution and arithmetic decoding Hidden quaternion variables of M layerInputting to a decoding network, reconstructing to obtain the conditional probability distribution of the M-1 layerRecovery of M-1 layer quaternion hidden variables using arithmetic decodingSequentially recovering gradually to obtain quaternion hidden variable of layer 1Conditional probability distribution of layer 1 of image XFinally, decompression of the image X is achieved.
The beneficial effects achieved by the invention are as follows:
The image compression and decompression method can obtain excellent lossless compression performance of the color image, has good time cost, and has wide application range.
Drawings
Fig. 1 is a flowchart of a color image lossless compression and decompression method based on a quaternion neural network.
Detailed description of the preferred embodiments
The process according to the invention is described in detail below with reference to the accompanying drawings. It should be noted that the described embodiments are for illustrative purposes only and are not limiting of the invention.
The invention provides a color image lossless compression and decompression method based on a quaternion neural network, which has the following working principle: first the input image X is converted into a matrix of quaternionsWherein r, g, b represent the red, green, blue channels of the color image, respectively, X r,Xg,Xb is a matrix representation of the red, green, blue channels of the color image, respectively,For a matrix representation of a quaternion of a color image, a, f, c represent 3 imaginary parts of the quaternion. Wherein the r, g, b channels of the color image respectively form an imaginary matrix of the quaternion matrix. SubsequentlyAnd inputting into a network of M layers, wherein M is a natural number. In the m layer, the input is subjected to quaternion coding network, dimension reduction, quantization and arithmetic coding to obtain hidden variable of quaternionThe encoded output bitstream. SubsequentlyThe quaternion decoding network input to the layer obtains output characteristicsThe probability evaluation module input to the layer obtains the conditional probability distributionAt this time, the combination is obtained in the M-1 layer through quaternion coding network, dimension reduction and quantizationThe pair can be obtained by arithmetic codingThe coded output bit stream can sequentially complete the coding output bit stream of hidden variables of each level. In the first level, can obtainThereby obtaining an encoded output bitstream of image data.
For the data decompression process, becauseThe coding is carried out by adopting even distribution, so that the output bit stream can be directly recovered by arithmetic decoding to obtain corresponding hidden variableSubsequentlyThe quaternion decoding network input to the layer obtains output characteristicsInput into the probability evaluation module of the layer to obtainCan be recovered by arithmetic decodingThe recovery of hidden variables of each level can be completed in turn. Finally in the first level, can be obtainedThereby recovering the image data using the encoded output bitstream of the image data.
Referring to fig. 1, the system describes the steps of implementing the invention as follows:
Step one, constructing a network model; constructing a lossless compression and decompression network model by using a quaternion neural network, and constructing a loss function for network learning;
Training a network model; constructing a training image data set, inputting image data in the image data set into the lossless compression and decompression network model constructed in the first step in batches, accumulating and calculating the output of a loss function in batches, updating parameters of the lossless compression and decompression network model by using the derivative relation between the loss function and the lossless compression and decompression network model, repeating the training process, and continuously updating the parameters of the lossless compression and decompression network model until the lossless compression and decompression network model converges;
step three, image lossless compression; the image X is converted into a matrix of quaternions, Wherein X r,Xg,Xb is the matrix representation of the red, green, blue channels of the color image respectively,For the quaternion matrix representation of a color image, a, f, c represent 3 imaginary numbers, and the quaternion matrix is used for the color imageInputting the four-element hidden variable sequence into a converged lossless compression and decompression network model, and outputting the four-element hidden variable sequence of the M layer through the operation of the lossless compression and decompression network modelConditional probability distribution of layers Wherein M is a natural number,Generating lossless compression coding data bits of the quaternion hidden variables from the Mth layer to the 1 st layer in sequence by using arithmetic coding, and finally generating lossless compression coding data bits of the image X;
step four, lossless decompression of the image; recovery of quaternion hidden variables for M-th layer using a priori uniform distribution and arithmetic decoding Hidden quaternion variables of M layerInputting to a decoding network, reconstructing to obtain the conditional probability distribution of the M-1 layerRecovery of M-1 layer quaternion hidden variables using arithmetic decodingSequentially recovering gradually to obtain quaternion hidden variable of layer 1Conditional probability distribution of layer 1 of image XFinally, decompression of the image X is achieved.
Further, the lossless compression and decompression network model adopts an M-level structure, and the output of the former layer of network is the input of the latter layer of network, so that a cascade structure is constructed; in each hierarchy, the lossless compression and decompression network model comprises a corresponding quaternion neural network encoder, a dimension reduction network, a quantizer, a conditional probability evaluation network and a quaternion neural network decoder; the quaternion neural network encoder is constructed by adopting a quaternion convolution residual error network, and the quaternion convolution residual error network adopts a multi-level depth network structure; the output of the quaternion neural network encoder realizes data dimension reduction and quantization through a dimension reduction network and a quantizer, wherein the dimension reduction network is realized by using a quaternion convolutional neural network with the size of C1*1, wherein C is a natural number; the conditional probability evaluation network is used for realizing probability model parameter evaluation of the characteristic information of each level, and each level of the conditional probability evaluation network is used for evaluating the conditional probability distribution of the quaternion hidden variables of the upper level; the quaternion neural network decoder takes the quaternion hidden variable of the current layer and the output of the quaternion neural network decoder of the upper layer as inputs, and gradually rebuilds the layer characteristics. In the invention, the body of the quaternion neural network encoder is formed by an 8-layer quaternion convolution residual error network, the front end of the quaternion neural network encoder comprises a quaternion convolution neural network with a stride of 2 and a kernel of 5*5, and the channel number is N. The dimension reduction network adopts a quaternion convolution neural network with a kernel size of 1*1, and the channel number is C. The body of the quaternion neural network decoder is formed by an 8-layer quaternion convolution residual error network, and up-sampling is realized by utilizing PixelShuffle.
Further, quantization of the hidden quaternion variable is carried out on four channels of the quaternion by adopting a separation function, wherein the quantization function adopted by each channel of the separation function is shown as a formula (1), a formula (2) is a quantization function adopted during compression, and a formula (3) is a quantization function adopted during quaternion neural network training and is conductive;
Wherein, Z r,zi,zj,zk is the hidden variable of the quaternionIs provided with a plurality of channels for each of the channels,To latent variables for quaternionsWherein Q (z r),Q(zi),Q(zj),Q(zk) is the quantization operation performed on each quaternion channel z r,zi,zj,zk;
Where z is a real variable, g l represents the value of the quantization parameter network, argmin represents taking the index 1 corresponding to g l closest to z, Is the quantized output; the expression of absolute value is calculated;
Wherein, Representing the output of the quantization function, exp representing the exponential operation, β q representing an superparameter that quantizes the degree of softness, L representing the order of quantization, g l,gk representing the value of the quantization parameter network.
Further, maximizing the learned conditional probability distribution of the image X is equivalent to minimizing the quaternion hidden variables of each level and the negative log likelihood function of the image X, and the loss function is the average value of the negative log likelihood functions of the probabilities of the multiple images, where the loss function L s is:
where L s represents the loss function, N represents the number of images of a batch of batch that calculate the loss, Representing network parameters of the quaternion neural network,Representing the conditional probability distribution of the image X,Quaternion hidden variables representing layer iLogarithm of conditional probability distribution of (c).
Further, the quaternion hidden variable of the M layerAdopting uniform distribution without parameters, and directly recovering the quaternion hidden variables of the M layer by arithmetic decoding; hidden variable of the mth layerM is more than or equal to 1 and less than or equal to M-1, the conditional probability distribution of the m+1th layer quaternion neural network decoder takes the output of the m+1th layer quaternion neural network decoder as the input, and the parameter value of the conditional probability distribution function of the m+1th layer quaternion neural network decoder is obtainedAdopts mixed logic distribution;
For the probability distribution of the red, green and blue channels of the image X, a mixed logic is selected, r, g and b are marked as red, green and blue channels of the color image respectively, and the conditional probability distribution of the red, green and blue channels is estimated according to the sequence from r to g to b, wherein the mean value of the green channel g is linearly related to the mean value of the red channel r, and the mean value of the blue channel b is linearly related to the red channel r and the red channel g.
It will be readily appreciated by those skilled in the art that the foregoing description is merely a preferred embodiment of the invention and is not intended to limit the invention, but any modifications, equivalents, improvements or alternatives falling within the spirit and principles of the invention are intended to be included within the scope of the invention.
Claims (5)
1. The color image lossless compression and decompression method based on the quaternion neural network is characterized by comprising the following steps of:
Step one, constructing a network model; constructing a lossless compression and decompression network model by using a quaternion neural network, and constructing a loss function for network learning;
Training a network model; constructing a training image data set, inputting image data in the image data set into the lossless compression and decompression network model constructed in the first step in batches, accumulating and calculating the output of a loss function in batches, updating parameters of the lossless compression and decompression network model by using the derivative relation between the loss function and the lossless compression and decompression network model, repeating the training process, and continuously updating the parameters of the lossless compression and decompression network model until the lossless compression and decompression network model converges;
step three, image lossless compression; the image X is converted into a matrix of quaternions, Wherein X r,Xg,Xb is the matrix representation of the red, green, blue channels of the color image,For the quaternion matrix representation of a color image, a, f, c represent 3 imaginary numbers, and the quaternion matrix is used for the color imageInputting the four-element hidden variable sequence into a converged lossless compression and decompression network model, and outputting the four-element hidden variable sequence of the M layer through the operation of the lossless compression and decompression network modelConditional probability distribution of layers Wherein M is a natural number,Generating lossless compression coding data bits of the quaternion hidden variables from the Mth layer to the 1 st layer in sequence by using arithmetic coding, and finally generating lossless compression coding data bits of the image X;
step four, lossless decompression of the image; recovery of quaternion hidden variables for M-th layer using a priori uniform distribution and arithmetic decoding Hidden quaternion variables of M layerInputting to a decoding network, reconstructing to obtain the conditional probability distribution of the M-1 layerRecovery of M-1 layer quaternion hidden variables using arithmetic decodingSequentially recovering gradually to obtain quaternion hidden variable of layer 1Conditional probability distribution of layer 1 of image XFinally, decompression of the image X is achieved.
2. The method for lossless compression and decompression of color images based on quaternion neural networks according to claim 1, wherein the lossless compression and decompression network model adopts an M-level structure, and the output of the former layer of network is the input of the latter layer of network, so as to construct a cascade structure; in each hierarchy, the lossless compression and decompression network model comprises a corresponding quaternion neural network encoder, a dimension reduction network, a quantizer, a conditional probability evaluation network and a quaternion neural network decoder; the quaternion neural network encoder is constructed by adopting a quaternion convolution residual error network, and the quaternion convolution residual error network adopts a multi-level depth network structure; the output of the quaternion neural network encoder realizes data dimension reduction and quantization through a dimension reduction network and a quantizer, wherein the dimension reduction network is realized by using a quaternion convolutional neural network with the size of C1*1, wherein C is a natural number; the conditional probability evaluation network is used for realizing probability model parameter evaluation of the characteristic information of each level, and each level of the conditional probability evaluation network is used for evaluating the conditional probability distribution of the quaternion hidden variables of the upper level; the quaternion neural network decoder takes the quaternion hidden variable of the current layer and the output of the quaternion neural network decoder of the upper layer as inputs, and gradually rebuilds the layer characteristics.
3. The lossless compression and decompression method for the color image based on the quaternion neural network according to claim 2, wherein quantization of the quaternion hidden variable is carried out on four channels of the quaternion by adopting a separation function respectively, and a quantization function adopted by each channel of the separation function is shown as a formula (1), wherein a formula (2) is a quantization function adopted during compression, and a formula (3) is a quantization function adopted during quaternion neural network training and is conductive;
Wherein, Z r,zi,zj,zk is the hidden variable of the quaternionIs provided with a plurality of channels for each of the channels,To latent variables for quaternionsWherein Q (z r),Q(zi),Q(zj),Q(zk) is the quantization operation performed on each quaternion channel z r,zi,zj,zk;
where z is a real variable, g l represents the value of the quantization parameter network, argmin represents the index l corresponding to g l closest to z, Is the quantized output; the expression of absolute value is calculated;
Wherein, Representing the output of the quantization function, exp representing the exponential operation, β q representing an superparameter that quantizes the degree of softness, L representing the order of quantization, g l,gk representing the value of the quantization parameter network.
4. The method for lossless compression and decompression of color images based on quaternion neural network according to claim 1, wherein maximizing the learned conditional probability distribution of image X is equivalent to minimizing the quaternion hidden variables of each level and the negative log likelihood function of image X, and the loss function is the average value of the negative log likelihood functions of the conditional probability distribution of a plurality of images, wherein the loss function L s is:
where L s represents the loss function, N represents the number of images of a batch of batch that calculate the loss, Representing network parameters of the quaternion neural network,Representing the conditional probability distribution of the image X,Quaternion hidden variables representing layer iLogarithm of conditional probability distribution of (c).
5. The method for lossless compression and decompression of color images based on quaternion neural network according to claim 1, wherein the quaternion hidden variable of the Mth layer is characterized in thatAdopting uniform distribution without parameters, and directly recovering the quaternion hidden variables of the M layer by arithmetic decoding; hidden variable of the mth layerM is more than or equal to 1 and less than or equal to M-1, the conditional probability distribution of the m+1th layer quaternion neural network decoder takes the output of the m+1th layer quaternion neural network decoder as the input, and the parameter value of the conditional probability distribution function of the m+1th layer quaternion neural network decoder is obtainedAdopts mixed logic distribution;
For the probability distribution of the red, green and blue channels of the image X, a mixed logic is selected, r, g and b are marked as red, green and blue channels of the color image respectively, and the conditional probability distribution of the red, green and blue channels is estimated according to the sequence from r to g to b, wherein the mean value of the green channel g is linearly related to the mean value of the red channel r, and the mean value of the blue channel b is linearly related to the red channel r and the red channel g.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410006795.0A CN117893624B (en) | 2024-01-03 | 2024-01-03 | Color image lossless compression and decompression method based on quaternion neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410006795.0A CN117893624B (en) | 2024-01-03 | 2024-01-03 | Color image lossless compression and decompression method based on quaternion neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117893624A CN117893624A (en) | 2024-04-16 |
| CN117893624B true CN117893624B (en) | 2024-07-12 |
Family
ID=90640436
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410006795.0A Active CN117893624B (en) | 2024-01-03 | 2024-01-03 | Color image lossless compression and decompression method based on quaternion neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117893624B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107945154A (en) * | 2017-11-10 | 2018-04-20 | 西安电子科技大学 | Color image quality evaluation method based on quaternary number discrete cosine transform |
| CN113034628A (en) * | 2021-04-29 | 2021-06-25 | 南京信息工程大学 | Color image JPEG2000 recompression detection method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR102621827B1 (en) * | 2022-06-22 | 2024-01-08 | 성균관대학교산학협력단 | Neural network-based image compression method |
-
2024
- 2024-01-03 CN CN202410006795.0A patent/CN117893624B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107945154A (en) * | 2017-11-10 | 2018-04-20 | 西安电子科技大学 | Color image quality evaluation method based on quaternary number discrete cosine transform |
| CN113034628A (en) * | 2021-04-29 | 2021-06-25 | 南京信息工程大学 | Color image JPEG2000 recompression detection method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117893624A (en) | 2024-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109996071B (en) | Variable code rate image coding and decoding system and method based on deep learning | |
| CN110248190B (en) | Multilayer residual coefficient image coding method based on compressed sensing | |
| US11544606B2 (en) | Machine learning based video compression | |
| CN111147862B (en) | End-to-end image compression method based on target coding | |
| Zhang et al. | Learned scalable image compression with bidirectional context disentanglement network | |
| CN106658003A (en) | quantization method of dictionary learning-based image compression system | |
| CN110753225A (en) | Video compression method and device and terminal equipment | |
| CN111246206A (en) | Optical flow information compression method and device based on self-encoder | |
| Li et al. | Multiple description coding based on convolutional auto-encoder | |
| Sadeeq et al. | Image compression using neural networks: a review | |
| Akbari et al. | Learned multi-resolution variable-rate image compression with octave-based residual blocks | |
| CN117893624B (en) | Color image lossless compression and decompression method based on quaternion neural network | |
| Khmelevskiy et al. | Model of Transformation of the Alphabet of the Encoded Data as a Tool to Provide the Necessary Level of Video Image Qualityi in Aeromonitoring Systems. | |
| Kumar et al. | Vector quantization with codebook and index compression | |
| CN116109714A (en) | Data coding storage method and system based on neural network | |
| Huang et al. | Multi-channel multi-loss deep learning based compression model for color images | |
| CN115471576A (en) | Point cloud lossless compression method and device based on deep learning | |
| CN112437300B (en) | Distributed video coding method based on self-adaptive interval overlapping factor | |
| Khan | An implementation of vector quantization using the genetic algorithm approach | |
| CN114422795A (en) | Face video coding method, decoding method and device | |
| EP4022518A1 (en) | Neural network model compression with block partitioning | |
| CN112950729A (en) | Image compression method based on self-encoder and entropy coding | |
| Bachard et al. | CoCliCo: Extremely low bitrate image compression based on CLIP semantic and tiny color map | |
| CN109302614B (en) | Video compression method based on third-order tensor self-coding network | |
| CN114222124B (en) | Encoding and decoding method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |