CN116542981A - Quality assessment method for reference-point-free cloud - Google Patents
Quality assessment method for reference-point-free cloud Download PDFInfo
- Publication number
- CN116542981A CN116542981A CN202310824543.4A CN202310824543A CN116542981A CN 116542981 A CN116542981 A CN 116542981A CN 202310824543 A CN202310824543 A CN 202310824543A CN 116542981 A CN116542981 A CN 116542981A
- Authority
- CN
- China
- Prior art keywords
- point
- representing
- feature
- quality
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000001303 quality assessment method Methods 0.000 title claims abstract description 26
- 238000011176 pooling Methods 0.000 claims abstract description 39
- 238000000605 extraction Methods 0.000 claims abstract description 37
- 238000010586 diagram Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000012935 Averaging Methods 0.000 claims abstract description 7
- 238000000034 method Methods 0.000 claims description 48
- 238000005070 sampling Methods 0.000 claims description 22
- 230000004927 fusion Effects 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 13
- 238000004422 calculation algorithm Methods 0.000 claims description 12
- 238000010276 construction Methods 0.000 claims description 7
- 230000003252 repetitive effect Effects 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 abstract description 10
- 238000013135 deep learning Methods 0.000 abstract description 6
- 230000000007 visual effect Effects 0.000 abstract description 6
- 238000013441 quality evaluation Methods 0.000 description 13
- 230000008569 process Effects 0.000 description 8
- 239000000284 extract Substances 0.000 description 7
- 238000012549 training Methods 0.000 description 6
- 238000013459 approach Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 4
- 230000006835 compression Effects 0.000 description 3
- 238000007906 compression Methods 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001010 compromised effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001627 detrimental effect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000004438 eyesight Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 210000001328 optic nerve Anatomy 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The invention relates to the technical field of deep learning, and discloses a quality assessment method of reference point-free cloud, which is based on a neural network consisting of a 3D space KNN graph module, a geometric and color feature extraction layer, a pooling layer and a quality regression module. Firstly, constructing a 3DKNN diagram of a local space for each point in a point cloud; secondly, extracting local space geometric and color features of the points by utilizing a geometric and color feature extraction layer; then, further processing the processed product by using a pooling layer so as to extract the characteristics from thick to thin; and finally, carrying out regression fitting on the characteristics by using a quality regression module, averaging the quality score of one point cloud, and finally outputting the quality score of the point cloud as the final score of the point cloud. According to the quality assessment method for the point cloud without reference, provided by the invention, the color and geometric information graph neural network is fused to carry out quality assessment for the point cloud without reference, and the visual quality of the point cloud is assessed from the local space angle of the point cloud.
Description
Technical Field
The invention relates to the technical field of deep learning, in particular to a quality assessment method for reference-point-free cloud.
Background
In applications such as augmented reality, virtual reality, and autopilot, point clouds are the most widely used form of data. However, since a point cloud object contains hundreds of thousands or even millions of points, it is necessarily difficult to compress and transmit it without loss. For this reason, algorithms for lossy compression are introduced, and the quality of the point cloud is compromised. Therefore, in order to optimize the point cloud compression algorithm, it is necessary to perform quality assessment on the point cloud. Point cloud quality evaluation (point cloud quality assessment, PCQA) algorithms are generally classified into three categories according to the presence or absence of a reference point cloud: full reference assessment (FR-PCQA), half reference assessment (RR-PCQA) and no reference assessment (NR-PCQA). Since reference point clouds do not exist in many practical applications, in recent years, non-reference point cloud quality evaluation has been studied more widely.
The construction of the point cloud dataset is often time consuming and laborious, as active participation by humans is required to subjectively score the distorted point cloud, resulting in a final Mean Opinion Score (MOS). This subjective scoring approach is detrimental to real-time system applications such as compression and transmission of point clouds. For this reason, an objective point cloud quality evaluation algorithm needs to be established.
The point cloud quality evaluation methods without reference are mainly divided into two types: one is to extract the quality features of the point cloud by a manual design method, for example, 3D-NSS extracts the curvature, plane, color and other features of the point cloud by manual design, and the manually extracted features obtain the final quality fraction by a regression module. The manually extracted features, namely the manually designed features, are directly designed, and the features are not sensitive to the features according to the characteristics imitating human vision, so that the extracted features often have specific physical meanings. At present, the better manually extracted point cloud features include curvature, euclidean distance, color gradient and the like, which are designed according to the characteristics of the human visual system. The defects are that: (1) Careful design is required according to the characteristics of data, and subjective consciousness of people still exists although the support of the optic nerve theory exists; (2) Often, it depends on specific databases, i.e., the features of the design perform well only for some databases and not for others; (3) The process of extracting the characteristics consumes long time and complex steps, which is not beneficial to the extraction of the point cloud quality characteristics.
The other is to convert the 3D point cloud into a 2D image by a projection mode, research the quality evaluation of the point cloud by utilizing the existing mature image quality evaluation (image quality assessment, IQA) algorithm, for example, the PQA-Net projects a point cloud object to 6 two-dimensional planes (up, down, front, back, left and right) along 3 orthogonal directions, and extract the quality characteristics of the point cloud by utilizing a standard convolutional neural network (convolutional neural network); IT-PCQA is also a method of converting an irregular point cloud into a regular 2D image, and then evaluating the quality of the point cloud through a Domain Adaptation (DA) algorithm. While such projection-based approaches can address irregular data forms of the point cloud, such a transformation approach is suboptimal because it inevitably loses internal geometry and detailed spatial texture information. Thus, the performance obtained in this way is not as good.
The existing reference point-free cloud quality evaluation methods such as a mode based on manual feature extraction and a mode based on multi-view projection have respective limitations. For example, 3D-NSS based on manually extracted features is not an end-to-end deep learning approach, cannot fully mine the inherent structural features of the point cloud, and the manual method is time consuming and often depends on a specific database; methods based on multi-view projection, such as PQA and IT-PCQA, lose detail information of many points during projection, thereby losing part of geometry and spatial texture information.
Disclosure of Invention
The invention provides a quality assessment method of point cloud without reference, which is used for carrying out quality assessment of the point cloud without reference by fusing a color and geometric information graph neural network and evaluating the visual quality of the point cloud from the local space angle of the point cloud.
The invention provides a quality assessment method of reference-point-free cloud, which is based on a neural network consisting of a 3D space KNN graph module, a geometric and color feature extraction layer, a pooling layer and a quality regression module, and specifically comprises the following steps:
obtaining K sampling points of an original point cloud through the furthest point sampling, respectively taking the K sampling points as centers, searching N adjacent points through a KNN algorithm, and finally dividing the original point cloud into K patches, wherein each patch comprises N points;
extracting one of the patches as a target patch, inputting the target patch into a 3D space KNN graph module, and constructing a KNN graph of a 3D space for each point in the target patch;
inputting each point in the target patch into a geometric and color feature extraction layer to extract geometric and color features through extended 3D convolution, gradually expanding the repetitive files in a layered hierarchical structure, and obtaining quality representation features after multi-layer convolution operation;
inputting the quality representation features into a pooling layer for feature processing to obtain coarse-to-fine features;
inputting the coarse-fine characteristics into a quality regression module consisting of a full connection layer and a BN layer to carry out quality regression so as to obtain the quality fraction of the target patch;
traversing the K patches in the original point cloud to obtain the quality scores of the K patches, and averaging the quality scores of the K patches to serve as the final score of the original point cloud.
Further, the step of extracting one of the patches as a target patch, inputting the target patch into a 3D space KNN graph module to construct a KNN graph of the 3D space for each point in the target patch includes:
extracting one of the patches as a target patch, and representing the target patch asThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Point cloud patch representing N points, ++>Representing an nth point in the target patch;
will beExpressed as a related D-dimensional feature vector to describe each point in the target patch +.>Is characterized by ++in KNN diagram of 3D space>Representing two points +.>To->Direction vector between>Representation dot->Feature vector of>Representation dot->Is defined in the drawing) is provided.
Further, the step of inputting each point in the target patch into the geometric and color feature extraction layer to extract geometric and color features through extended 3D convolution, and gradually expanding the repeated features in a layered hierarchical structure, and obtaining quality representation features after multi-layer convolution operation includes:
performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract color features;
performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features;
and fusing the color feature and the geometric feature as a new feature, and performing a third 3D convolution operation on the new feature to obtain a quality representation feature.
Further, in the step of performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract the color feature, performing the first 3D convolution operation includes:
wherein->A first 3D convolution operation representing extraction of color features,/->Represents the center point +.>Color-related features of (2); />Color-related features representing other neighboring points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Color characteristics of->Representing neighbor Point->Is a color feature of (a); />And->Respectively represent for extracting feature->And->Is a learning weight parameter of (a);representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
Further, in the step of performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features, performing the second 3D convolution operation includes:
wherein->A second 3D convolution operation representing the extracted geometric features,/->Represents the center point +.>Is a geometric correlation feature of (2); />Representing geometrically related features of other neighboring points;
,/>wherein->Representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Geometric coordinates of>Representing neighbor pointsIs a geometric correlation feature of (2); />And->Representation for extraction of->And->A learnable weight parameter of the feature; />Representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
Further, the step of fusing the color feature and the geometric feature as a new feature, and performing a third 3D convolution operation on the new feature to obtain a quality representation feature, where the third 3D convolution operation includes:
wherein->A second 3D convolution operation representing the fusion feature,/->Represents the center point +.>Is a fusion feature of (2); />Representing fusion characteristics of other adjacent points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>New feature of fusion of geometrical coordinates and color properties, < ->Representing neighbor Point->New features of fusion of geometric coordinates and color attributes; />And->Representation of extraction->And->A learnable weight parameter of the feature; />Represents a direction vector, the expression is +.>；Representing a learnable direction vector.
Further, the step of inputting the quality representation feature to a pooling layer for feature processing to obtain a coarse-to-fine feature includes:
the pooling layer adopts the maximum pooling of each point in the KNN diagram of the 3D space by each channel to obtain pooling characteristics of each point, and adopts a random sampling mode to downsample each point, and the formula is as follows:
wherein->Representing pooling operations, +.>Representing the point of input->Representing the input point features; />Point representing output +.>Representing the point characteristics of the output.
The invention also provides a quality assessment device without reference point cloud, which is based on a neural network consisting of a 3D space KNN graph module, a geometric and color feature extraction layer, a pooling layer and a quality regression module, and specifically comprises the following steps:
the searching unit is used for obtaining K sampling points of an original point cloud through the furthest point sampling, searching N adjacent points through a KNN algorithm by taking the K sampling points as centers respectively, and finally dividing the original point cloud into K patches, wherein each patch comprises N points;
the construction unit is used for extracting one of the patches as a target patch, inputting the target patch into the 3D space KNN graph module, and constructing a KNN graph of the 3D space for each point in the target patch; the construction unit 2 specifically includes:
an extraction subunit, configured to extract one of the patches as a target patch, and represent the target patch asThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Point cloud patch representing N points, ++>Representing an nth point in the target patch;
a description subunit for convertingExpressed as a related D-dimensional feature vector to describe each point in the target patch +.>Is characterized by ++in KNN diagram of 3D space>Representing two points +.>To->Direction vector between>Representation dot->Feature vector of>Representation dot->Is defined by the spatial coordinates of (a);
the extraction unit is used for inputting each point in the target patch into the geometric and color feature extraction layer, extracting geometric and color features through extended 3D convolution, gradually expanding the repeated filed in a layered hierarchical structure, and obtaining quality representation features after multi-layer convolution operation; the extraction unit 3 specifically includes:
a first execution subunit for performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract color features;
a second execution subunit for performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features;
a third execution subunit, configured to fuse the color feature and the geometric feature as a new feature, and perform a third 3D convolution operation on the new feature to obtain a quality representation feature;
the processing unit is used for inputting the quality representation characteristics into the pooling layer for characteristic processing to obtain coarse-to-fine characteristics;
the regression unit is used for inputting the characteristics from the thick to the thin into a quality regression module consisting of a full connection layer and a BN layer to carry out quality regression so as to obtain the quality fraction of the target patch;
the traversing unit is used for traversing the K patches in the original point cloud to obtain the quality scores of the K patches, and averaging the quality scores of the K patches to be used as the final score of the original point cloud.
The invention also provides a computer device comprising a memory storing a computer program and a processor implementing the steps of the above method when executing the computer program.
The invention also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method.
The beneficial effects of the invention are as follows:
the non-reference point cloud quality evaluation method provided by the invention is based on the local space geometry combined color network, and can remarkably improve the performance of point cloud quality evaluation. The invention provides a local space geometry and color extraction network, which can extract the features of geometry and color in local space by using extended 3D convolution and capture fine-grained (fine-grained) structural information; the invention relates to an end-to-end deep learning training mode, which can rapidly and effectively extract the characteristics of a distorted point cloud.
Drawings
FIG. 1 is a flow chart of a method according to an embodiment of the invention.
Fig. 2 is a schematic diagram of a neural network according to the present invention.
Fig. 3 is a schematic diagram of a 3D space KNN diagram according to the present invention.
Fig. 4 is a schematic diagram of an apparatus structure according to an embodiment of the invention.
Fig. 5 is a schematic diagram illustrating an internal structure of a computer device according to an embodiment of the invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
In image quality evaluation (IQA), since the patch of an image has a regular grid structure, various convolution kernels in CNN can be used to extract image quality evaluation features (such as color gradient, contrast, etc.). However, for points in the point cloud to be scattered in 3D space, there is no regular structured information.
According to the visual perception principle of the HVS, the eye cannot directly perceive the visual properties of a single point, but rather perceives a local neighbor structure filtered by the low-pass function of the eye optics. The graph represents features that can aggregate local points in a way that implicitly embeds local neighbor connections. Although the points are unstructured according to the visual mechanism principle of the HVS, the visual mechanism has good relation with the graph representation by implicitly applying a point spread function to connect local neighbor points for geometric and color synthesis, and the local connection mode is not displayed between the points.
1-2, a quality assessment method of a non-reference point cloud is based on a neural network composed of a 3D space KNN graph module, a geometric and color feature extraction layer, a pooling layer and a quality regression module, wherein the 3D space KNN graph module is mainly used for constructing a KNN graph of a local space region, the geometric and color feature extraction layer is mainly used for extracting local geometric and color features of a distorted point cloud, the pooling layer is mainly used for meeting replacement invariance of the point cloud, compressing the features and reducing the number of network parameters, and the quality regression module is mainly used for carrying out quality regression on the local features and global features to finally obtain quality scores of the point cloud. The invention relates to an end-to-end model training mode, which comprises the steps of firstly, constructing a 3DKNN graph of a local space for each point in a point cloud; secondly, extracting local space geometric and color features of the points by utilizing a geometric and color feature extraction layer; further processing it with a pooling layer to extract coarse-to-fine (coarse-to-fine) features; and finally, carrying out regression fitting on the characteristics by utilizing a quality regression module, and finally outputting the quality score of the point cloud.
The invention is realized on a pytorch experiment platform, and the method specifically comprises the following steps:
s1, obtaining K sampling points of an original point cloud through furthest point sampling, respectively taking the K sampling points as centers, searching N adjacent points through a KNN algorithm, and finally dividing the original point cloud into K patches, wherein each patch comprises N points; the purpose of increasing the size of the database is further achieved, so that the requirement of the deep neural network (deep neural network, DNN) for having a large amount of data in the training process is met.
S2, extracting one of the patches as a target patch, and inputting the target patch into a 3D space KNN graph module to construct a KNN graph of a 3D space for each point in the target patch;
in order to be able to perform similar convolution operations in a 3D point cloud reference-free quality assessment, the convolution formula in 2D is generalized to a 3D point cloud quality assessment. However, since the point cloud is scattered in 3D space, it is an unstructured data structure, to this end, the graph in the 2D plane is extended into 3D space, and a 3D space KNN graph is built for each point in the point cloud, similar to the received filter at the 2D convolution center point, as shown in fig. 3.
In the 3D space KNN graph module, a 3D space KNN graph is built mainly for each point in the point cloud patch. In particular, because the amount of point cloud data to be processed is very large (e.g., a point cloud object typically contains hundreds of thousands or even millions of points) and computer memory resources are limited (e.g., memory, computing power, etc.), the point cloud is divided into many patches that are used as inputs to the network.
The step S2 specifically comprises the following steps:
s21, extracting one of the patches as a target patch, and representing the target patch asThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Point cloud patch representing N points, ++>Representing an nth point in the target patch; that is, one contains NThe point cloud patch of points can be expressed as +.>Wherein->Represents the nth point in the point cloud patch.
S22, in order to describe the characteristics of each point in the point cloud patch, the following will be performedExpressed as a related D-dimensional feature vector to describe each point in the target patch +.>In the KNN diagram of the 3D space, as shown in FIG. 3, +.>Representing two points +.>To->Direction vector between>Representation dot->Feature vector of>Representation dot->Spatial coordinates of>Points representing neighbors.
S3, inputting each point in the target patch into a geometric and color feature extraction layer to extract geometric and color features through extended 3D convolution, gradually expanding the repeated files in a layered hierarchical structure, and obtaining quality representation features after multi-layer convolution operation;
after constructing the repetitive filter of the 3D space, convolving and expanding the 2D image into a 3D point cloud, and executing a first 3D convolution operation on each point in a KNN diagram of the 3D space to extract color features; performing a first 3D convolution operation includes:
wherein->A first 3D convolution operation representing extraction of color features,/->Represents the center point +.>Color-related features of (2); />Color-related features representing other neighboring points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Color characteristics of->Representing neighbor Point->Is a color feature of (a); />And->Respectively represent for extracting feature->And->Is a learning weight parameter of (a);representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
As shown in fig. 2, in the constructed 3D space KNN diagram, K neighboring points of the center point are distributed on a curved surface, which can be approximated as a plane in a small local area, and thus the defined convolution formula can be approximated as an extension of the convolution in the 2D plane in the 3D space. However, it is considered that the distortion in the 3D point cloud includes not only color distortion but also geometric distortion. For this, geometric features are extracted using a 3D convolution formula; that is, a second 3D convolution operation is performed on each point in the KNN graph of the 3D space to extract geometric features; performing a second 3D convolution operation includes:
wherein->A second 3D convolution operation representing the extracted geometric features,/->Represents the center point +.>Is a geometric correlation feature of (2); />Representing geometrically related features of other neighboring points;
,/>wherein->Representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Geometric coordinates of>Representing neighbor pointsIs a geometric correlation feature of (2); />And->Representation for extraction of->And->A learnable weight parameter of the feature; />Representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
Fusing the geometric and color features together as a new feature, and then performing convolution operation with 3D convolution, wherein the fused feature has better effect than a single feature; fusing the color feature and the geometric feature as a new feature, and performing a third 3D convolution operation on the new feature to obtain a quality representation feature; the third 3D convolution operation includes:
wherein->A second 3D convolution operation representing the fusion feature,/->Represents the center point +.>Is a fusion feature of (2); />Representing fusion characteristics of other adjacent points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>New feature of fusion of geometrical coordinates and color properties, < ->Representing neighbor Point->New features of fusion of geometric coordinates and color attributes; />And->Representation of extraction->And->A learnable weight parameter of the feature; />Represents a direction vector, the expression is +.>；Representing a learnable direction vector.
S4, inputting the quality representation features into a pooling layer for feature processing to obtain coarse-to-fine features, which are more identifying features; to extract coarse-to-fine features, a pooling layer is added after the convolution layer to extract more efficient feature representations while reducing training parameters of the network.
The pooling operation summarizes the main reactions within each scale for the subsequent high-dimensional feature processing to extract coarse-to-fine (coarse-to-fine) features. The pooling layer adopts maximum pooling of each point in the 3D space KNN diagram by channel-wise (channel-wise), so as to obtain pooling characteristics of each point, and downsamples each point in a random sampling mode, so that the number of points is reduced, and the process (formula) is expressed as follows:
wherein->Representing pooling operations, +.>Representing the point of input->Representing the input point features; />Point representing output +.>Representing the point characteristics of the output. The pooling layer can learn multi-scale 3D point cloud characteristics and enable learning and calculation to be more efficient, which is a key factor of 3D point cloud deep learning. Furthermore, by learning the directional information in the 3D spatial KNN map, in combination with the pooling mechanism, displacement invariance is demonstrated.
S5, inputting the characteristics from the thick to the thin into a quality regression module composed of a full connection layer and a BN layer to carry out quality regression, so as to obtain the quality fraction of the target patch; the quality regression module consists of a full connection layer and a BN (Batch Normalization) layer and is mainly used for converting the geometric and color characteristics extracted before into final quality scores.
S6, traversing the K patches in the original point cloud to obtain the quality scores of the K patches, and averaging the quality scores of the K patches to serve as the final score of the original point cloud.
The invention provides a local space geometry combined color network structure, which mainly comprises a 3D space KNN graph module, a geometry and color feature extraction layer, a pooling layer and a quality regression module. The 3D space KNN graph module is mainly used for constructing a KNN graph of a local space region, the geometric and color feature extraction layer is mainly used for extracting local geometric and color features of distorted point clouds, the pooling layer is mainly used for meeting the replacement invariance of the point clouds, compressing the features and reducing the number of network parameters, and the quality regression module is mainly used for carrying out quality regression on the local features and the global features to finally obtain the quality scores of the point clouds.
The invention provides an end-to-end model training mode. Specifically, a 3DKNN diagram of a local space is built for each point in the point cloud; extracting local space geometric and color features of the points by using the geometric and color feature extraction layer; further processing it with a pooling layer to extract coarse-to-fine (coarse-to-fine) features; and finally processing the characteristics by using a quality regression module, and finally outputting the quality score of the point cloud.
The non-reference point cloud quality evaluation method provided by the invention is based on the local space geometry combined color network, and can remarkably improve the performance of point cloud quality evaluation. The invention provides a local space geometry and color extraction network, which can extract the features of geometry and color in local space by using extended 3D convolution and capture fine-grained (fine-grained) structural information; the invention relates to an end-to-end deep learning training mode, which can rapidly and effectively extract the characteristics of a distorted point cloud.
As shown in fig. 4, the present invention further provides a quality assessment device without reference point cloud, based on a neural network composed of a 3D space KNN graph module, a geometric and color feature extraction layer, a pooling layer and a quality regression module, where the device specifically includes:
the searching unit 1 is used for obtaining K sampling points of an original point cloud through the farthest point sampling, searching N adjacent points through a KNN algorithm by taking the K sampling points as centers respectively, and finally dividing the original point cloud into K patches, wherein each patch comprises N points;
a construction unit 2, configured to extract one of the patches as a target patch, and input the target patch into a 3D space KNN graph module, so as to construct a KNN graph of the 3D space for each point in the target patch;
the extracting unit 3 is configured to input each point in the target patch into a geometric and color feature extracting layer, extract geometric and color features through extended 3D convolution, gradually extend the repeated features in a layered hierarchical structure, and obtain quality representation features after multi-layer convolution operation;
the processing unit 4 is used for inputting the quality representation characteristics into a pooling layer for characteristic processing to obtain coarse-to-fine characteristics;
the regression unit 5 is used for inputting the characteristics from the thick to the thin into a quality regression module consisting of a full connection layer and a BN layer to carry out quality regression so as to obtain the quality fraction of the target patch;
and the traversing unit 6 is used for traversing the K patches in the original point cloud to obtain the quality scores of the K patches, and averaging the quality scores of the K patches to be used as the final score of the original point cloud.
In one embodiment, the construction unit 2 comprises:
an extraction subunit, configured to extract one of the patches as a target patch, and represent the target patch asThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Point cloud patch representing N points, ++>Representing an nth point in the target patch;
descriptor sheetA member for connectingExpressed as a related D-dimensional feature vector to describe each point in the target patch +.>Is characterized by ++in KNN diagram of 3D space>Representing two points +.>To->Direction vector between>Representation dot->Feature vector of>Representation dot->Is defined in the drawing) is provided.
In one embodiment, the extraction unit 3 includes:
a first execution subunit for performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract color features;
a second execution subunit for performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features;
and the third execution subunit is used for fusing the color feature and the geometric feature as a new feature, and performing a third 3D convolution operation on the new feature to obtain a quality representation feature.
In one embodiment, in the first execution subunit, performing the first 3D convolution operation includes:
wherein->A first 3D convolution operation representing extraction of color features,/->Represents the center point +.>Color-related features of (2); />Color-related features representing other neighboring points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Color characteristics of->Representing neighbor Point->Is a color feature of (a); />And->Respectively represent for extracting feature->And->Is a learning weight parameter of (a);representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
In one embodiment, in the second execution subunit, performing the second 3D convolution operation includes:
wherein->A second 3D convolution operation representing the extracted geometric features,/->Represents the center point +.>Is a geometric correlation feature of (2); />Representing geometrically related features of other neighboring points;
,/>wherein->Representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Geometric coordinates of>Representing neighbor pointsIs a geometric correlation feature of (2); />And->Representation for extraction of->And->A learnable weight parameter of the feature; />Representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing learnableIs a direction vector of (a).
In one embodiment, in the third execution subunit, the third 3D convolution operation includes:
wherein->A second 3D convolution operation representing the fusion feature,/->Represents the center point +.>Is a fusion feature of (2); />Representing fusion characteristics of other adjacent points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>New feature of fusion of geometrical coordinates and color properties, < ->Representing neighbor Point->New features of fusion of geometric coordinates and color attributes; />And->Representation of extraction->And->A learnable weight parameter of the feature; />Represents a direction vector, the expression is +.>；Representing a learnable direction vector.
In one embodiment, the processing unit 4 comprises:
the pooling layer adopts the maximum pooling of each point in the KNN diagram of the 3D space by each channel to obtain pooling characteristics of each point, and adopts a random sampling mode to downsample each point, and the formula is as follows:
wherein->Representing pooling operations, +.>Representing the point of input->Representing the input point features; />Point representing output +.>Representing the point characteristics of the output.
The above units and sub-units are all configured to correspondingly execute each step in the quality assessment method of the reference-point-free cloud, and specific implementation manners thereof are described with reference to the above method embodiments and are not described herein again.
As shown in fig. 5, the present invention also provides a computer device, which may be a server, and the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store all data required for the process of the quality assessment method without reference point clouds. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program, when executed by a processor, implements a method of quality assessment for a point-of-reference-free cloud.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of a portion of the architecture in connection with the present application and is not intended to limit the computer device to which the present application is applied.
An embodiment of the present application further provides a computer readable storage medium, on which a computer program is stored, where the computer program when executed by a processor implements any one of the quality assessment methods without reference point clouds described above.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by hardware associated with a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium provided herein and used in embodiments may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual speed data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, apparatus, article or method that comprises the element.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the invention, and all equivalent structures or equivalent processes using the descriptions and drawings of the present invention or directly or indirectly applied to other related technical fields are included in the scope of the invention.
Claims (8)
1. The quality assessment method without the reference point cloud is characterized by comprising the following steps of:
obtaining K sampling points of an original point cloud through the furthest point sampling, respectively taking the K sampling points as centers, searching N adjacent points through a KNN algorithm, and finally dividing the original point cloud into K patches, wherein each patch comprises N points;
extracting one of the patches as a target patch, inputting the target patch into a 3D space KNN graph module, and constructing a KNN graph of a 3D space for each point in the target patch; the method specifically comprises the following steps:
extracting one of the patches as a target patch, and representing the target patch asThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Point cloud patch representing N points, ++>Representing an nth point in the target patch;
will beExpressed as a related D-dimensional feature vector to describe each point in the target patch +.>Is characterized by ++in KNN diagram of 3D space>Representing two points +.>To->Direction vector between>Representation dot->Feature vector of>Representation dot->Is defined by the spatial coordinates of (a);
inputting each point in the target patch into a geometric and color feature extraction layer to extract geometric and color features through extended 3D convolution, gradually expanding the repetitive files in a layered hierarchical structure, and obtaining quality representation features after multi-layer convolution operation; the method specifically comprises the following steps:
performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract color features;
performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features;
fusing the color feature and the geometric feature as a new feature, and performing a third 3D convolution operation on the new feature to obtain a quality representation feature;
inputting the quality representation features into a pooling layer for feature processing to obtain coarse-to-fine features;
inputting the coarse-fine characteristics into a quality regression module consisting of a full connection layer and a BN layer to carry out quality regression so as to obtain the quality fraction of the target patch;
traversing the K patches in the original point cloud to obtain the quality scores of the K patches, and averaging the quality scores of the K patches to serve as the final score of the original point cloud.
2. The method for quality assessment of a reference-point-free cloud of claim 1, wherein in the step of performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract color features, performing the first 3D convolution operation comprises:
wherein->A first 3D convolution operation representing extraction of color features,/->Represents the center point +.>Color-related features of (2); />Color-related features representing other neighboring points;
,/>wherein->Representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Color characteristics of->Representing neighbor Point->Is a color feature of (a); />And->Respectively represent for extracting feature->And->Is a learning weight parameter of (a); />Representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
3. The method for quality assessment of a point-free cloud as claimed in claim 2, wherein the step of performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features, the performing the second 3D convolution operation comprises:
wherein->A second 3D convolution operation representing the extracted geometric features,/->Represents the center point +.>Is a geometric correlation feature of (2); />Representing geometrically related features of other neighboring points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>Geometric coordinates of>Representing neighbor Point->Is a geometric correlation feature of (2); />And->Representation for extraction of->And->A learnable weight parameter of the feature; />Representing two points +.>To->The direction vector between the two is expressed as +.>；/>Representing a learnable direction vector.
4. The method for evaluating quality of reference-point-free cloud as claimed in claim 3, wherein said fusing said color feature with said geometric feature as a new feature, and performing a third 3D convolution operation on said new feature to obtain a quality representation feature, wherein said third 3D convolution operation comprises:
wherein->A second 3D convolution operation representing the fusion feature,/->Represents the center point +.>Is a fusion feature of (2); />Representing fusion characteristics of other adjacent points;
,/>wherein, the method comprises the steps of, wherein,representing an inner product operation; at the first layer of the network,/a>Represents the center point +.>New feature of fusion of geometrical coordinates and color properties, < ->Representing neighbor Point->New features of fusion of geometric coordinates and color attributes; />And->Representation of extraction->And->A learnable weight parameter of the feature; />Represents a direction vector, the expression is +.>；Representing a learnable direction vector.
5. The quality assessment method without reference point cloud according to claim 1, wherein the step of inputting the quality representation feature to a pooling layer for feature processing to obtain a coarse-to-fine feature comprises:
the pooling layer adopts the maximum pooling of each point in the KNN diagram of the 3D space by each channel to obtain pooling characteristics of each point, and adopts a random sampling mode to downsample each point, and the formula is as follows:
wherein->Representing pooling operations, +.>Representing the point of input->Representing the input point features; />Point representing output +.>Representing the point characteristics of the output.
6. A quality assessment device without reference point cloud, characterized in that the device specifically comprises:
the searching unit is used for obtaining K sampling points of an original point cloud through the furthest point sampling, searching N adjacent points through a KNN algorithm by taking the K sampling points as centers respectively, and finally dividing the original point cloud into K patches, wherein each patch comprises N points;
the construction unit is used for extracting one of the patches as a target patch, inputting the target patch into the 3D space KNN graph module, and constructing a KNN graph of the 3D space for each point in the target patch; the construction unit 2 specifically includes:
an extraction subunit, configured to extract one of the patches as a target patch, and represent the target patch asThe method comprises the steps of carrying out a first treatment on the surface of the Wherein (1)>Point cloud patch representing N points, ++>Representing an nth point in the target patch;
a description subunit for convertingExpressed as a related D-dimensional feature vector to describe each point in the target patch +.>Is characterized by ++in KNN diagram of 3D space>Representing two points +.>To->The direction vector between the two directions is defined,representation dot->Feature vector of>Representation dot->Is defined by the spatial coordinates of (a);
the extraction unit is used for inputting each point in the target patch into the geometric and color feature extraction layer, extracting geometric and color features through extended 3D convolution, gradually expanding the repeated filed in a layered hierarchical structure, and obtaining quality representation features after multi-layer convolution operation; the extraction unit 3 specifically includes:
a first execution subunit for performing a first 3D convolution operation on each point in the KNN graph of the 3D space to extract color features;
a second execution subunit for performing a second 3D convolution operation on each point in the KNN graph of the 3D space to extract geometric features;
a third execution subunit, configured to fuse the color feature and the geometric feature as a new feature, and perform a third 3D convolution operation on the new feature to obtain a quality representation feature;
the processing unit is used for inputting the quality representation characteristics into the pooling layer for characteristic processing to obtain coarse-to-fine characteristics;
the regression unit is used for inputting the characteristics from the thick to the thin into a quality regression module consisting of a full connection layer and a BN layer to carry out quality regression so as to obtain the quality fraction of the target patch;
the traversing unit is used for traversing the K patches in the original point cloud to obtain the quality scores of the K patches, and averaging the quality scores of the K patches to be used as the final score of the original point cloud.
7. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 5 when the computer program is executed.
8. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310824543.4A CN116542981B (en) | 2023-07-06 | 2023-07-06 | Quality assessment method for reference-point-free cloud |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310824543.4A CN116542981B (en) | 2023-07-06 | 2023-07-06 | Quality assessment method for reference-point-free cloud |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116542981A true CN116542981A (en) | 2023-08-04 |
| CN116542981B CN116542981B (en) | 2023-08-29 |
Family
ID=87447550
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310824543.4A Active CN116542981B (en) | 2023-07-06 | 2023-07-06 | Quality assessment method for reference-point-free cloud |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116542981B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107452065A (en) * | 2017-07-05 | 2017-12-08 | 山东理工大学 | The border spot identification method of surface sampled data in kind |
| CN111866518A (en) * | 2020-07-29 | 2020-10-30 | 西安邮电大学 | Self-adaptive three-dimensional point cloud compression method based on feature extraction |
| CN113486963A (en) * | 2021-07-12 | 2021-10-08 | 厦门大学 | Density self-adaptive point cloud end-to-end sampling method |
| CN113850748A (en) * | 2020-06-09 | 2021-12-28 | 上海交通大学 | Point cloud quality evaluation system and method |
| CN115147317A (en) * | 2022-05-30 | 2022-10-04 | 山东大学 | Point cloud color quality enhancement method and system based on convolutional neural network |
| CN116137059A (en) * | 2023-04-17 | 2023-05-19 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
-
2023
- 2023-07-06 CN CN202310824543.4A patent/CN116542981B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107452065A (en) * | 2017-07-05 | 2017-12-08 | 山东理工大学 | The border spot identification method of surface sampled data in kind |
| CN113850748A (en) * | 2020-06-09 | 2021-12-28 | 上海交通大学 | Point cloud quality evaluation system and method |
| CN111866518A (en) * | 2020-07-29 | 2020-10-30 | 西安邮电大学 | Self-adaptive three-dimensional point cloud compression method based on feature extraction |
| CN113486963A (en) * | 2021-07-12 | 2021-10-08 | 厦门大学 | Density self-adaptive point cloud end-to-end sampling method |
| CN115147317A (en) * | 2022-05-30 | 2022-10-04 | 山东大学 | Point cloud color quality enhancement method and system based on convolutional neural network |
| CN116137059A (en) * | 2023-04-17 | 2023-05-19 | 宁波大学科学技术学院 | Three-dimensional point cloud quality evaluation method based on multi-level feature extraction network model |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116542981B (en) | 2023-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110569721B (en) | Recognition model training method, image recognition method, device, equipment and medium | |
| EP3933693B1 (en) | Object recognition method and device | |
| KR102591961B1 (en) | Model training method and device, and terminal and storage medium for the same | |
| CN111179177A (en) | Image reconstruction model training method, image reconstruction method, device and medium | |
| EP4163831A1 (en) | Neural network distillation method and device | |
| US20240143977A1 (en) | Model training method and apparatus | |
| CN111476806B (en) | Image processing method, image processing device, computer equipment and storage medium | |
| CN112580572B (en) | Training method, using method, equipment and storage medium of multi-task recognition model | |
| CN111598779A (en) | Image super-resolution processing method and device, electronic device and storage medium | |
| CN112529068B (en) | Multi-view image classification method, system, computer equipment and storage medium | |
| CN110633640A (en) | Method for identifying complex scene by optimizing PointNet | |
| CN113435330A (en) | Micro-expression identification method, device, equipment and storage medium based on video | |
| CN115545166A (en) | Improved ConvNeXt convolutional neural network and remote sensing image classification method thereof | |
| CN114241459B (en) | Driver identity verification method and device, computer equipment and storage medium | |
| CN114283294A (en) | Neural network point cloud feature extraction method, system, equipment and storage medium | |
| CN116542981B (en) | Quality assessment method for reference-point-free cloud | |
| CN104252715B (en) | Single line image-based three-dimensional reconstruction method | |
| CN113255511A (en) | Method, apparatus, device and storage medium for living body identification | |
| CN114170231A (en) | Image semantic segmentation method and device based on convolutional neural network and electronic equipment | |
| CN116977539A (en) | Image processing method, apparatus, computer device, storage medium, and program product | |
| CN113469091B (en) | Face recognition method, training method, electronic device and storage medium | |
| CN112634136B (en) | Image super-resolution method and system based on image feature rapid stitching | |
| CN114821743A (en) | 3D face recognition method of chip, face recognition chip and readable storage medium | |
| CN114817595A (en) | Sketch-based three-dimensional model retrieval method, device, equipment and medium | |
| Lv et al. | A 3D face recognition method using region-based extended local binary pattern |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |