CN117872354A - Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud - Google Patents

Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud Download PDF

Info

Publication number
CN117872354A
CN117872354A CN202410271242.8A CN202410271242A CN117872354A CN 117872354 A CN117872354 A CN 117872354A CN 202410271242 A CN202410271242 A CN 202410271242A CN 117872354 A CN117872354 A CN 117872354A
Authority
CN
China
Prior art keywords
preset
target
neighborhood
points
coordinates
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202410271242.8A
Other languages
Chinese (zh)
Other versions
CN117872354B (en
Inventor
姜梦馨
王培栋
朱健楠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shaanxi Orca Electronic Intelligent Technology Co ltd
Original Assignee
Shaanxi Orca Electronic Intelligent Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shaanxi Orca Electronic Intelligent Technology Co ltd filed Critical Shaanxi Orca Electronic Intelligent Technology Co ltd
Priority to CN202410271242.8A priority Critical patent/CN117872354B/en
Publication of CN117872354A publication Critical patent/CN117872354A/en
Application granted granted Critical
Publication of CN117872354B publication Critical patent/CN117872354B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Landscapes

  • Radar Systems Or Details Thereof (AREA)

Abstract

The embodiment of the invention discloses a fusion method, a device, equipment and a medium of a multi-millimeter wave Lei Dadian cloud. The method comprises the following steps: acquiring point clouds detected by each millimeter wave radar, and determining a neighborhood range and a neighborhood density of each point cloud according to a preset neighborhood radius; clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm to determine a target cluster; detecting a current target cluster and a target cluster reaching the target cluster Yu Lei according to a preset key area and a preset association detection method, obtaining association relations of the target clusters of different radars, and associating the target clusters through a preset association algorithm; and associating the coordinates of the points in the associated target cluster one by one, and carrying out weighted fusion on the coordinates of the two associated points to determine the target coordinates. By implementing the method provided by the embodiment of the invention, the problem that the point cloud fusion of the multi-millimeter wave radar cannot exert the advantages of multi-source data fusion can be solved.

Description

Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud
Technical Field
The invention relates to the technical field of mobile robots and automatic driving, in particular to a fusion method, a device, equipment and a medium of a multi-millimeter wave Lei Dadian cloud.
Background
With the continuous development of application technologies in the fields of mobile robots and autopilot, autopilot equipment can remain stable in the face of complex environments, which is an urgent problem to be solved. In the prior art, millimeter wave radars are often used as sensors of automatic driving equipment and are operation detection data of the automatic driving equipment, but because the view angle of a single millimeter wave radar is limited and the point cloud data detected by the single millimeter wave radar is sparse, the point clouds of a plurality of millimeter wave radars are often fused to realize more accurate and comprehensive perception of a complex scene, but because clutter points of millimeter wave Lei Dadian clouds are more, the direct accumulation of the point clouds of a plurality of radars can lead to clutter point superposition, the accuracy of environment perception is affected, the accuracy of the target point clouds cannot be improved, and the point cloud redundancy is increased by the difficulty of perception tasks, so that the multi-millimeter wave radar point cloud fusion method cannot fully exert the advantage of multi-source data fusion.
Disclosure of Invention
The embodiment of the invention provides a fusion method, device, equipment and medium of a multi-millimeter wave Lei Dadian cloud, and aims to solve the problem that point cloud fusion of a multi-millimeter wave radar cannot exert the advantages of multi-source data fusion.
In a first aspect, an embodiment of the present invention provides a fusion method of a multi-millimeter wave Lei Dadian cloud, including: acquiring point clouds detected by each millimeter wave radar, and determining the neighborhood range and the neighborhood density of each point cloud according to a preset neighborhood radius; clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm to determine a target cluster; performing relevance detection on the current target cluster and the target cluster reaching Yu Lei according to a preset key area and a preset relevance detection method, acquiring relevance relations of the target clusters among different radars, and relating the target clusters through a preset relevance algorithm; and associating the coordinates of the points in the associated target cluster between the two radars one by one according to a preset association algorithm, and carrying out weighted fusion on the coordinates of the two associated points through a preset weighting algorithm to determine the target coordinates.
In a second aspect, an embodiment of the present invention further provides a fusion apparatus of a multi-millimeter wave Lei Dadian cloud, including: the neighborhood determining unit is used for obtaining the point cloud detected by each millimeter wave radar and determining the neighborhood range and the neighborhood density of each point cloud according to a preset neighborhood radius; the clustering unit is used for clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm so as to determine a target cluster; the association unit is used for carrying out association detection on the current target cluster and the target cluster which is reached by the current target cluster Yu Lei according to a preset key area and a preset association detection method, obtaining association relations of the target clusters among different radars, and associating the target clusters through a preset association algorithm; and the coordinate determining unit is used for associating the coordinates of the points in the related target cluster between the two radars one by one according to a preset association algorithm, and carrying out weighted fusion on the coordinates of the related two points through a preset weighting algorithm to determine the target coordinates.
In a third aspect, an embodiment of the present invention further provides a computer device, which includes a memory and a processor, where the memory stores a computer program, and the processor implements the method when executing the computer program.
In a fourth aspect, embodiments of the present invention also provide a computer readable storage medium storing a computer program comprising program instructions which, when executed by a processor, implement the above-described method.
The embodiment of the invention provides a fusion method, a device, equipment and a medium of a multi-millimeter wave Lei Dadian cloud. Wherein the method comprises the following steps: acquiring point clouds detected by each millimeter wave radar, and determining the neighborhood range and the neighborhood density of each point cloud according to a preset neighborhood radius; clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm to determine a target cluster; performing relevance detection on the current target cluster and the target cluster reaching Yu Lei according to a preset key area and a preset relevance detection method, acquiring relevance relations of the target clusters among different radars, and relating the target clusters through a preset relevance algorithm; and associating the coordinates of the points in the associated target cluster between the two radars one by one according to a preset association algorithm, and carrying out weighted fusion on the coordinates of the two associated points through a preset weighting algorithm to determine the target coordinates. In the embodiment of the invention, the point clouds in the millimeter wave radars are clustered and associated, and each point is associated step by step, so that clutter points in the point clouds can be removed, the coordinates of the associated points are subjected to weighted fusion to determine the target coordinates, more accurate point cloud coordinates can be obtained, more comprehensive and more accurate observation of the target is realized, the accuracy of a subsequent perception task can be effectively improved by fused point cloud data, and the advantage of multi-source data fusion is fully exerted.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of a method for fusing a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention;
fig. 2 is a schematic sub-flowchart of a fusion method of a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention;
fig. 3 is a schematic sub-flowchart of a fusion method of a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention;
fig. 4 is a schematic sub-flowchart of a fusion method of a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention;
fig. 5 is a schematic sub-flowchart of a fusion method of a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention;
fig. 6 is a schematic sub-flowchart of a fusion method of a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention;
fig. 7 is a schematic block diagram of a fusion device of a multi-millimeter wave Lei Dadian cloud provided by an embodiment of the present invention;
Fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It will be understood that the terms "comprises" and "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Referring to fig. 1, fig. 1 is a flow chart of a method for fusing a plurality of millimeter waves Lei Dadian cloud according to an embodiment of the present invention. The fusion method of the multi-millimeter wave Lei Dadian cloud in the embodiment can be applied to unmanned ships with more than two millimeter wave radar sensors, and the point cloud detected by the millimeter wave radar sensors can be effectively fused by adopting the method, so that the accuracy of environment sensing is improved, the redundancy of the target point cloud is reduced, and the calculated amount of subsequent sensing tasks is reduced.
Fig. 1 is a flow chart of a method for fusing a multi-millimeter wave Lei Dadian cloud according to an embodiment of the present invention. As shown, the method includes the following steps S110-S140.
S110, acquiring point clouds detected by each millimeter wave radar, and determining the neighborhood range and the neighborhood density of each point cloud according to a preset neighborhood radius.
In the present embodiment, the millimeter wave radar is a radar that operates in millimeter wave band detection. The point cloud is a data set of points in space, and the position of each point in the point cloud is described by a set of cartesian coordinates. The neighborhood refers to a basic topology on the collection. The preset neighborhood radius is a range radius of the neighborhood. And determining the preset neighborhood radius according to the effective angle quantization precision of the millimeter wave radar. The neighborhood density refers to the number of the rest point clouds in the neighborhood range of the point cloud. And the neighborhood range and the neighborhood density of each point cloud are determined according to the preset neighborhood radius by receiving and acquiring the point cloud detected by each millimeter wave radar, so that the subsequent standardization processing of the detected point cloud is facilitated.
In one embodiment, as shown in FIG. 2, the step S110 further includes steps S111-S113.
S111, determining the neighborhood range of the point cloud according to the preset neighborhood radius;
s112, calculating the distance between every two point clouds, and judging that the two point clouds are adjacent points if the distance is in the neighborhood range of the two point clouds;
s113, counting the number of the adjacent points of the point clouds to determine the neighborhood density of each point cloud.
In this embodiment, the step of determining the neighborhood range of the point cloud according to the preset neighborhood radius includes determining the preset neighborhood radiusThe neighborhood radius is designed based on the self-adaptive change of the target distance because the target interval on two adjacent angle quantization units is more distant due to the limited angle quantization precision of the millimeter wave radar, and the method for determining the preset neighborhood radius comprises the following steps:wherein (1)>For the angle quantization precision of the millimeter wave radar, R is the distance from the point cloud to the coordinate origin of the millimeter wave radar, < ->For scaling factor +.>And presetting the neighborhood radius. The neighborhood range of the point cloud may be determined from the neighborhood radius. The distance between each two of the point clouds is calculated, in particular, the euclidean distance between the two point clouds can be calculated, since euclidean distance is the most common distance measure, measuring the absolute distance between two points in a multidimensional space. If the distance is within the neighborhood range of the two point clouds, judging that the two point clouds are adjacent to each other, specifically, </i > >,/>Is a point cloud->European distance,/, of->Respectively->Distance to the origin of coordinates of the millimeter wave radar. That is, two points meeting the above conditions are adjacent points, and the number of the adjacent points of the point cloud is counted and determinedDetermining the neighborhood density of each point cloud. By determining the adjacent point density of the point cloud, the neighborhood condition of the point cloud can be known, and the point cloud can be clustered conveniently.
S120, clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm to determine a target cluster.
In this embodiment, the preset clustering algorithm is a Density-based clustering algorithm (Density-Based Spatial Clustering of Applications with Noise, DBSCAN), which may cluster the point cloud according to the neighborhood Density. Clustering a certain point cloud and the point cloud in the neighborhood range, and gathering the point clouds of the same type to form a target cluster. The non-clustered point cloud appearing after clustering is an outlier, and is judged to be an independent noise point and is removed from the point cloud, and only the target cluster is reserved. By clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm to determine a target cluster, independent noise points can be identified and filtered, so that subsequent calculated amount is reduced.
In one embodiment, as shown in FIG. 3, the step S120 further includes steps S121-S123.
S121, randomly selecting one point cloud to cluster the point clouds in the neighborhood range;
s122, detecting whether core points exist in the neighborhood range of the selected point cloud, and if the core points exist, clustering the point cloud in the neighborhood range of the core points together to determine the target cluster.
In this embodiment, the selecting one of the point clouds at random clusters the point clouds in the neighborhood range, specifically, from any one of the point clouds, clusters the point clouds in the neighborhood range. And monitoring whether the core points exist in the neighborhood range of the selected point cloud in the clustering process, wherein the core points are the point clouds with the neighborhood density exceeding a preset density threshold. If the core point exists, the core is processedAnd clustering the point clouds in the neighborhood range of the heart point together. Specifically, the point cloud may be divided into a core point, a boundary point and a noise point according to the neighborhood density, the point cloud is defined as the core point if the neighborhood density exceeds a preset density threshold, the points in the neighborhood of the core point, which do not exceed the preset density threshold, are defined as the boundary points, and the rest points are noise points. It will be appreciated that clustering the point cloud, i.e. clustering the core points of the same type with boundary points, assigns the same label to these points. After clustering, the point clouds with the same label will be used as a target cluster. Specifically, for the ith millimeter wave radar, the target cluster obtained after the point cloud included in the ith millimeter wave radar is clustered can be represented Wherein->The total number of clusters under the radar. After the target clusters are determined, each target cluster can be used as a whole for subsequent association operation.
S130, performing relevance detection on the current target cluster and the target cluster reached by the target cluster Yu Lei according to a preset key area and a preset relevance detection method, acquiring relevance relations of the target clusters among different radars, and relating the target clusters through a preset relevance algorithm.
In this embodiment, the preset critical area is a clearer area that can be detected by the millimeter wave radar, and according to a transmit waveform adopted by the millimeter wave radar and an antenna design of the radar, a sector area of a maximum non-ambiguous detection distance, in which each radar is located in a 3dB beam broadband and the distance is less than 80%, is defined as the preset critical area for detection of the radar. The millimeter wave radar in the preset key area has stronger emission energy, the signal-to-noise ratio of the target is higher, the real target positioned in the area is detected with higher detection probability, and the real target positioned outside the area can be missed by the radar due to lower signal-to-noise ratio. Focusing the target of each millimeter wave radar And detecting whether other target clusters with relevance exist in the key areas of other radars or not, namely acquiring the relevance relation between each target cluster. The method can be used for identifying and filtering cluster clusters which are multipath clutter according to the association relation, and after the cluster clusters of the multipath clutter are filtered, the cluster clusters are associated one to one by a preset association algorithm according to the space distance among the target cluster clusters for the rest target cluster clusters. Specifically, according to the spatial distance between the target clusters, the target clusters with the association relationship between the two radars are associated. For example, for a cluster of radars iCluster with radar mConverting the clustered point cloud into a body coordinate system by using known radar external parameters to obtain +.>And->. Calculating the spatial distance +.>Obtaining distance matrix->Wherein->Respectively->The centroid is the point obtained by averaging the coordinates of all point clouds of the target cluster). Based on distance matrix->And correlating the target cluster by adopting a Hungary algorithm. By according toThe association relation is associated with the target cluster which are associated with each other by a preset association algorithm, so that the association relation of the target clusters among different radars can be obtained, namely fusion can be carried out according to the associated target cluster, and the accuracy of detecting the targets by the multi-millimeter wave radar is improved.
In one embodiment, as shown in FIG. 4, the step S130 further includes steps S131-S133.
S131, determining the mass center of each target cluster according to the coordinates of point clouds in each target cluster, and judging whether the current target cluster is in the preset key areas of the rest millimeter wave radars according to the mass center;
s132, if so, detecting whether the centroids of the rest target clusters exist in an intersecting range, wherein the intersecting range is a range in which the preset key areas of the rest millimeter wave radars coincide with the neighborhood range of the current target cluster;
and S133, if the target cluster exists, judging that the target cluster and the other target cluster closest to the target cluster have the association relation.
In this embodiment, the centroid is a point determined according to the coordinates of each point in the point cloud in the target cluster, specifically, the coordinates of all points in the target cluster are averaged, and the coordinates of the average value are the centroid. Judging whether the current target cluster is in the preset key area of the rest millimeter wave radars according to the mass center, for example, detecting whether the mass center is in the preset key area of the millimeter wave radars m according to the fact that the current target cluster is i, converting the coordinates of the mass center into the coordinates of the millimeter wave radars m, and calculating the azimuth angle and the distance of the mass center under radar observation, wherein the azimuth angle and the distance are determined in the following manner: Wherein, said->Said azimuth and distance, respectively, said +.>Is the coordinates of the centroid under millimeter wave radar m. If->And->Judging that the centroid is in a preset key area of the millimeter wave radar m, namely the target cluster +.>Is located in the critical area of radar m detection. Wherein (1)>Azimuth size corresponding to 3dB beamwidth for radar m, +.>The maximum unambiguous detection distance under the waveform used for radar m. If the centroid is not in the preset critical area detected by the radar m, the subsequent relevance detection is not carried out. And if the mass centers of the rest target clusters exist in the intersection range, detecting whether the mass centers of the rest target clusters exist in the intersection range, wherein the intersection range is a range in which the preset key regions of the rest millimeter wave radars coincide with the neighborhood range of the current target cluster. If the target cluster exists, judging that the target cluster has the association relation with another target cluster closest to the target cluster. For example, there is a nearest centroid within the preset critical area of radar m and in the neighborhood of said centroid>Then determine the- >And->Has an association relationship. Traversing all radars except the radar i to obtain a target cluster +.>Association of target clusters of any radar. It should be noted that, due to the different reflection paths of the target under different radars, the "ghosted" target generated by the multipath effect is generally not observed by multiple millimeter wave radars at the same time, but the real target in the overlapping fields of view of multiple radars is more likely to be detected by multiple radars at the same time. Based on the above, whether the target cluster is a multipath clutter point is judged according to the association relation of the target cluster, and concretely, the following two cases are included: if the target cluster is not located in the key area detected by other radars, the target cluster is not detected by other radars possibly because the signal to noise ratio is low, the result of spatial correlation cannot indicate whether the target cluster is a multipath clutter point, and the target cluster is reserved. If the target cluster is located in the key area detected by some other radars and at least one target cluster with an association relation exists, the target cluster is identified as a cluster from a real target, otherwise, the target cluster is identified as a multipath clutter point and filtered from the point cloud of the radars. And through the association of the target cluster, multipath clutter points in each radar point cloud can be filtered. The reserved cluster can achieve improvement of cloud quality of the target point through multi-radar fusion in the following steps.
And S140, associating coordinates of points in the associated target cluster between two radars one by one according to a preset association algorithm, and carrying out weighted fusion on the coordinates of the associated two points through a preset weighting algorithm to determine target coordinates.
In this embodiment, the preset joint algorithm is an algorithm that correlates coordinates of the points. And associating the coordinates of each point among the associated target clusters, namely determining the relationship among the points. After the determination, the coordinates of the two related points are subjected to weighted fusion through a preset weighting algorithm, so that the target coordinates of the points under a local coordinate system can be determined. And the coordinates of the two related points are subjected to weighted fusion to determine the target coordinates, so that the two points with the association relationship in all the associated cluster clusters can be optimized, and the cloud quality of the target point is improved.
In one embodiment, as shown in fig. 5, the step S140 further includes steps S141-S144.
S141, acquiring a first coordinate of each point in the two related target clusters through a coordinate conversion method, wherein the first coordinate is a coordinate of the point under a preset local coordinate system;
S142, acquiring a first speed and a second speed of each point in the two related target clusters by a speed acquisition method, wherein the first speed is the Doppler speed of the point under the millimeter wave radar to which the point belongs, and the second speed is the Doppler speed of the point under the millimeter wave radar to which the other point belongs;
s143, acquiring joint distances from the first coordinates, the first speed and the second speed of any point in the two associated target clusters through the preset joint algorithm, and determining the distance matrix according to the joint distances;
s144, the first coordinates of the points in the associated target cluster are associated one by one according to a preset nearest neighbor method and the distance matrix.
In this embodiment, the coordinate conversion method is a method of performing coordinate conversion by calibration acquired in advance. Specifically, the calibration is represented by a rotation matrix R and a translation vector t, and the transformed coordinates can be represented as: r·coordinate before conversion+t. And acquiring the first speed and the second speed of each point in the two related target clusters by a speed acquisition method, and specifically, firstly fitting the movement speed of the radar by using the Doppler speed of the point cloud under each radar. For example, the millimeter wave radar at one of the two related points is i, the radar at the other point is m, and the first radar i after clutter filtering is performed jThe individual points can be expressed as:wherein->The distance, horizontal azimuth, altitude and doppler velocity of the point measured by the radar, respectively. The y-axis of the radar is directed directly in front of the radar and the x-axis is directed to the right, the speed of the radar can be determined by the speed in the direction of the x-axis +.>And speed in the y-axis direction +.>And (3) representing. When the radar moves, the doppler velocity of the stationary target point cloud is a projection of the radar velocity in the radial direction, so the relationship between the doppler velocity of the point cloud and the radar movement velocity can be expressed as: />The relation between the Doppler velocity and the spatial position of all point clouds under the radar can be fitted by using a least square method, so that the motion velocity of the radar i can be estimated>. Similarly, the movement speed of radar m can be estimated +.>. Acquiring the second speed of each point in the two related target clusters by a speed acquisition method, for example, by the speed of the radar m and the space coordinates of the point cloud of the radar i under the coordinate system of the radar m, the Doppler speed of the point cloud of the radar i under the radar m can be estimated, and the target cluster of the radar i can be clustered>The kth point in (2) is denoted +.>Cluster of radars m->The nth point of (2) is denoted +.>. The kth point is +. >Converting the coordinate system of the radar i into the coordinate system of the radar m to obtain the radar m observation +.>Azimuth angle +.>And height angle. According to the estimated speed of motion of radar m +.>And the relationship between the cloud Doppler velocity and the radar velocity, the +.>Doppler velocity observed under radar m +.>. Similarly, predict +.>Doppler velocity observed under radar i +.>I.e. the second speed of the two points that have been associated is obtained. In the case of the coordinate transformation method +.>、/>Converting to a unified body coordinate system to obtain a first coordinate: />. Acquiring a joint distance from the first coordinate, the first speed and the second speed of any point in the two associated target clusters through the preset joint algorithm, wherein the preset joint algorithm is as follows:wherein,for joint distance->Is super-parameter (herba Cinchi Oleracei)>、/>Said first coordinates of two of said points respectively,for said second speed of two said points, and (2)>,/>Said first speed being two of said points; wherein the hyper-parameters are used to balance the spatial range and the Doppler range. Obtaining a distance matrix from the joint distances >Wherein->Cluster +.>The number of point clouds. Clustering targets by nearest neighbor method based on distance matrixThe point clouds in the clusters are associated point by point. And sequentially carrying out point-to-point association on the points of all the associated clusters between the two radars. By associating the points of the associated target cluster one by one. The method can improve the accuracy of the detection target, reduce the redundancy of the target point cloud and reduce the calculated amount of the subsequent perception task.
In one embodiment, as shown in FIG. 6, the step S140 further includes steps S145-S146.
S145, determining covariance matrixes of the two points according to the first coordinates of the two related points and preset space information of the preset local coordinate system;
and S146, carrying out weighted fusion on the first coordinates of the two related points according to the covariance matrixes of the two points, the first coordinates and the preset weighting algorithm to determine the target coordinates.
In this embodiment, the preset spatial information is a distance, an azimuth angle and an altitude angle of a certain point measured between the millimeter wave radars. Determining covariance matrices of the two points according to the preset spatial information and the first coordinates of the two related points, specifically determining spatial information of the two related points in a local coordinate system according to the preset spatial information of the two related points and external parameters of the radar, namely the first coordinates, and determining covariance matrices of the two points in the local coordinate system according to the spatial information, wherein, for example, the conversion of the related coordinates of the two points into the coordinates in the local coordinate system is respectively as follows: The determined covariance matrices are respectively: />. And carrying out weighted fusion on the first coordinates of the two related points according to the covariance matrixes of the two points, the first coordinates and the preset weighting algorithm to determine the target coordinates. Specifically, the preset additionThe weight algorithm is as follows:wherein (1)>For the target coordinates +.>For the covariance matrix of two of the points,>the difference between the reference point and the first coordinates of the two points, respectively. The first coordinates of the two associated points are subjected to weighted fusion to determine the target coordinates, so that the two points with association relations in all the associated clusters can be optimized, the quality of the target point cloud is improved, and the advantages of multi-millimeter wave Lei Dadian cloud multi-source data fusion are fully exerted.
Fig. 7 is a schematic block diagram of a fusion device 200 of a multi-millimeter wave Lei Dadian cloud provided by an embodiment of the invention. As shown in fig. 7, the invention further provides a fusion device of the multi-millimeter-wave Lei Dadian cloud, corresponding to the fusion method of the multi-millimeter-wave Lei Dadian cloud. The multi-millimeter wave Lei Dadian cloud fusion device comprises a unit for executing the multi-millimeter wave Lei Dadian cloud fusion method, and the device can be configured in a desktop computer, a tablet computer, a portable computer, and other terminals. Specifically, referring to fig. 7, the fusion apparatus of the multi-millimeter wave Lei Dadian cloud includes a neighborhood determining unit 210, a clustering unit 220, an associating unit 230, and a coordinate determining unit 240.
The neighborhood determining unit 210 is configured to obtain a point cloud detected by each millimeter wave radar, and determine a neighborhood range and a neighborhood density of each point cloud according to a preset neighborhood radius.
In an embodiment, the neighborhood determining unit 210 includes a range determining unit, a calculating unit, and a statistics unit.
The range determining unit is used for determining the neighborhood range of the point cloud according to the preset neighborhood radius;
the computing unit is used for computing the distance between every two point clouds, and if the distance is in the neighborhood range of the two point clouds, the two point clouds are judged to be adjacent points;
and the statistics unit is used for counting the number of the adjacent points of the point clouds and determining the neighborhood density of each point cloud.
And the clustering unit 220 is configured to cluster the point cloud with other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm, so as to determine a target cluster.
In an embodiment, the clustering unit 220 includes a selecting unit and a detecting unit.
The selecting unit is used for randomly selecting one point cloud to cluster the point clouds in the neighborhood range;
the detection unit is used for detecting whether core points exist in the neighborhood range of the selected point cloud, if so, the point clouds in the neighborhood range of the core points are clustered together to determine the target cluster.
And the association unit 230 is configured to perform association detection on the current target cluster and the target cluster under Yu Lei according to a preset key area and a preset association detection method, obtain association relations between the target clusters among different radars, and associate the target clusters through a preset association algorithm.
In an embodiment, the association unit 230 includes a determination unit, a presence unit, and a determination unit.
The judging unit is used for determining the mass center of each target cluster according to the coordinates of point clouds in each target cluster, and judging whether the current target cluster is in the preset key areas of the rest millimeter wave radars according to the mass center;
the existence unit is used for detecting whether the centroids of the rest target clusters exist in an intersection range if the centroids are located in the intersection range, wherein the intersection range is a range in which the preset key areas of the rest millimeter wave radars coincide with the neighborhood range of the current target cluster;
and the judging unit is used for judging that the target cluster has the association relation with the other target cluster closest to the target cluster if the target cluster exists.
The coordinate determining unit 240 is configured to associate coordinates of points in the target cluster that are associated between two radars one by one according to a preset association algorithm, and determine target coordinates by performing weighted fusion on the coordinates of the two associated points through a preset weighting algorithm.
In one embodiment, the coordinate determining unit 240 includes a coordinate converting unit, a speed acquiring unit, a distance unit, and a point associating unit.
The coordinate conversion unit is used for obtaining first coordinates of each point in the two related target clusters through a coordinate conversion method, wherein the first coordinates are coordinates of the point under a preset local coordinate system;
a speed acquisition unit, configured to acquire a first speed and a second speed of each of the points in the two related target clusters by using a speed acquisition method, where the first speed is a doppler speed of the point under a millimeter wave radar to which the point belongs, and the second speed is a doppler speed of the point under a millimeter wave radar to which another point belongs;
the distance unit is used for acquiring a joint distance from the first coordinate, the first speed and the second speed of any point in the two associated target cluster clusters through the preset joint algorithm, and determining the distance matrix according to the joint distance;
And the point association unit is used for associating the first coordinates of the points in the associated target cluster one by one according to a preset nearest neighbor method and the distance matrix.
In one embodiment, the coordinate determining unit 240 includes a matrix determining unit and a fusing unit.
A matrix determining unit, configured to determine covariance matrices of the two points according to the first coordinates of the two associated points and preset spatial information of the preset local coordinate system;
and the fusion unit is used for carrying out weighted fusion on the first coordinates of the two related points according to the covariance matrixes of the two points, the first coordinates and the preset weighting algorithm to determine the target coordinates.
It should be noted that, as those skilled in the art can clearly understand, the specific implementation process of the above-mentioned fusion device 200 and each unit of the multi-millimeter wave Lei Dadian cloud may refer to the corresponding description in the foregoing method embodiment, and for convenience and brevity of description, the description is omitted here.
The above-described fusion apparatus of the multi-millimeter wave Lei Dadian cloud may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 8.
Referring to fig. 8, fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 may be a terminal or a server, wherein the terminal may be an unmanned ship. The server may be an independent server or a server cluster formed by a plurality of servers.
With reference to FIG. 8, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a fusion method of a multi-millimeter wave Lei Dadian cloud.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of a computer program 5032 in the non-volatile storage medium 503, which computer program 5032, when executed by the processor 502, causes the processor 502 to perform a method of fusion of the multiple millimeter wave Lei Dadian clouds.
The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the architecture shown in fig. 8 is merely a block diagram of a portion of the architecture in connection with the present application and is not intended to limit the computer device 500 to which the present application is applied, and that a particular computer device 500 may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
Wherein the processor 502 is adapted to run a computer program 5032 stored in a memory for implementing the steps of the above method.
It should be appreciated that in embodiments of the present application, the processor 502 may be a central processing unit (Central Processing Unit, CPU), the processor 502 may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSPs), application specific integrated circuits (Application Specific Integrated Circuit, ASICs), off-the-shelf programmable gate arrays (Field-Programmable Gate Array, FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Those skilled in the art will appreciate that all or part of the flow in a method embodying the above described embodiments may be accomplished by computer programs instructing the relevant hardware. The computer program comprises program instructions, and the computer program can be stored in a storage medium, which is a computer readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer readable storage medium. The storage medium stores a computer program, wherein the computer program includes program instructions. The program instructions, when executed by a processor, cause the processor to perform the steps of the method as described above.
The storage medium may be a U-disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, or other various computer-readable storage media that can store program codes.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps described in connection with the embodiments disclosed herein may be embodied in electronic hardware, in computer software, or in a combination of the two, and that the elements and steps of the examples have been generally described in terms of function in the foregoing description to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be combined, divided and deleted according to actual needs. In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The integrated unit may be stored in a storage medium if implemented in the form of a software functional unit and sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention is essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a terminal, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.

Claims (10)

1. A method for fusing a multi-millimeter wave Lei Dadian cloud, comprising:
acquiring point clouds detected by each millimeter wave radar, and determining the neighborhood range and the neighborhood density of each point cloud according to a preset neighborhood radius;
clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm to determine a target cluster;
performing relevance detection on the current target cluster and the target cluster reaching Yu Lei according to a preset key area and a preset relevance detection method, acquiring relevance relations of the target clusters among different radars, and relating the target clusters through a preset relevance algorithm;
and associating the coordinates of the points in the associated target cluster between the two radars one by one according to a preset association algorithm, and carrying out weighted fusion on the coordinates of the two associated points through a preset weighting algorithm to determine the target coordinates.
2. The method of claim 1, wherein the step of determining the neighborhood range and neighborhood density for each of the point clouds according to a preset neighborhood radius comprises:
determining the neighborhood range of the point cloud according to the preset neighborhood radius;
calculating the distance between every two point clouds, and if the distance is in the neighborhood range of the two point clouds, judging that the two point clouds are adjacent points;
and counting the number of the adjacent points of the point clouds to determine the neighborhood density of each point cloud.
3. The method of claim 1, wherein the step of clustering the point cloud with other point clouds within the neighborhood according to the neighborhood density and a preset clustering algorithm to determine a target cluster comprises:
randomly selecting one point cloud to cluster the point clouds in the neighborhood range;
detecting whether core points exist in the neighborhood range of the selected point cloud, if so, clustering the point clouds in the neighborhood range of the core points together to determine the target cluster.
4. The method according to claim 1, wherein the step of performing relevance detection on the current target cluster and the target cluster reached by the current target cluster Yu Lei according to a preset key area and preset relevance detection method to obtain the relevance of the target cluster between different radars includes:
Determining the mass center of each target cluster according to the coordinates of point clouds in each target cluster, and judging whether the current target cluster is in the preset key areas of the rest millimeter wave radars according to the mass center;
if so, detecting whether the centroids of the rest target cluster exist in an intersecting range, wherein the intersecting range is a range in which the preset key areas of the rest millimeter wave radars coincide with the neighborhood range of the current target cluster;
if the target cluster exists, judging that the target cluster has the association relation with another target cluster closest to the target cluster.
5. The method according to claim 1, wherein the step of associating coordinates of points in the target cluster, which have been associated between two radars, one by one according to a preset association algorithm comprises:
acquiring a first coordinate of each point in the two related target clusters through a coordinate conversion method, wherein the first coordinate is the coordinate of the point under a preset local coordinate system;
acquiring a first speed and a second speed of each point in the two related target clusters by a speed acquisition method, wherein the first speed is the Doppler speed of the point under the millimeter wave radar to which the point belongs, and the second speed is the Doppler speed of the point under the millimeter wave radar to which the other point belongs;
Acquiring a joint distance from the first coordinate, the first speed and the second speed of any point in the two associated target clusters through the preset joint algorithm, and determining the distance matrix according to the joint distance;
and according to a preset nearest neighbor method and the distance matrix, the first coordinates of the points in the associated target cluster are associated one by one.
6. The method according to claim 5, wherein the step of determining the target coordinates by weighted fusion of the coordinates of the two associated points by a preset weighting algorithm comprises:
determining covariance matrixes of the two points according to the first coordinates of the two related points and preset space information of the preset local coordinate system;
and carrying out weighted fusion on the first coordinates of the two related points according to the covariance matrixes of the two points, the first coordinates and the preset weighting algorithm to determine the target coordinates.
7. The method of claim 6, wherein the predetermined joint algorithm is:wherein, the method comprises the steps of, wherein,for the joint distance- >Is super-parameter (herba Cinchi Oleracei)>Said first coordinates of two said points, respectively,>for said second speed of two said points, and (2)>,/>Said first speed being two of said points; and
the preset weighting algorithm is as follows:wherein, the method comprises the steps of, wherein,for the target coordinates +.>For the covariance matrix of two of the points,>the difference between the reference point and the first coordinates of the two points, respectively.
8. A fusion device of a multi-millimeter wave Lei Dadian cloud, comprising:
the neighborhood determining unit is used for obtaining the point cloud detected by each millimeter wave radar and determining the neighborhood range and the neighborhood density of each point cloud according to a preset neighborhood radius;
the clustering unit is used for clustering the point cloud and other point clouds in the neighborhood range according to the neighborhood density and a preset clustering algorithm so as to determine a target cluster;
the association unit is used for carrying out association detection on the current target cluster and the target cluster which is reached by the current target cluster Yu Lei according to a preset key area and a preset association detection method, obtaining association relations of the target clusters among different radars, and associating the target clusters through a preset association algorithm;
And the coordinate determining unit is used for associating the coordinates of the points in the related target cluster between the two radars one by one according to a preset association algorithm, and carrying out weighted fusion on the coordinates of the related two points through a preset weighting algorithm to determine the target coordinates.
9. A computer device, characterized in that it comprises a memory on which a computer program is stored and a processor which, when executing the computer program, implements the method according to any of claims 1-7.
10. A storage medium storing a computer program comprising program instructions which, when executed by a processor, implement the method of any one of claims 1-7.
CN202410271242.8A 2024-03-11 2024-03-11 Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud Active CN117872354B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410271242.8A CN117872354B (en) 2024-03-11 2024-03-11 Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410271242.8A CN117872354B (en) 2024-03-11 2024-03-11 Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud

Publications (2)

Publication Number Publication Date
CN117872354A true CN117872354A (en) 2024-04-12
CN117872354B CN117872354B (en) 2024-05-31

Family

ID=90583332

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410271242.8A Active CN117872354B (en) 2024-03-11 2024-03-11 Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud

Country Status (1)

Country Link
CN (1) CN117872354B (en)

Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018119902A1 (en) * 2016-12-29 2018-07-05 华为技术有限公司 Method and apparatus for detecting ground environment
CN113139607A (en) * 2021-04-27 2021-07-20 苏州挚途科技有限公司 Obstacle detection method and device
CN113219452A (en) * 2021-05-07 2021-08-06 电子科技大学 Distributed multi-radar joint registration and multi-target tracking method under unknown vision field
CN114387462A (en) * 2021-12-30 2022-04-22 中山大学 Dynamic environment sensing method based on binocular camera
CN114675275A (en) * 2022-03-21 2022-06-28 北京航空航天大学合肥创新研究院(北京航空航天大学合肥研究生院) Target detection method based on fusion of 4D millimeter wave radar and laser radar
CN114708585A (en) * 2022-04-15 2022-07-05 电子科技大学 Three-dimensional target detection method based on attention mechanism and integrating millimeter wave radar with vision
CN114779230A (en) * 2022-03-22 2022-07-22 福瑞泰克智能系统有限公司 Object tracking method and device, electronic equipment and vehicle
CN115147813A (en) * 2022-07-13 2022-10-04 纵目科技(上海)股份有限公司 Object space feature-based sparse point cloud processing method, system, device and medium
US20230184921A1 (en) * 2020-09-15 2023-06-15 Smart Radar System, Inc. Radar point cloud-based posture determination system
CN116580217A (en) * 2023-05-04 2023-08-11 桂林电子科技大学 Millimeter wave radar traffic target clustering method based on prediction error feedback
CN116703966A (en) * 2022-03-02 2023-09-05 福特全球技术公司 Multi-object tracking
WO2023197979A1 (en) * 2022-04-13 2023-10-19 腾讯科技(深圳)有限公司 Data processing method and apparatus, and computer device and storage medium
JP2023155187A (en) * 2022-04-08 2023-10-20 富士通株式会社 Target tracking method and device based upon radar point group
WO2023206888A1 (en) * 2022-04-25 2023-11-02 广东玖智科技有限公司 Ppg signal cluster center acquisition method and apparatus, and ppg signal processing method and apparatus
WO2023232165A1 (en) * 2022-06-01 2023-12-07 湖南大学无锡智能控制研究院 Multi-radar data fusion obstacle detection method, and system

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018119902A1 (en) * 2016-12-29 2018-07-05 华为技术有限公司 Method and apparatus for detecting ground environment
US20230184921A1 (en) * 2020-09-15 2023-06-15 Smart Radar System, Inc. Radar point cloud-based posture determination system
CN113139607A (en) * 2021-04-27 2021-07-20 苏州挚途科技有限公司 Obstacle detection method and device
CN113219452A (en) * 2021-05-07 2021-08-06 电子科技大学 Distributed multi-radar joint registration and multi-target tracking method under unknown vision field
CN114387462A (en) * 2021-12-30 2022-04-22 中山大学 Dynamic environment sensing method based on binocular camera
CN116703966A (en) * 2022-03-02 2023-09-05 福特全球技术公司 Multi-object tracking
CN114675275A (en) * 2022-03-21 2022-06-28 北京航空航天大学合肥创新研究院(北京航空航天大学合肥研究生院) Target detection method based on fusion of 4D millimeter wave radar and laser radar
CN114779230A (en) * 2022-03-22 2022-07-22 福瑞泰克智能系统有限公司 Object tracking method and device, electronic equipment and vehicle
JP2023155187A (en) * 2022-04-08 2023-10-20 富士通株式会社 Target tracking method and device based upon radar point group
WO2023197979A1 (en) * 2022-04-13 2023-10-19 腾讯科技(深圳)有限公司 Data processing method and apparatus, and computer device and storage medium
CN114708585A (en) * 2022-04-15 2022-07-05 电子科技大学 Three-dimensional target detection method based on attention mechanism and integrating millimeter wave radar with vision
WO2023206888A1 (en) * 2022-04-25 2023-11-02 广东玖智科技有限公司 Ppg signal cluster center acquisition method and apparatus, and ppg signal processing method and apparatus
WO2023232165A1 (en) * 2022-06-01 2023-12-07 湖南大学无锡智能控制研究院 Multi-radar data fusion obstacle detection method, and system
CN115147813A (en) * 2022-07-13 2022-10-04 纵目科技(上海)股份有限公司 Object space feature-based sparse point cloud processing method, system, device and medium
CN116580217A (en) * 2023-05-04 2023-08-11 桂林电子科技大学 Millimeter wave radar traffic target clustering method based on prediction error feedback

Non-Patent Citations (5)

* Cited by examiner, † Cited by third party
Title
CHENG, YUWEI ET AL.: "FloW: A Dataset and Benchmark for Floating Waste Detection in Inland Waters", 2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV 2021), 24 June 2022 (2022-06-24) *
曾碧;黄文;: "一种融合多特征聚类集成的室内点云分割方法", 计算机工程, no. 03, 31 March 2018 (2018-03-31) *
胡炜薇;李争;蒲书缙;杨莘元;: "杂波环境下雷达组网的多目标聚类融合跟踪", 哈尔滨工程大学学报, no. 04, 31 August 2006 (2006-08-31) *
赵恩来;郝文宁;赵飞;陈刚;邵校莎莎;: "改进的基于密度的航迹聚类算法", 计算机工程, no. 09, 31 May 2011 (2011-05-31) *
陈健华;: "基于机载LiDAR点云的道路提取算法研究", 测绘工程, no. 03, 31 May 2020 (2020-05-31) *

Also Published As

Publication number Publication date
CN117872354B (en) 2024-05-31

Similar Documents

Publication Publication Date Title
US5396252A (en) Multiple target discrimination
WO2021104497A1 (en) Positioning method and system based on laser radar, and storage medium and processor
US6522288B1 (en) Method and apparatus for determining location of objects based on range readings from multiple sensors
KR100899472B1 (en) A system and method for improving the precision of localization estimates
CN108882225B (en) Safe positioning method based on distance measurement in wireless sensor network
US20140372073A1 (en) Two-stage filtering based method for multiple target tracking
CN110088642B (en) System and method for detecting heading and speed of target object
CN113777600B (en) Multi-millimeter wave radar co-location tracking method
CN115542308B (en) Indoor personnel detection method, device, equipment and medium based on millimeter wave radar
CN112394334A (en) Radar reflection point clustering device and method and electronic equipment
US5604683A (en) Evaluating target tracking when using multiple sensors
US5821896A (en) Radar system
US20040243344A1 (en) Analytical estimation of performance of a sensor system
US6532191B2 (en) System and method for target tracking and motion anyalysis
CN109190647B (en) Active and passive data fusion method
CN109996205B (en) Sensor data fusion method and device, electronic equipment and storage medium
CN117872354B (en) Fusion method, device, equipment and medium of multi-millimeter wave Lei Dadian cloud
JP2010133960A (en) Method for multi-dimensional temporal kinematic filtering of radar blip, from antenna rotation to antenna rotation
US7868822B2 (en) Method and apparatus for determining an integrity risk in a satellite reference system
CN111523619A (en) Target existence probability calculation method and device, electronic equipment and storage medium
KR102572546B1 (en) Device and method of detecting multiple signal differences in single frequency receiver
CN116449329A (en) Method, system, equipment and storage medium for disambiguating speed of millimeter wave radar
CN111722297B (en) Target existence probability calculation method and device, electronic equipment and storage medium
CN117872310B (en) Radar-based water surface target tracking method, device, equipment and medium
CN114488191A (en) Laser radar scanning method, control equipment and laser radar

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant