KR102428050B1 - Information supplement method, lane recognition method, intelligent driving method and related products - Google Patents

Abstract

An embodiment of the present invention discloses an information supplementation method, a lane recognition method, an intelligent driving method, and related products, wherein the information supplementation method includes: acquiring multi-frame point cloud data including depth information and intensity information in each frame ; Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame of the multi-frame point cloud data except for the reference point cloud data obtaining; wherein the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is intensity information of the reference point cloud data More compact, the reference point cloud data is any one of the multiple frame point cloud data.

Description

Information supplement method, lane recognition method, intelligent driving method and related products

This application claims priority to the Chinese patent application filed with the Chinese Patent Office on October 31, 2019, the application number is 201911050904.4, and the application is entitled "Information Supplementation Method, Lane Recognition Method, Intelligent Driving Method and Related Products" and all contents are incorporated into the present application by reference.

This application relates to the field of computer vision technology, and more specifically, to an information supplementation method, a lane recognition method, an intelligent driving method, and related products.

Currently, in terms of target detection, laser radar (Light Detection And Ranging, abbreviation: LiDAR) is used to project a pulse laser of several rows of scan lines toward a target, and measure the target through backward scattering reflection. Since the laser does not belong to visible light, even in the case of poor lighting conditions, LiDAR can work well, and therefore laser radar is widely applied in various fields. For example, during unmanned driving, a lane can be extracted using LiDAR. Although LiDAR is not affected by light rays, due to the limitations of vertical and horizontal angular resolution, LiDAR measurement data is very sparse, and in order to fully utilize LiDAR survey data, sparse deep information among the survey data must first be gathered into dense depth. complemented by information. However, although LiDAR's survey data contains more intensity information, the intensity signal is related to other factors such as the target surface material and the direction of the laser's angle of incidence, making it impossible to supplement the intensity information. use is low. Since the intensity information cannot be supplemented, if the lane is recognized only with the deep information after supplementation, the environmental luminance may affect the acquisition of the depth information, resulting in a low recognition rate and poor robustness.

An embodiment of the present application provides an information supplementation method, a lane recognition method, an intelligent driving method, and related products that can increase the utilization rate of point cloud data resources and increase the lane recognition precision.

As a first aspect, an embodiment of the present application provides an information supplementation method, which

acquiring multi-frame point cloud data including depth information and intensity information in each frame;

Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame of the multi-frame point cloud data except for the reference point cloud data Including;

Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data, and the reference point cloud data is any one point cloud data among the multi-frame point cloud data.

In a second aspect, an embodiment of the present application provides a lane recognition method, which

collecting point cloud data of the road;

acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data using the information supplementation method of the first aspect;

and obtaining a lane recognition result by recognizing a lane according to the target intensity map.

In a third aspect, an embodiment of the present application provides an intelligent driving method, which

collecting point cloud data of scenes around the intelligent driving device;

acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data using the information supplementation method of the first aspect;

recognizing a target object in the surrounding scene according to the target intensity map;

and controlling the driving of the intelligent driving device according to the recognition result.

In a fourth aspect, an embodiment of the present application provides an information supplementation device, which

an acquisition unit for acquiring point cloud data of multiple frames including depth information and intensity information in each frame;

Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame except for the reference point cloud data among the point cloud data of the multiple frames a supplementary unit for obtaining

Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data, and the reference point cloud data is any one point cloud data among the multi-frame point cloud data.

In a fifth aspect, an embodiment of the present application provides a lane recognition device, which

a collection unit for collecting road point cloud data;

a supplementation unit for supplementing the point cloud data by using the information supplementation method of the first aspect to obtain a target intensity map corresponding to the point cloud data;

and a recognition unit for recognizing a lane according to the target intensity map to obtain a lane recognition result.

In a sixth aspect, an embodiment of the present application provides an intelligent driving device, which

a collecting unit for collecting point cloud data of scenes around the intelligent driving device;

a supplementation unit for supplementing the point cloud data by using the information supplementation method of the first aspect to obtain a target intensity map corresponding to the point cloud data;

a recognition unit for recognizing a target object in the surrounding scene according to the target intensity map;

and a control unit for controlling the driving of the intelligent driving device according to the recognition result.

In a seventh aspect, an embodiment of the present application provides an electronic device including a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory, and the processor wherein the program comprises instructions for executing the steps of the method of the first aspect, the second aspect or the third aspect.

As an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and the computer is configured to use the first aspect, the second aspect or the third aspect by the computer program. Execute the above method.

In a ninth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a non-transitory computer readable storage medium storing a computer program, the computer readable storage medium through the first aspect, The method of the second aspect or the third aspect is carried out.

Implementation of the embodiments of the present application has the following advantageous effects:

In the embodiment of the present application, by supplementing the depth information and the intensity information in the point cloud data, the collected sparse point cloud data is supplemented with the dense point cloud data, and by implementing the supplementation of the intensity information in the point cloud data, the point Utilization of cloud data resources is improved.

In order to more clearly explain the technical solutions in the embodiments of the present application, the accompanying drawings necessary for the description of the embodiments are briefly introduced below, and the accompanying drawings described below are some embodiments of the present application, and are provided to those skilled in the art. Therefore, it is self-evident that other drawings can be obtained based on these drawings under the premise that creative labor is not spent.
1 is a flowchart of an information supplementation method provided by an embodiment of the present application.
2 is a flowchart of another information supplementation method provided by an embodiment of the present application.
3 is a structural diagram of an information complementation model provided by an embodiment of the present application.
4 is a flowchart of a lane recognition method provided by an embodiment of the present application.
5 is a flowchart of an intelligent driving method provided by an embodiment of the present application.
6 is a structural diagram of an information supplementing device provided by an embodiment of the present application.
7 is a structural diagram of a lane recognition apparatus provided by an embodiment of the present application.
8 is a structural diagram of an intelligent driving device provided by an embodiment of the present application.
9 is a block diagram of a functional unit of an information supplementing device provided by an embodiment of the present application.
10 is a block diagram of a functional unit of a lane recognition apparatus provided by an embodiment of the present application.
11 is a block diagram of a functional unit of an intelligent driving device provided by an embodiment of the present application.

Hereinafter, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. It is apparent that the described embodiments are not all of the examples, but only some examples of the present application. According to the embodiment in the present application, all other embodiments obtained by a person skilled in the art without creative labor fall within the protection scope of the present application.

The terms "first", "second", "third", and "fourth" in the specification and claims of the present application and the accompanying drawings are for distinguishing different objects from each other, and not for describing a specific order. . Also, the terms "comprise" and "comprises" and any variations thereof are intended to encompass non-exclusive inclusions, e.g., a process, method, system, product or apparatus comprising a series of steps or units. is not limited to already listed steps or units, and may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to such a process, method, product or device. .

"Example" referred to in the text means that a specific feature, result or characteristic described in combination with the example may be included in at least one embodiment of the present application. The appearance of the word in each position in the specification does not necessarily refer to the same embodiment, nor does it refer to an independent or selected embodiment that is mutually exclusive from other embodiments. Those skilled in the art may explicitly or implicitly understand that the embodiments described herein may be combined with other embodiments.

LiDAR emits a laser beam to a target (observation point), receives an echo signal that is reflected back from the target, compares the echo signal with the firing signal to obtain the target point cloud data, and the point cloud data includes the target's point cloud data. Depth information (XYZ coordinate information), number of echoes, intensity information, RGB pixel values, scan angles, scan directions, and the like are included. LiDAR has limited vertical and horizontal angular resolution, so point cloud data is very sparse. In order to fully utilize the point cloud data, the sparse depth information in the point cloud data is first supplemented with dense depth information. However, the point cloud data contains more intensity information, and since the intensity information is related to other factors such as the incident angle of the LiDAR, the distance to the target, and the surface material of the target, the intensity information cannot be supplemented, and furthermore, the intensity information is not available. The information becomes unavailable, resulting in low utilization of the measurement data of the LiDAR. Since the intensity information cannot be used, when the lane is recognized only with the depth information after supplementation, the environmental luminance may affect the acquisition of the depth information, and thus the lane recognition rate may be lowered and robustness may be deteriorated. In order to solve the above shortcomings, the technical solution of the present application is specially proposed to realize synchronous complementation of scarce depth information and intensity information of point cloud data, and to improve the robustness of lane recognition.

Referring to FIG. 1 , FIG. 1 is a flowchart of an information supplementation method provided by an embodiment of the present application, the method including but not limited to the following steps.

101: Acquiring point cloud data of multiple frames.

Here, the multi-frame point cloud data is point cloud data obtained by continuously collecting, that is, the point cloud of the multi-frame may be neighboring point cloud data or may be point cloud data obtained non-continuously. In the present application, the neighboring point cloud data will be described in detail as an example.

In addition, the frame point cloud data includes depth information and intensity information.

102: According to depth information and intensity information of at least one point cloud data other than the reference point cloud data among the multi-frame point cloud data, supplement the reference point cloud data, and target point cloud data corresponding to the reference point cloud data step to obtain.

The reference point cloud data is any one point cloud data among multi-frame point cloud data, and the at least one point cloud data includes all point clouds among the point cloud data except for the reference point cloud data among the point cloud data of the multi-frame. data or some point cloud data.

In general, the reference point cloud data is point cloud data placed in an intermediate position of the point cloud data of the multiple frames, and accordingly, it has the same amount of point cloud data before and after the reference point cloud data. , it is better to supplement the reference point cloud data with the information-dense target point cloud data.

Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data.

In addition, the target point cloud data includes a target depth map and a target intensity map. The depth information in the target depth map is denser than the depth information corresponding to the reference point cloud data, and the intensity information in the target intensity map is denser than the intensity information in the intensity map corresponding to the reference point cloud data.

Hereinafter, a process of supplementing depth information and intensity information for reference point cloud data will be described in detail.

Referring to FIG. 2 , FIG. 2 is a flowchart of another information supplementation method provided by an embodiment of the present application, the method including but not limited to the following steps:

201: Re-projecting depth information of the point cloud data of each frame in the point cloud data set to obtain a depth map corresponding to the point cloud data of each frame in the point cloud data set.

The point cloud data set includes the point cloud data of the at least one frame and the reference point cloud data.

Re-projection is performed on the point cloud data of each frame in the point cloud set to obtain a depth map and an intensity map corresponding to the point cloud data of each frame. Since re-projection is a conventional technique, a description thereof will be omitted.

202: According to the depth map corresponding to the point cloud data of each frame in the point cloud data set, performing depth information supplementation on the reference point cloud data to obtain a target depth map corresponding to the reference point cloud data step.

Optionally, since the laser radar may change the position and attitude when collecting the point cloud data of each frame, the radar coordinate system when collecting the point cloud data of each frame is different, and thus the at least one point cloud data acquiring a first radar coordinate system corresponding to the point cloud data of each frame, and acquiring a reference radar coordinate system corresponding to the reference point cloud data; obtaining a corresponding respective first transformation matrix when transforming each first radar coordinate system into the reference radar coordinate system; transforming the depth map of the point cloud data corresponding to the first transformation matrix among the at least one point cloud data according to each first transformation matrix, that is, converting the depth map of the point cloud data into a reference radar coordinate system, acquiring a first depth map corresponding to the point cloud data of each frame among the at least one point cloud data by acquiring a first depth map corresponding to the point cloud data; Thereafter, a first depth map corresponding to the point cloud data of each frame among the at least one point cloud data is superimposed on the depth map corresponding to the reference point cloud data to obtain the reference point cloud data.

It should be explained that a depth map of the point cloud data of each frame among the at least one point cloud data is overlapped with the depth map of the reference point cloud data to obtain a new depth map. If the depth map of the point cloud data of the i-th frame among the at least one point cloud data is superimposed, the latest depth map already includes depth information of one observation point in the point cloud data depth map of the i-th frame, Ignoring the overlap of the depth information of the viewpoint, if the depth information of the viewpoint is not included, the depth information of the viewpoint is superimposed on the latest depth map to obtain a current latest depth map, wherein the at least one point cloud If all depth maps corresponding to the data are completely overlapped, a target depth map corresponding to the reference point cloud data may be obtained.

203: Acquiring a target intensity map corresponding to the reference point cloud data by supplementing intensity information with respect to the reference point cloud data according to the target depth map corresponding to the reference point cloud data.

As can be seen, in the embodiment of the present application, the collected sparse point cloud data is supplemented with dense point cloud data by supplementing the depth information and intensity information in the point cloud data; Since the depth information and the intensity information are synchronously supplemented, the two pieces of information are referred to each other in the supplementation process, and furthermore, the accuracy of information supplementation is improved, and supplementary errors can be avoided; In addition, the utilization rate of the point cloud data resource is improved by implementing the supplementation of the intensity information in the point cloud data.

In some possible implementations, distance correction is performed on the intensity map corresponding to the point cloud data of each frame in the point cloud data set to obtain a first intensity map corresponding to the point cloud data of each frame in the point cloud data set. obtain; Thereafter, angle correction is performed on the first intensity map corresponding to the point cloud data of each frame in the point cloud data set to obtain a second intensity map corresponding to the point cloud data of each frame in the point cloud data set, ; Finally, by overlapping the second intensity map corresponding to the point cloud data of the at least one frame and the second intensity map corresponding to the reference point cloud data, a target intensity map corresponding to the reference point cloud data is obtained. can

Here, when the distance correction and the angle correction are performed on the point cloud data of each frame, there is no forward and backward sequence. For example, the distance compensation may be performed first and then the angle compensation may be performed, the angle compensation may be performed first, and then the distance compensation may be performed, or the angle compensation and the distance compensation may be performed simultaneously.

)을 확정한다. 여기서, 상기 거리 보정항(g(D)) 및 제1 강도 맵(

)은 공식 (1)을 만족시킨다:In some possible implementations, a distance correction term may be obtained according to a radar parameter of the laser radar; Based on the distance correction term (g(D)), a first intensity map (

) is confirmed. Here, the distance correction term (g(D)) and the first intensity map (

) satisfies formula (1):

(1)

(One)

는 상기 포인트 클라우드 집합 중의 하나의 프레임의 포인트 클라우드 데이터의 강도 맵이고, D는 상기 프레임의 포인트 클라우드 데이터의 깊이 맵이며,

,

와

는 레이저 레이더의 고유 파라미터이다.

is the intensity map of the point cloud data of one frame of the point cloud set, D is the depth map of the point cloud data of the frame,

,

Wow

is an intrinsic parameter of laser radar.

Since the intensity information of each observation point (the surface of the object irradiated with the laser radar beam) is related to the distance (depth information) of each observation point and the laser radar, therefore, there is one distance for the intensity map of the point cloud data of each frame. A correction term, that is, a parameter opposite to the effect of the distance, is added, and the influence of the distance on the intensity information is removed through the distance correction term. For example, if the intensity information of each observation point and the distance show a direct proportional relationship, the effect of the distance on the intensity information of each observation point is removed through inverse proportion. Accordingly, intensity information of viewpoints having different distances in the first intensity map is independent of the distance.

In addition, after performing distance correction to obtain a first intensity map corresponding to the point cloud data of each frame among the point cloud data set, each of the target depth map corresponding to the reference point cloud data and the point cloud data set The target intensity map corresponding to the reference point cloud data may be determined according to the first intensity map corresponding to the point cloud data of the frame.

)를 획득한 후, 상기 기준 포인트 클라우드 데이터의 목표 깊이 맵 중 각 관측점의 깊이 좌표(X, Y, Z)와 법선 벡터(

)에 따라, 상기 기준 포인트 클라우드 데이터의 목표 깊이 맵 중 각 관측점의 입사각을 획득하며, 이에 따라 상기 기준 포인트 클라우드 데이터의 목표 깊이 맵에 대응되는 목표 입사각 매트릭스(

)를 획득할 수 있으며, 상기 목표 입사각 매트릭스(

)가 바로 기준 포인트 클라우드 데이터의 제1 입사각 매트릭스이기도 하다.More specifically, depth information of each observation point in the depth map, ie, (X, Y, Z) coordinates, indicates a spatial position of the observation point. Accordingly, in the reference radar coordinate system, plane fitting is performed for each observation point on the target depth map of the reference point cloud data to obtain a target space plane and a direction of the target space plane, and based on the direction of the target space plane, the target space The normal vector of the plane (

), the depth coordinates (X, Y, Z) of each observation point in the target depth map of the reference point cloud data and the normal vector (

), acquire the incidence angle of each observation point in the target depth map of the reference point cloud data, and accordingly, the target incidence angle matrix (

) can be obtained, and the target angle of incidence matrix (

) is also the first incident angle matrix of the reference point cloud data.

이고, 법선 벡터가

이면, 즉 상기 관측점의 입사각은

이고, 또한

이다.For example, if the depth information of the viewpoint is

, and the normal vector is

If , that is, the angle of incidence of the observation point is

is, and also

to be.

Thereafter, the target incidence angle matrix is inversely transformed according to a first transformation matrix (ie, a transformation matrix between two radar coordinate systems) of the point cloud data of each frame among the point cloud data of the at least one frame. That is, inverse transform is performed on the target incident angle matrix using the inverse matrix corresponding to the first transformation matrix to obtain a first incident angle matrix corresponding to the point cloud data of the frame, and through this, each of the point cloud sets A first incident angle matrix of the point cloud data of the frame may be obtained. Each incident angle in the first incident angle matrix is a narrow angle between the incident ray of the laser radar and the normal of the target plane when the point cloud data of each frame in the point cloud data set is collected, and the target plane is in the point cloud data of the frame. The space plane in which all corresponding viewpoints are located, for example the target space plane.

In addition, the intensity information of each observation point may be affected by the angle of incidence, that is, the observation point having the same material and the same distance from the laser radar may have different intensity information when the incident angle is different.

Accordingly, angle correction is performed on the first intensity map of the point cloud data of each frame in the point cloud set using the first incident angle matrix corresponding to the point cloud data of each frame in the point cloud set, and the point cloud A second intensity map of the point cloud data of each frame in the set is obtained, that is, the influence of an angle (incident angle) on intensity information is removed from the second intensity map.

More specifically, by performing angle correction on the first intensity map of the point cloud data through a first incident angle matrix and a preset constraint coefficient of the point cloud data of each frame in the point cloud set, the point cloud data A second intensity map of the point cloud data of each frame in the set is obtained. Here, the second intensity map satisfies formula (2):

(2);

(2);

는 포인트 클라우드 데이터 집합 중의 하나의 프레임의 포인트 클라우드 데이터의 제2 강도 맵이고,

는 상기 프레임의 포인트 클라우드 데이터에 대응되는 제1 강도 맵이며,

는 상기 프레임의 포인트 클라우드 데이터에 대응되는 제1 입사각 매트릭스이고,

와

은 사전설정된 구속계수이다.

is a second intensity map of the point cloud data of one frame of the point cloud data set,

is a first intensity map corresponding to the point cloud data of the frame,

is a first incident angle matrix corresponding to the point cloud data of the frame,

Wow

is a preset constraint factor.

Since the intensity information corresponding to each observation point is related to the cosine value of the angle of incidence, by correcting the intensity information of each observation point in a manner opposite to the cosine value, the effect of the angle of incidence on the intensity information of each observation point is eliminated, Accordingly, the intensity information corresponding to each observation point in the second intensity map is related only to the surface material of the observation point, and is no longer affected by distance and angle. That is, the intensity information of each observation point is only related to the reflectance of the observation point.

In addition, after obtaining the second intensity map corresponding to the point cloud data of each frame, the second intensity map corresponding to the point cloud data of each frame among the point cloud data of the at least one frame and corresponding to the reference point cloud data A target intensity map corresponding to the reference point cloud data is obtained by overlapping the second intensity map.

)을 획득한다.Similarly, the latest intensity map is obtained by overlapping a second intensity map of the point cloud data of each frame among the point cloud data of the at least one frame with the second intensity map corresponding to the reference point cloud data. When the second intensity map of the point cloud data of the i-th frame among the point cloud data of the at least one frame is superimposed, the latest intensity map already contains a view point of one of the second intensity maps of the point cloud data of the i-th frame. If the intensity information is included, that is, the overlap of the intensity information of the observation point is ignored, and if the intensity information of the observation point is not included, the intensity information of the observation point is superimposed on the latest intensity map, and the current latest intensity map is obtained, and if all second intensity maps corresponding to the point cloud data of the at least one frame are all overlaid, the target intensity map corresponding to the reference point cloud data (

) is obtained.

) 중의 관측점은 표면 재질만 같다면, 관측점과 레이저 레이더의 거리가 얼마이든, 입사각이 얼마이든, 그 대응되는 강도 정보는 동일하다는 것을 이해할 수 있다. 그러나, 실제 수집 시, 거리가 다르고, 입사각이 다른 관측점은 강도 정보가 상이하기 때문에, 이에 대응되는 강도 정보가 다르다. 만약 강도 맵을 직접 중첩할 수 있다면, 중첩 후의 강도 맵 중 재질이 동일한 관측점이지만, 만약 레이저 레이더와의 사이의 거리가 다르고, 입사각이 다를 경우, 이들에 대응되는 강도 정보가 다르기 때문에, 이를 통해 실제 강도 맵을 획득한다. 따라서, 목표 강도 맵(

)을 획득한 후, 목표 강도 맵(

)에 대해 역정규화 처리를 수행하며, 즉 상기 목표 입사각 매트릭스 및 사전 설정된 구속계수에 따라 상기 목표 강도 맵(

)에 대해 역정규화 처리를 실시하여, 상기 기준 포인트 클라우드 데이터에 대응되는 오리지널 강도 맵(

)을 획득하며, 오리지널 강도 맵(

)은 상기 적어도 하나의 프레임의 포인트 클라우드 데이터 중 각 프레임의 포인트 클라우드 데이터의 강도 맵과 상기 기준 포인트 클라우드 데이터의 강도 맵을 직접 중첩시켜 획득된 강도 맵에 해당한다. 여기서, 상기 오리지널 강도 맵(

)은 공식 (3)을 만족시킨다:The target intensity map (

), if only the surface material is the same, it can be understood that the corresponding intensity information is the same regardless of the distance between the observation point and the laser radar or the incident angle. However, during actual collection, since the intensity information is different at observation points having different distances and different incident angles, intensity information corresponding thereto is different. If the intensity map can be directly superimposed, the material is the same observation point in the intensity map after superposition, but if the distance between the laser radar and the laser radar is different and the incident angle is different, the corresponding intensity information is different. Obtain a strength map. Thus, the target intensity map (

), after obtaining the target intensity map (

) is denormalized, that is, according to the target angle of incidence matrix and a preset constraint factor, the target intensity map (

) is denormalized, and the original intensity map (

), and the original intensity map (

) corresponds to an intensity map obtained by directly overlapping the intensity map of the point cloud data of each frame among the point cloud data of the at least one frame and the intensity map of the reference point cloud data. Here, the original intensity map (

) satisfies formula (3):

(3);

(3);

는 목표 입사각 매트릭스이고,

는 목표 강도 맵이며,

와

은 사전 설정된 구속계수이다.here,

is the target angle of incidence matrix,

is the target intensity map,

Wow

is a preset constraint factor.

)을 획득한 후, 상기 오리지널 강도 맵(

)에 대해 가시화 프리젠테이션을 수행하여 포인트 클라우드 데이터의 강도 정보를 직접 보완한 후의 보완 결과를 표시함으로써 강도 정보의 보완을 구현할 수 있다.Optionally, the original intensity map (

), after obtaining the original intensity map (

), complementing the intensity information can be implemented by directly supplementing the intensity information of the point cloud data and displaying the supplementary results.

In some possible implementations, the information supplementation for the reference point cloud data is performed using a neural network, which is obtained by training the neural network in the following steps:

Step a: performing information supplementation on the sample point cloud data, and outputting the supplemented sample point cloud data;

Step b: according to the supplemented sample point cloud data, obtaining a predicted target depth map, a predicted target intensity map, and a predicted original intensity map of the sample point cloud data;

Step c: a labeled target depth map, a labeled target intensity map and a labeled original intensity map of the sample point cloud data, and a predicted target depth map, a predicted target intensity map and a predicted original intensity map of the sample point cloud data adjusting network parameters of the neural network according to the map.

Specifically, by first obtaining the depth information and intensity information supplementation result of the sample data, the target depth map, the target intensity map, and the original intensity map corresponding to the sample point cloud data, that is, the labeled target depth map and the labeled target It is necessary to obtain an intensity map and a labeled original intensity map (map label), where the complementary result may be implemented using other methods. Thereafter, the sample point cloud data is input to the initial neural network to obtain a target depth map, a target intensity map and an original intensity map after supplementation, that is, a predicted target intensity map, a predicted target intensity map, and a predicted original intensity map (prediction result), ; A target loss value is obtained according to the prediction result and the map label. That is, a first loss value corresponding to the target depth map after supplementation (determined according to the labeled target depth map and the predicted target depth map), a second loss value corresponding to the target intensity map (the labeled target intensity map and according to the predicted target intensity map), and a third loss value corresponding to the original intensity map (determined according to the labeled original intensity map and the predicted original intensity map), a target loss value is obtained. Here, the target loss value satisfies formula (4):

(4)

(4)

는 제1 손실값이고,

은 제2 손실값이며,

은 제3 손실값이고,

와

는 사전 설정 가중계수이다.here,

is the first loss value,

is the second loss value,

is the third loss value,

Wow

is a preset weighting factor.

The neural network is obtained by training the initial neural network through gradient descent and the target loss.

In some possible implementations, the structure of the neural network provided by the embodiment of the present invention may adopt the information beam and model shown in FIG. 3 . The information complementation model is a multitask learning model structure, and the information complementation model includes a predictive network block and a denormalized network block. here,

The prediction network block includes an encoding network block, a convolutional network block, and a decoding network block. The encoding network is composed of 4 residual networks, the convolutional network block is composed of a convolutional layer and an expansion network, and the decoding network block is composed of 5 pre-convolutional networks.

)과 예측 목표 강도 맵(

)을 획득하기 위한 것이고;Here, the prediction network block performs information supplementation on the input sample point cloud data through the encoding network block, the convolutional network block, and the decoding network block, and a prediction target depth map corresponding to the sample point cloud data (

) and the predicted target intensity map (

) to obtain;

)에 대해 역정규화 처리를 실시하여, 예측 초기 강도 맵(

)을 획득하기 위한 것이며;The denormalized network block is a prediction target intensity map (

) is denormalized to the predicted initial intensity map (

) to obtain;

Thereafter, a first loss value, a second loss value, and a third loss value corresponding to the sample point cloud data are calculated based on the target depth map, the intermediate intensity map, and the target intensity map, which are map information of the sample point cloud data; Weight processing is performed on the first loss value, the second loss value, and the third loss value to obtain a target loss value, and optimization training is performed on the information supplementation model based on the target loss value.

As can be seen, the collected sparse point cloud data is supplemented with dense point cloud data by supplementing the depth information and intensity information in the point cloud data through the complementary model; Since the depth information and the intensity information are supplemented at the same time, the two pieces of information can be referred to each other in the supplementation process, thereby improving the accuracy of information supplementation and preventing supplementation errors; In addition, by implementing a supplement to the intensity information in the point cloud data, the utilization rate of the point cloud data resource is improved.

Referring to FIG. 4 , FIG. 4 is a flowchart of a lane recognition method provided by an embodiment of the present application, the method including but not limited to the following steps:

401: Collecting point cloud data of the road.

Here, it is possible to collect point cloud data of the road through laser radar.

402: Acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data.

Here, by performing information supplementation on the point cloud data using the information supplementation method, a target intensity map corresponding to the point cloud data is obtained, and the target intensity map shows the effect of distance and incident angle on intensity information. As the removed target intensity map, the detailed supplementation process may refer to the above contents in 203, and thus description thereof will be omitted.

403: Recognizing a lane according to the target intensity map to obtain a lane recognition result.

Since the influence of distance and incident angle is removed from the intensity information in the target intensity map, the intensity information of the observation point in the target intensity map is only related to the surface material of the observation point, that is, the intensity information of the observation point in the target intensity map. reflects the surface reflectance of the material corresponding to the observation point. Accordingly, as long as all observation points corresponding to the intensity information (surface reflectance) and the intensity information (surface reflectance) of the lane among the target intensities are determined as points on the lane, the lane can be recognized through the target intensity map.

Accordingly, it can be seen that, in the embodiment of the present application, a lane can be recognized using the target intensity map after supplementation. Since the laser does not belong to the visible light band, even in dark or poor conditions, a target intensity map with dense intensity information can be obtained, improving the stability and success rate of lane recognition.

5 is a flowchart of an intelligent driving method provided by an embodiment of the present application, the method including but not limited to the following steps:

501: Collecting point cloud data of the surrounding scene of the intelligent driving device.

Here, the intelligent driving device may be an autonomous vehicle, a vehicle equipped with an Advanced Driving Assistant System (ADAS), a smart robot, or the like.

502: Acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data.

By supplementing the intensity information of the point cloud data using the information supplementation method, a target intensity map in which the effects of distance and incident angle are removed is obtained. .

503: Recognizing a target object in the surrounding scene according to the target intensity map.

Here, the target may be a lane, various traffic lights, traffic signs, obstacles, passers-by or other intelligent driving devices, and various objects that may be encountered while driving.

Specifically, after obtaining the intensity information of each observation point in the target intensity map, that is, the surface reflectance of each observation point, the surface reflectance of each observation point is compared with the surface reflectance corresponding to various objects, and the object type of each observation point is determined is determined, and accordingly, the target object in the surrounding scene is recognized.

504: Controlling driving of the intelligent driving device according to the recognition result.

Optionally, based on the recognized target object in the surrounding scene, plan a route, and automatically control the driving of the intelligent device according to the planned driving route; A guide message may be output to inform the operator of the intelligent driving device to control driving of the intelligent device based on the recognized target target through the guidance message.

Accordingly, in the implementation method of the present application, it can be seen that the target intensity map is obtained by supplementing the collected point cloud data. Since the laser does not belong to the visible light band, even in dark or poor conditions, it is possible to obtain a target intensity map with dense intensity information; In addition, since the reflectivity of the observation point of the target intensity map is not affected by the distance and the incident angle, when the target object is recognized according to the target intensity map, the recognition success rate of the target object may be increased. Therefore, it is possible to accurately obtain the layout of the environment around the intelligent driving device under any environmental conditions, control the driving of the intelligent driving device according to the environment layout, and further improve the driving stability of the intelligent driving device.

6 is a structural diagram of an information supplementing apparatus 600 provided by an embodiment of the present application. The information complementing device 600 includes a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are different from the one or more application programs, and the one or more programs include: stored in the memory and configured to be executed by the processor, the program comprising instructions for executing the following steps:

acquiring multi-frame point cloud data including depth information and intensity information in each frame;

Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame of the multi-frame point cloud data except for the reference point cloud data obtaining a;

Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data, and the reference point cloud data is any one point cloud data among the multi-frame point cloud data.

In some possible implementations, the reference point cloud data is supplemented according to the depth information and intensity information of the point cloud data of at least one frame excluding the reference point cloud data among the multiple frame point cloud data, and the reference point cloud data is added to the reference point cloud data. In terms of acquiring the corresponding target point cloud data, the program is specifically for executing the following instructions:

The depth information of the point cloud data of each frame among the point cloud data set composed of the point cloud data of the at least one frame and the reference point cloud data is re-projected to the point cloud data of each frame in the point cloud data set. obtaining a corresponding depth map;

acquiring a target depth map corresponding to the reference point cloud data by supplementing the reference point cloud data with depth information according to a depth map corresponding to the point cloud data of each frame in the point cloud data set;

Acquiring a target intensity map corresponding to the reference point cloud data by supplementing intensity information on the reference point cloud data according to the target depth map corresponding to the reference point cloud data.

In some possible implementation methods, depth information is supplemented on the reference point cloud data according to a depth map corresponding to the point cloud data of each frame in the point cloud data set, and a target depth corresponding to the reference point cloud data is performed. In terms of acquiring the map, the program is specifically for executing the following steps of instructions:

acquiring a first radar coordinate system of the point cloud data of each frame among the point cloud data of the at least one frame and a reference radar coordinate system corresponding to the reference point cloud data;

determining each first transformation matrix that transforms each first radar coordinate system into the reference radar coordinate system;

converting a depth map of the point cloud data of a corresponding frame according to each first transformation matrix to obtain a first depth map;

obtaining a target depth map corresponding to the reference point cloud data by overlapping a first depth map corresponding to the point cloud data of each frame among the point cloud data of the at least one frame with the depth map of the reference point cloud data step.

In some possible implementations, in terms of obtaining a target intensity map corresponding to the reference point cloud data by supplementing the intensity information on the reference point cloud data according to the target depth corresponding to the reference point cloud data, The program is specifically for executing the instructions of the following steps:

obtaining an intensity map corresponding to the point cloud data of each frame in the point cloud data set by re-projecting intensity information of the point cloud data of each frame in the point cloud data set;

Distance correction is performed to remove the effect of distance on intensity information on the intensity map corresponding to the point cloud data of each frame in the point cloud data set, and the point cloud data of each frame in the point cloud data set is subjected to distance correction. obtaining a corresponding first intensity map;

determining a target intensity map corresponding to the reference point cloud data according to a target depth map corresponding to the reference point cloud data and a first intensity map corresponding to the point cloud data of each frame among the point cloud data set.

In some possible implementations, distance correction is performed on the intensity map corresponding to the point cloud data of each frame in the point cloud data set, and the first intensity map corresponding to the point cloud data of each frame in the point cloud data set In the aspect of obtaining , the program is specifically for executing the instructions of the following steps:

obtaining a distance correction term according to a radar parameter of the laser radar;

Determining a first intensity map corresponding to the point cloud data of each frame among the point cloud data according to the distance correction term.

In some possible implementations, in terms of determining the target intensity map corresponding to the reference point cloud data according to the target depth map and the first intensity map corresponding to the point cloud data of each frame in the point cloud data set, The program is specifically for executing the instructions of the following steps:

obtaining a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set according to the target depth map corresponding to the reference point cloud data;

According to the first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, for the first intensity map corresponding to the point cloud data of each frame in the point cloud data set, the incident angle is the intensity information. obtaining a second intensity map corresponding to the point cloud data of each frame in the point cloud data set by performing angle correction to remove the influence;

A target intensity map corresponding to the reference point cloud data is obtained by overlapping a second intensity map corresponding to each point cloud data among the point cloud data of the at least one frame and a second intensity map corresponding to the reference point cloud data. steps to obtain.

In some possible implementations, in terms of obtaining, according to the target depth map corresponding to the reference point cloud data, a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, the program is specifically This is to run the following commands:

acquiring a target space plane by performing plane fitting according to depth information in a target depth map corresponding to the reference point cloud data;

determining a target depth map corresponding to the reference point cloud data and a target incidence angle matrix corresponding to the reference point according to an angle between an incident ray of a laser radar and a normal of a target plane when collecting the reference point cloud data, wherein the reference point the target depth map corresponding to the cloud data and the target incidence angle matrix corresponding to the cloud data are the first incidence angle matrix of the reference point cloud data;

The target depth map corresponding to the reference point cloud data and the corresponding target incidence angle matrix are inversely transformed according to the first transformation matrix of the point cloud data of the at least one frame, so that each frame of the point cloud data of the at least one frame is inversely transformed. Obtaining a first angle of incidence matrix of the point cloud data.

In some possible implementations, according to a first incidence angle matrix corresponding to the point cloud data of each frame in the set of point cloud data, an angle with respect to the first intensity map corresponding to the point cloud data of each frame in the set of point cloud data. In terms of performing correction to obtain a second intensity map corresponding to the point cloud data of each frame in the point cloud data set, the program is specifically for executing the following instructions:

According to a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, a preset constraint factor, and a first intensity map of the point cloud data of each frame in the point cloud data set, the point cloud data Determining a second intensity map corresponding to the point cloud data of each frame in the set.

In some possible implementations, the program is specifically for further executing the instructions of the following steps:

determining an original intensity map corresponding to the reference point cloud data according to the target intensity map corresponding to the reference point cloud data.

In some possible implementation manners, in terms of determining the original intensity map corresponding to the reference point cloud data according to the target intensity map corresponding to the reference point cloud data, the program is specifically for executing the instructions of the following steps :

By processing the target intensity map corresponding to the reference point cloud data according to the target depth map corresponding to the reference point cloud data, the target incident angle matrix corresponding to the reference point cloud data, and a preset constraint factor, the original intensity corresponding to the reference point cloud data Step to confirm the map.

In some possible implementations, the information supplementation device supplements the reference point cloud via a neural network; The program is also for executing the following steps of instructions:

performing information supplementation on the sample point cloud data and outputting the supplemented sample point cloud data;

obtaining, according to the supplemented sample point cloud data, a predicted target depth map, a predicted target intensity map, and a predicted original intensity map of the sample point cloud data;

of the neural network according to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data. Adjusting network parameters.

In some possible implementations, a labeled target depth map of the sample point cloud data, a labeled target intensity map and a labeled original intensity map, and a predicted target depth map, a predicted target intensity map and a predicted original intensity of the sample point cloud data In terms of adjusting the network parameters of the neural network according to the map, the program is specifically for executing the following steps:

According to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data, a first obtaining a loss value, a second loss value, and a third loss value; wherein the first loss value is determined according to the labeled target depth map and the predicted target depth map; the second loss value is determined according to the labeled target intensity map and the predicted target intensity map; the third loss value is determined according to the labeled original intensity map and the predicted original intensity map;

obtaining a target loss value by weighting the first loss value, the second loss value, and the third loss value;

adjusting network parameters of the neural network according to the target loss value.

7 is a structural diagram of the lane recognition device 700 provided by the embodiment of the present application. The lane recognition device 700 includes a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are different from the one or more application programs, and the one or more programs are stored in the memory and configured to be executed by the processor, the program comprising:

collecting point cloud data of the road;

acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data;

and recognizing a lane according to the target intensity map to obtain a lane recognition result.

8 is a structural diagram of the intelligent driving device 800 provided by the embodiment of the present application. The intelligent driving device 800 includes a processor, a memory, a communication interface and one or more programs, wherein the one or more programs are different from the one or more application programs, and the one or more programs are stored in the memory and configured to be executed by the processor, the program comprising:

collecting point cloud data of surrounding scenes;

acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data;

recognizing a target object in the surrounding scene according to the target intensity map;

Controlling driving according to the recognition result; includes a command for executing.

9 is a block diagram of a functional unit of an information supplementing device provided by an embodiment of the present application. The information supplementation device 900 includes an acquisition unit 910 and a supplementation unit 920, where,

The acquiring unit 910 is for acquiring point cloud data of multiple frames including depth information and intensity information in the point cloud data of each frame;

The complementation unit 920 supplements the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame except for the reference point cloud data among the point cloud data of the multiple frames, to acquire target point cloud data corresponding to the data;

Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data, and the reference point cloud data is the point cloud data of any one frame among the point cloud data of the multiple frames.

In some possible implementations, the supplemental unit 920 specifically,

The depth information of the point cloud data of each frame among the point cloud data set composed of the point cloud data of at least one frame and the reference point cloud data is re-projected to correspond to the point cloud data of each frame in the point cloud data set obtain a depth map to be;

performing depth information supplementation on the reference point cloud data according to a depth map corresponding to the point cloud data of each frame in the point cloud data set to obtain a target depth map corresponding to the reference point cloud data;

According to the target depth map corresponding to the reference point cloud data, the intensity information is supplemented on the reference point cloud data to obtain a target intensity map corresponding to the reference point cloud data.

In some possible implementation methods, depth information is supplemented on the reference point cloud data according to a depth map corresponding to the point cloud data of each frame in the point cloud data set, and a target depth corresponding to the reference point cloud data is performed. In terms of acquiring the map, the supplementary unit 920 is specifically,

acquiring a first radar coordinate system of the point cloud data of each frame among the point cloud data of the at least one frame and a reference radar coordinate system corresponding to the reference point cloud data;

determine respective first transformation matrices for transforming each first radar coordinate system into the reference radar coordinate system;

transforming the depth map of the point cloud data of the corresponding frame according to each first transformation matrix to obtain a first depth map;

obtaining a target depth map corresponding to the reference point cloud data by overlapping a first depth map corresponding to the point cloud data of each frame among the point cloud data of the at least one frame with the depth map of the reference point cloud data; it is for

In some possible implementations, in terms of obtaining a target intensity map corresponding to the reference point cloud data by supplementing the intensity information on the reference point cloud data according to the target depth corresponding to the reference point cloud data, Complementary unit 920 is specifically,

re-projecting intensity information of the point cloud data of each frame in the point cloud data set to obtain an intensity map corresponding to the point cloud data of each frame in the point cloud data set;

Distance correction is performed to remove the effect of distance on intensity information on the intensity map corresponding to the point cloud data of each frame in the point cloud data set, and the point cloud data of each frame in the point cloud data set is subjected to distance correction. obtain a corresponding first intensity map;

to determine a target intensity map corresponding to the reference point cloud data according to a target depth map corresponding to the reference point cloud data and a first intensity map corresponding to the point cloud data of each frame among the point cloud data set .

In some possible implementations, distance correction is performed on the intensity map corresponding to the point cloud data of each frame in the point cloud data set, and a first intensity map corresponding to the point cloud data of each frame in the point cloud data set is performed. In terms of obtaining , the supplementary unit 920 is specifically,

obtain a distance correction term according to the radar parameter of the laser radar;

According to the distance correction term, the first intensity map corresponding to the point cloud data of each frame among the point cloud data set is determined.

In some possible implementations, in terms of determining the target intensity map corresponding to the reference point cloud data according to the target depth map and the first intensity map corresponding to the point cloud data of each frame among the point cloud data set, Complementary unit 920 is specifically,

obtaining a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set according to the target depth map corresponding to the reference point cloud data;

According to the first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, for the first intensity map corresponding to the point cloud data of each frame in the point cloud data set, the incident angle is the intensity information. performing angle correction to remove the influence to obtain a second intensity map corresponding to the point cloud data of each frame among the point cloud data set;

A second intensity map corresponding to the point cloud data of each frame among the point cloud data of the at least one frame is overlapped with a second intensity map corresponding to the reference point cloud data, and a target intensity corresponding to the reference point cloud data is overlapped. to get a map.

In some possible implementations, according to the target depth map corresponding to the reference point cloud data, in the aspect of obtaining the first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, the complementation unit 920 is specifically,

performing plane fitting according to depth information in the target depth map corresponding to the reference point cloud data to obtain a target space plane;

When collecting the reference point cloud data, according to an angle between the incident ray of the laser radar and the normal of the target plane, a target depth map corresponding to the reference point cloud data, which is the first incident angle matrix of the reference point cloud data, and a target angle of incidence determine the matrix,

The at least one frame by inversely transforming a target depth map corresponding to the reference point cloud data and a target incidence angle matrix corresponding to the reference point cloud data according to a first transformation matrix of the point cloud data of each frame among the point cloud data of the at least one frame to obtain a first incident angle matrix of the point cloud data of each frame among the point cloud data of .

In some possible implementations, according to a first incidence angle matrix corresponding to the point cloud data of each frame in the point cloud data set, an angle with respect to the first intensity map corresponding to the point cloud data of each frame in the point cloud data set In terms of obtaining a second intensity map corresponding to the point cloud data of each frame in the point cloud data set by performing the correction, the complementation unit 920 specifically,

According to a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, a preset constraint factor, and a first intensity map of the point cloud data of each frame in the point cloud data set, the point cloud data This is to determine a second intensity map corresponding to the point cloud data of each frame in the set.

In some possible implementations, the complementing unit 920 is further configured to determine the original intensity map corresponding to the reference point cloud data according to the target intensity map corresponding to the reference point cloud data.

In some possible implementations, in terms of determining the original intensity map corresponding to the reference point cloud data according to the target intensity map corresponding to the reference point cloud data, the complementation unit 920 specifically,

By processing the target intensity map corresponding to the reference point cloud data according to the target depth map corresponding to the reference point cloud data, the target incident angle matrix corresponding to the reference point cloud data, and a preset constraint factor, the original intensity corresponding to the reference point cloud data This is to confirm the map.

In some possible implementations, the information supplementing device supplements the reference point cloud through a neural network; The information supplementing device 900 further includes a training unit 930, and the training unit 930 is

performing information supplementation on the sample point cloud data and outputting the supplemented sample point cloud data;

obtaining, according to the supplemented sample point cloud data, a predicted target depth map, a predicted target intensity map, and a predicted original intensity map of the sample point cloud data;

of the neural network according to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data. adjusting network parameters; to train the neural network by executing it.

In some possible implementations, a labeled target depth map of the sample point cloud data, a labeled target intensity map and a labeled original intensity map, and a predicted target depth map, a predicted target intensity map and a predicted original intensity map of the sample point cloud data In terms of adjusting the network parameters of the neural network according to the map, the training unit 930 specifically,

According to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data, a first obtain a loss value, a second loss value, and a third loss value; wherein the first loss value is determined according to the labeled target depth map and the predicted target depth map; the second loss value is determined according to the labeled target intensity map and the predicted target intensity map; the third loss value is determined according to the labeled original intensity map and the predicted original intensity map;

performing weight processing on the first loss value, the second loss value, and the third loss value to obtain a target loss value;

It is to adjust a network parameter of the neural network according to the target loss value.

Here, the training unit 930 is not essential for the information supplementation device.

10 is a block diagram of a functional unit of a lane recognition apparatus provided by an embodiment of the present application. The lane recognition device 1000 includes a collection unit 1100, a supplementary unit 1200, and a recognition unit 1300, where,

The collecting unit 1100 is for collecting the point cloud data of the road;

The supplementation unit 1200 is the information supplementation method of the first aspect, for supplementing point cloud data to obtain a target intensity map corresponding to the point cloud data;

The recognition unit 1300 is configured to recognize a lane according to the target intensity map and obtain a lane recognition result.

11 is a block diagram of a functional unit of an intelligent driving device provided by an embodiment of the present application. The intelligent driving device 1100 includes a collection unit 1110 , a complementation unit 1120 , a recognition unit 1130 , and a control unit 1140 , where,

the collecting unit 1210 is for collecting point cloud data of the surrounding scene;

The supplementation unit 1220 is configured to supplement the point cloud data by using the information supplementation method of the first aspect to obtain a target intensity map corresponding to the point cloud data;

the recognition unit 1230 is for recognizing a target object in the surrounding scene according to the target intensity map;

The control unit 1240 is for controlling driving according to the recognition result.

An embodiment of the present application further provides a computer storage medium, the computer program is stored in the computer readable storage medium, the computer program is executed by a processor to partially or all of the information supplementation method described in the embodiment of the method step, some or all of the steps of the lane recognition method, or some or all of the steps of the intelligent driving method.

An embodiment of the present application further provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium in which the computer program is stored. Through the execution of the computer program, the computer implements some or all of the steps of the information supplementation method described in the embodiment of the method, some or all of the steps of the lane recognition method, or some or all of the steps of the intelligent driving method.

It should be noted that, for the sake of brevity, the embodiments of each method described above are all expressed as a combination of a series of operations, but those skilled in the art should understand that the present application is not limited by the order of operations described. This is because, in accordance with the present application, certain steps may be performed using a different order or performed concurrently. Next, those skilled in the art should also understand that all of the embodiments described in the specification belong to any embodiment, and the related operations and modules are not necessarily essential to the present application.

In the above embodiment, the descriptions for each embodiment are all focused, and parts not described in detail in one embodiment may refer to the related technology of another embodiment.

In some embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other ways. For example, the embodiment of the device is merely exemplary, for example, the division of the unit is merely a division of a kind of logical function, and there may be a separate division method in actual implementation. For example, multiple units or assemblies may be combined or integrated into other systems, or some features may be omitted or not implemented. Further, the mutual coupling shown or discussed may be a direct coupling or a communication connection through some interface, and an indirect connection or communication connection of a device or unit may be in an electrical or other form.

The unit described as the separation member may or may not be physically separated, and the portion indicated as a unit may or may not be a physical unit. That is, it may be located in one place, or it may be distributed in a plurality of network units. According to actual needs, some or all of the units may be selected to implement the purpose of the scheme of the present embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist alone physically, and two or two or more units may be integrated into one unit. . The integrated unit may be implemented in the form of hardware or may be implemented in the form of a software program module.

When the integrated unit is implemented in the form of a software program module and sold or used as an independent product, it may be stored in one computer readable memory. According to this understanding, the technical solution of the present application essentially or a part contributing to the prior art, or all or part of the technical solution may be implemented in the form of a software product, wherein the computer software product is stored in one memory and , a computer device (which may be a personal computer, a server or a network device, etc.) and some instructions for executing all or some steps of the above method of each embodiment of the present application. The aforementioned memory includes a medium capable of storing various program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a removable hard disk, a magnetic disk, or an optical disk. .

A person skilled in the art can understand that all or some steps of the various methods of the above embodiments can be accomplished by instructing the relevant hardware through a program, and the program can be stored in a computer readable memory. The memory may include a flash disk, a read-only memory (English: Read-Only Memory, abbreviation: ROM), a random access memory (English: Random Access Memory, abbreviation: RAM), a magnetic disk or an optical disk.

The embodiments of the present application have been introduced in detail above, and the principles and implementation methods of the present application have been described by applying specific examples in the text. The description of the above embodiments is merely to help the understanding of the method of the present application and its core idea. At the same time, for those of ordinary skill in the art, based on the spirit of the present application, there may be changes within specific implementation methods and application ranges, and in conclusion, the content of the present specification should not be understood as limiting the present application. Can not be done.

Claims (19)

In the information supplement method,
acquiring multi-frame point cloud data including depth information and intensity information in each frame; and
Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame of the multi-frame point cloud data except for the reference point cloud data obtaining - supplementing the intensity information of the reference point cloud data using a depth map obtained from the depth information;
Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data, and the reference point cloud data is any one point cloud data among the multi-frame point cloud data. According to claim 1,
Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame of the multi-frame point cloud data except for the reference point cloud data The steps to obtain
obtaining a depth map corresponding to the point cloud data of each frame in the point cloud data set by re-projecting the depth information of the point cloud data of each frame in the point cloud data set;
acquiring a target depth map corresponding to the reference point cloud data by performing depth information supplementation on the reference point cloud data according to a depth map corresponding to the point cloud data of each frame in the point cloud data set;
acquiring a target intensity map corresponding to the reference point cloud data by supplementing intensity information with respect to the reference point cloud data according to a target depth map corresponding to the reference point cloud data; Way. 3. The method of claim 2,
Acquiring a target depth map corresponding to the reference point cloud data by performing depth information supplementation on the reference point cloud data according to a depth map corresponding to the point cloud data of each frame in the point cloud data set, ,
acquiring a first radar coordinate system of the point cloud data of each frame among the point cloud data of the at least one frame and a reference radar coordinate system corresponding to the reference point cloud data;
determining each first transformation matrix that transforms each first radar coordinate system into the reference radar coordinate system;
obtaining a first depth map by transforming a depth map of point cloud data of a corresponding frame according to each first transformation matrix;
obtaining a target depth map corresponding to the reference point cloud data by overlapping a first depth map corresponding to the point cloud data of each frame among the at least one point cloud data and a depth map of the reference point cloud data; A method comprising a. 3. The method of claim 2,
The step of obtaining a target intensity map corresponding to the reference point cloud data by performing intensity information supplementation on the reference point cloud data according to a target depth corresponding to the reference point cloud data,
obtaining an intensity map corresponding to the point cloud data of each frame in the point cloud data set by re-projecting intensity information of the point cloud data of each frame in the point cloud data set;
For the intensity map corresponding to the point cloud data of each frame in the point cloud data set, distance correction is performed to remove the effect of the distance on the intensity information, and the point cloud data of each frame in the point cloud data set obtaining a corresponding first intensity map;
determining a target intensity map corresponding to the reference point cloud data according to a target depth map corresponding to the reference point cloud data and a first intensity map corresponding to the point cloud data of each frame among the point cloud data set; A method comprising a. 5. The method of claim 4,
Obtaining a first intensity map corresponding to the point cloud data of each frame in the point cloud data set by performing distance correction on the intensity map corresponding to the point cloud data of each frame in the point cloud data set,
obtaining a distance correction term according to a radar parameter of the laser radar;
and determining a first intensity map corresponding to the point cloud data of each frame among the point cloud data sets according to the distance correction term. 5. The method of claim 4,
determining the target intensity map corresponding to the reference point cloud data according to the target depth map and the first intensity map corresponding to the point cloud data of each frame among the point cloud data set;
obtaining a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set according to the target depth map corresponding to the reference point cloud data;
According to the first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, for the first intensity map corresponding to the point cloud data of each frame in the point cloud data set, the incident angle is the intensity information. obtaining a second intensity map corresponding to the point cloud data of each frame in the point cloud data set by performing angle correction to remove the influence;
A target intensity map corresponding to the reference point cloud data is obtained by overlapping a second intensity map corresponding to each point cloud data among the point cloud data of the at least one frame and a second intensity map corresponding to the reference point cloud data. A method comprising: obtaining; 7. The method of claim 6,
obtaining a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set according to the target depth map corresponding to the reference point cloud data;
acquiring a target space plane by performing plane fitting according to depth information in a target depth map corresponding to the reference point cloud data;
determining a target depth map corresponding to the reference point cloud data and a target incidence angle matrix corresponding to the reference point according to the angle between the incident ray of the laser radar and the normal of the target plane when collecting the reference point cloud data, wherein the reference point the target depth map corresponding to the cloud data and the target incidence angle matrix corresponding to the cloud data are the first incidence angle matrix of the reference point cloud data;
The target depth map corresponding to the reference point cloud data and the corresponding target incidence angle matrix are inversely transformed according to the first transformation matrix of the point cloud data of the at least one frame, so that each frame of the point cloud data of the at least one frame is inversely transformed. and obtaining a first angle of incidence matrix of the point cloud data. 7. The method of claim 6,
In accordance with the first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, angle correction is performed on the first intensity map corresponding to the point cloud data of each frame in the point cloud data set, Acquiring a second intensity map corresponding to the point cloud data of each frame among the point cloud data set includes:
According to a first incident angle matrix corresponding to the point cloud data of each frame in the point cloud data set, a preset constraint factor, and a first intensity map of the point cloud data of each frame in the point cloud data set, the point cloud data and determining a second intensity map corresponding to the point cloud data of each frame in the set. 7. The method of claim 6,
the method
The original intensity corresponding to the reference point cloud data is processed by processing the target intensity map corresponding to the reference point cloud data according to the target depth map corresponding to the reference point cloud data, the target incident angle matrix corresponding to the reference point cloud data, and a preset constraint factor. A method comprising determining a map. According to claim 1,
The information supplementation method is implemented through a neural network; The neural network is
performing information supplementation on the sample point cloud data and outputting the supplemented sample point cloud data;
obtaining, according to the supplemented sample point cloud data, a predicted target depth map, a predicted target intensity map, and a predicted original intensity map of the sample point cloud data;
of the neural network according to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data. A method characterized in that the training is performed using; adjusting network parameters. 11. The method of claim 10,
of the neural network according to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data. The steps to adjust the network parameters are:
According to the labeled target depth map of the sample point cloud data, the labeled target intensity map and the labeled original intensity map, and the predicted target depth map, the predicted target intensity map and the predicted original intensity map of the sample point cloud data, a first obtaining a loss value, a second loss value, and a third loss value; wherein the first loss value is determined according to the labeled target depth map and the predicted target depth map; the second loss value is determined according to the labeled target intensity map and the predicted target intensity map; the third loss value is determined according to the labeled original intensity map and the predicted original intensity map;
obtaining a target loss value by weighting the first loss value, the second loss value, and the third loss value;
adjusting a network parameter of the neural network according to the target loss value. In the lane recognition method,
collecting point cloud data of the road;
acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data using the information supplementation method according to any one of claims 1 to 11;
and obtaining a lane recognition result by recognizing a lane according to the target intensity map. In the intelligent driving method,
collecting point cloud data of scenes around the intelligent driving device;
12. A method comprising: acquiring a target intensity map corresponding to the point cloud data by supplementing the point cloud data using the information supplementation method according to any one of claims 1 to 11;
recognizing a target object in the surrounding scene according to the target intensity map;
and controlling the driving of the intelligent driving device according to the recognition result. An information supplementing device comprising:
an acquisition unit for acquiring point cloud data of multiple frames including depth information and intensity information in each frame; and
Target point cloud data corresponding to the reference point cloud data by supplementing the reference point cloud data according to the depth information and intensity information of the point cloud data of at least one frame of the point cloud data of the multiple frames except for the reference point cloud data a complementation unit for obtaining, wherein the complementation of the intensity information of the reference point cloud data uses a depth map obtained from the depth information;
Here, the depth information of the target point cloud data is denser than the depth information of the reference point cloud data, and the intensity information of the target point cloud data is denser than the intensity information of the reference point cloud data, and the reference point cloud data is any one point cloud data among the multi-frame point cloud data. In the lane recognition device,
a collection unit for collecting road point cloud data;
a supplementation unit for supplementing the point cloud data using the information supplementation method according to any one of claims 1 to 11 to obtain a target intensity map corresponding to the point cloud data;
and a recognition unit configured to recognize a lane according to the target intensity map and obtain a lane recognition result. In the intelligent driving device,
a collecting unit for collecting point cloud data of scenes around the intelligent driving device;
a supplementation unit for supplementing the point cloud data by using the information supplementation method according to any one of claims 1 to 11 to obtain a target intensity map corresponding to the point cloud data;
a recognition unit for recognizing a target object in the surrounding scene according to the target intensity map;
and a control unit for controlling the driving of the intelligent driving device according to the recognition result. In an electronic device,
A processor, comprising: a memory, wherein the memory is for storing computer readable instructions, the processor calls the instructions stored in the memory to execute the method according to any one of claims 1 to 11 Electronic device, characterized in that for. A computer-readable storage medium comprising:
12. A computer-readable storage medium for storing a computer program, characterized in that when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 11. delete

Images