WO2020223868A1

WO2020223868A1 - Terrain information processing method and apparatus, and unmanned vehicle

Info

Publication number: WO2020223868A1
Application number: PCT/CN2019/085655
Authority: WO
Inventors: 刘晓洋; 郑杨杨; 张晓炜
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-05-06
Filing date: 2019-05-06
Publication date: 2020-11-12
Also published as: CN111712855A

Abstract

A terrain information processing method and apparatus, and an unmanned vehicle. The method comprises: obtaining N frames of depth maps acquired by a depth sensor (S301); performing terrain segmentation on each frame of depth maps to obtain a terrain area in each frame of depth maps (S302); according to the terrain area in each frame of depth maps, obtaining a height map of the terrain area in each frame of depth maps (S303); and according to the height map of the terrain area in each frame of depth maps, obtaining a fusion height map after N frames of height maps are fused, the fusion height map being used for terrain estimation of the terrain area (S304). Since the fusion height map fuses height information of the terrain area in the N frames of height maps, the influence of a single frame of height map on a height value of the terrain area is reduced, and the height value of the fusion height map is more approximate to the actual height value of the terrain area. Terrain estimation of the terrain area obtained by means of the fusion height map is more accurate.

Description

Ground information processing method, device and unmanned vehicle

Technical field

The embodiments of the present application relate to the field of unmanned driving technology, and in particular, to a ground information processing method, device, and unmanned vehicle.

Background technique

Unmanned driving technology has developed rapidly in recent years. Among them, ground model estimation technology is the basic technology in unmanned driving technology. After ground estimation, the estimated ground model can be used to convert the lane line from the camera perspective to the overhead perspective for subsequent follow-up The processing of lane lines can also be used to detect the passable area of unmanned vehicles, and it can also be used to detect the current posture of unmanned vehicles for map-based online positioning systems.

The current ground model estimation process is: obtain a multi-frame depth map from the camera's perspective when the vehicle is moving, and then perform ground segmentation on each frame of the depth map to obtain the ground area, and then perform ground based on the ground area of the depth map of each frame Estimate and obtain the ground model. However, this method will cause serious loss of ground information, resulting in low accuracy of the acquired ground model.

Summary of the invention

The embodiments of the present application provide a ground information processing method, device, and unmanned vehicle to obtain a fusion height map, so that the ground estimation of the ground area obtained through the fusion height map is more accurate.

In the first aspect, an embodiment of the present application provides a ground information processing method, including:

Obtain N frames of depth maps collected by the depth sensor, where N is an integer greater than or equal to 2;

Perform ground segmentation on each depth map to obtain the ground area in each depth map;

According to the ground area in the depth map of each frame, obtain the height map of the ground area in the depth map of each frame;

According to the height map of the ground area in each frame depth map, a fusion height map after N frames of height map fusion is obtained, and the fusion height map is used for ground estimation of the ground area.

In a second aspect, an embodiment of the present application provides a ground information processing device, including a depth sensor and a processor;

The depth sensor is used to collect a depth map;

The processor is configured to obtain N frames of depth maps collected by the depth sensor, where N is an integer greater than or equal to 2; perform ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; according to the depth of each frame For the ground area in the figure, obtain the height map of the ground area in the depth map of each frame; according to the height map of the ground area in the depth map of each frame, obtain the fusion height map after N frames of height map fusion, and the fusion height map is used for the ground Ground estimate of the area.

In the third aspect, an embodiment of the present application provides an unmanned vehicle, including: a depth sensor and a processor;

The depth sensor is used to collect a depth map;

In a fourth aspect, an embodiment of the present application provides an unmanned vehicle, including: a vehicle body and the ground information processing device according to the embodiment of the present application in the second aspect, wherein the ground information processing device is installed in the vehicle On the body.

In a fifth aspect, an embodiment of the present application provides a readable storage medium with a computer program stored on the readable storage medium; when the computer program is executed, it realizes the ground surface described in the embodiment of the present application in the first aspect. Information processing methods.

In a sixth aspect, an embodiment of the present application provides a program product, the program product includes a computer program, the computer program is stored in a readable storage medium, and at least one processor of a ground information processing device or an unmanned vehicle can be downloaded from The readable storage medium reads the computer program, and the at least one processor executes the computer program to cause the ground information processing device or the unmanned vehicle to implement the ground information processing method according to the embodiment of the present application in the first aspect .

The ground information processing method, device, and unmanned vehicle provided by the embodiments of the present application acquire N frames of depth maps collected by a depth sensor; perform ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; Obtain the height map of the ground area in the depth map of each frame for the ground area in the depth map of the frame; obtain the fusion height map after N frames of height map fusion according to the height map of the ground area in the depth map of each frame, and the fusion height map is used The ground estimate for the ground area. Since the fusion height map integrates the height information of the ground area in the N frame height map, the influence of noise in the single frame height map on the height value of the ground area is reduced, making the height value of the fusion height map closer to the actual ground area The height value. The ground estimation of the ground area obtained by the fusion height map is more accurate.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic architecture diagram of an unmanned vehicle 100 according to an embodiment of the present application;

Figure 2 is a schematic diagram of an application scenario provided by an embodiment of the application;

FIG. 3 is a flowchart of a ground information processing method provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a height map provided by an embodiment of the application;

5 is a schematic structural diagram of a ground information processing device provided by an embodiment of the application;

FIG. 6 is a schematic structural diagram of an unmanned vehicle provided by an embodiment of the application;

FIG. 7 is a schematic structural diagram of an unmanned vehicle provided by another embodiment of the application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of this application.

It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or a central component may also exist. When a component is considered to be "connected" to another component, it can be directly connected to another component or there may be a centered component at the same time.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of this application. The terms used in the description of the application herein are only for the purpose of describing specific embodiments, and are not intended to limit the application. The term "and/or" as used herein includes any and all combinations of one or more related listed items.

Hereinafter, some embodiments of the present application will be described in detail with reference to the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

The embodiments of the present application provide ground information processing methods, devices, and unmanned vehicles. FIG. 1 is a schematic architecture diagram of an unmanned vehicle 100 according to an embodiment of the present application.

The unmanned vehicle 100 may include a sensing system 110, a control system 120, and a mechanical system 130.

Among them, the perception system 110 is used to measure the status information of the unmanned vehicle, that is, the perception data of the unmanned vehicle 100. The perception data can represent the location information and/or status information of the unmanned vehicle 100, for example, position, angle, Speed, acceleration and angular velocity, etc. The sensing system 110 may include, for example, a visual sensor (for example, including multiple monocular or binocular vision devices), lidar, millimeter wave radar, inertial measurement unit (IMU), global navigation satellite system, gyroscope, ultrasonic sensor At least one of sensors such as, electronic compass, and barometer. For example, the global navigation satellite system may be a global positioning system (Global Positioning System, GPS).

After the sensing system 110 obtains the sensing data, it can transmit the sensing data to the control system 120. Among them, the control system 120 is used to make decisions for controlling the driving of the unmanned vehicle 100 according to the sensing data, for example: how much speed to travel, or how much braking acceleration to brake, or whether to change lanes, or , Turn left/right, etc. The control system 120 may include, for example, a computing platform, such as a vehicle-mounted supercomputing platform, or at least one device having processing functions such as a central processing unit and a distributed processing unit. The control system 120 may also include a communication link for various data transmission on the vehicle.

The control system 120 may output one or more control commands to the mechanical system 130 according to the determined decision. Wherein, the mechanical system 130 is used to control the unmanned vehicle 100 in response to one or more control commands from the control system 120 to complete the aforementioned decision. For example, the mechanical system 130 can drive the wheels of the unmanned vehicle 100 to rotate, thereby Power is provided for the driving of the unmanned vehicle 100, wherein the rotation speed of the wheels can affect the speed of the unmanned vehicle. Among them, the mechanical system 130 may include, for example, at least one of a mechanical body engine/motor, a controlled wire control system, and the like.

It should be understood that the aforementioned naming of the components of the unmanned vehicle is only for identification purposes and should not be understood as a limitation to the embodiments of the present application.

Figure 2 is a schematic diagram of an application scenario provided by an embodiment of this application. As shown in Figure 2, an unmanned vehicle can drive on the ground, and the unmanned vehicle can drive on the ground (for example, through the aforementioned perception The system 110) collects environmental information, which may include ground information, and then processes the ground information. For details on how to process it, please refer to the following embodiments of this application.

FIG. 3 is a flowchart of a ground information processing method provided by an embodiment of the application. As shown in FIG. 3, the method in this embodiment may include:

S301: Acquire N frames of depth maps collected by the depth sensor.

In this embodiment, the depth sensor may sequentially acquire depth maps according to its acquisition frequency, for example, N frames of depth maps are acquired in total, where N is an integer greater than or equal to 2, and this embodiment can acquire N frames of depth maps collected by the depth sensor.

Among them, the method of this embodiment can be applied to an unmanned vehicle, but this embodiment is not limited to this, and can also be applied to other movable platforms, such as robots. This embodiment is applied to an unmanned vehicle as an example. The unmanned vehicle may be equipped with a depth sensor, and the depth sensor may be used to collect a depth map of the environment processed by the unmanned vehicle during ground movement. Accordingly, N frames of depth maps collected by the depth sensor during the ground movement of the unmanned vehicle can be obtained.

Optionally, the depth sensor includes: a binocular camera, a time of flight (TOF) sensor or a lidar, which is not limited in this embodiment.

S302: Perform ground segmentation on each frame of depth map to obtain a ground area in each frame of depth map.

In this embodiment, each frame of the depth map collected by the depth sensor can be ground segmented to obtain the ground area in each frame of the depth map. Among them, the process of performing map segmentation can refer to the u-disparity-based solution, which will not be repeated in this embodiment. It should be noted that this embodiment is not limited to the u-disparity-based solution.

S303: Obtain a height map of the ground area in the depth map of each frame according to the ground area in the depth map of each frame.

In this embodiment, after obtaining the ground area of the depth map of each frame, the height map of the ground area of the depth map of the frame is obtained according to the ground area of the depth map of each frame.

Optionally, a possible implementation of S303 is: first obtain the point cloud data corresponding to the ground area according to the ground area in the depth map of each frame; and then according to the point cloud data corresponding to the ground area in each frame of the depth map, Obtain the height map of the ground area in the depth map of each frame. In this embodiment, obtaining the point cloud data corresponding to the ground area in the depth map of each frame according to the ground area in the depth map of each frame may include determining each depth pixel in the ground area of the depth map in each frame as a point cloud, In order to obtain the point cloud data corresponding to the ground area. Then, according to the point cloud data corresponding to the ground area in the depth map of each frame, the height map of the ground area in the depth map of each frame is obtained. For example, the point cloud data corresponding to the ground area in each frame of the depth map is orthogonally projected to the top view perspective to obtain The height map in the top view perspective, and the height map in the top view perspective is the height map of the ground area in the depth map of the frame. Wherein, the height value of each height pixel in the obtained height map has the height of the point cloud corresponding to the height pixel.

Optionally, the perspective of the top view is a downward direction of world coordinates.

Optionally, if multiple point cloud data are projected onto the same height pixel in the height map, the height value of the height pixel in the height map is the minimum value of the height values corresponding to the multiple point cloud data. For example: if multiple point cloud data have the same x coordinate value and y coordinate value, this is equivalent to having multiple point clouds on the vertical cylinder of the same ground point. For example, in a scene with traffic lights on the ground, the ground The same x-coordinate and y-coordinate of the area correspond to multiple point cloud data with different z-coordinates; at this time, the minimum height in these point clouds can be taken as the height value of the height pixel point of the ground point corresponding to the height map.

Optionally, multiple depth pixels in multiple point clouds or depth maps can correspond to one height pixel. For example, the depths of multiple depth pixels (for example, 2*2) in the x direction and the y direction can be combined. The pixel area corresponds to a height pixel. First, the minimum height value corresponding to each depth pixel point of the multiple depth pixels can be obtained, and then the average value or maximum value or minimum value of the minimum height value corresponding to these depth pixels can be determined Is the height value of this height pixel.

It should be noted that, in the implementation of this embodiment, S302 and S303 may be executed after obtaining N frames of depth maps collected by the depth sensor. It is also possible to execute S302 and S303 after acquiring each frame of the depth map collected by the depth sensor, for example: acquire the first frame of depth map collected by the depth sensor, perform the operations of S302 and S303 above on the first frame of depth map, and then acquire The second frame of depth map collected by the depth sensor performs the above-mentioned S302 and S303 operations on the second frame of depth map, and so on, so as to obtain a total of N frames of depth maps of the height map of the ground area in each frame of the depth map.

S304: Obtain a fusion height map after fusion of N frames of height maps according to the height map of the ground area in each frame depth map, and the fusion height map is used for ground estimation of the ground area.

In this embodiment, a frame of height map is obtained according to the height map of the ground area in each frame of the depth map of N frames of depth map, and the frame of height map is a fused height map after the fusion of N frames of height maps. Can be used for ground estimation of ground area. Since the fusion height map integrates the height information of the ground area in the N frame height map, the influence of noise in the single frame height map on the height value of the ground area is reduced, making the height value of the fusion height map closer to the actual ground area The height value.

Optionally, the height value of each height pixel in the fusion height map is an average value of the height value of the height pixel in the N frames of height maps.

The ground information processing method provided in this embodiment obtains N frames of depth maps collected by a depth sensor; performs ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; according to the ground area in each frame of depth map, Obtain a height map of the ground area in the depth map of each frame; according to the height map of the ground area in the depth map of each frame, a fusion height map after fusion of the height maps of N frames is obtained, and the fusion height map is used for ground estimation of the ground area. Since the fusion height map integrates the height information of the ground area in the N frame height map, the influence of noise in the single frame height map on the height value of the ground area is reduced, making the height value of the fusion height map closer to the actual ground area The height value. The ground estimation of the ground area obtained by the fusion height map is more accurate.

In some embodiments, a possible implementation of S304 above may include: S3041 and S3042:

S3041, according to the first N-1 frame height map of the N frame height map, obtain a fused height map after the fusion of the first N-1 frame height map.

S3042, according to the fusion height map after the fusion of the height maps of the previous N-1 frames and the height map of the Nth frame, obtain the fusion height map after the fusion of the height maps of the first N frames.

In this embodiment, the N frame height map is the height map of the ground area in the N frame depth map, and the N frame height map is the first frame height map, the second frame height map,..., the N-1th frame height map, The height map of the Nth frame can be obtained according to the height map of the first frame, the height map of the second frame,..., and the height map of the N-1th frame to obtain the fusion height of the first N-1 frame of the height map of the N frame. Figure. Then, according to the fusion height map after the fusion of the height maps of the first N-1 frames and the height map of the Nth frame (that is, the height map of the ground area in the depth map of the Nth frame), the fusion height map after the fusion of the height maps of the first N frames is obtained. Optionally, the height value of each height pixel in the fused height map obtained after the height map fusion of the previous N frames is obtained may be the average value of the height value of the height pixel in the height map of the previous N frames.

Optionally, a possible implementation of the above S3041 is to obtain the fused height map after the fusion of the height map of the first i-1 frame in the height map of the previous N-1 frames, where i is greater than or equal to 2 and less than or equal to N-1 Integer; then according to the fusion height of the previous i-1 frame height map and the i-th frame height map, the fusion height map of the previous i frame height map is obtained; then i is equal to i+1 until i is equal to N-1, thus Obtain the fusion height map of the first N-1 frame height map.

In this embodiment, starting with i equal to 2, the height map of the previous frame in the height map of frame N-1 is the first frame height map, and the fused height map after the fusion of the first frame height map is the first frame Height map, according to the height map of the first frame and the height map of the second frame, the fusion height map of the height map of the first 2 frames is obtained; after the height map of the third frame is obtained, the fusion height map of the height map of the first 2 frames and the third frame are obtained Height map, get the fusion height map of the height map of the first 3 frames; and so on, get the fusion height map of the height map of the first N-2 frames; after obtaining the height map of the N-1th frame, according to the height of the previous N-2 frames The fusion height map of the image and the height map of the N-1th frame are obtained, and the fusion height map after the fusion of the height map of the previous N-1 frames is obtained. Optionally, after obtaining the fusion height map of the height map of the first 2 frames, you can delete the height map of the first frame and the height map of the second frame. After obtaining the fusion height map of the height map of the first 3 frames, you can delete the height map of the first 2 frames. The fusion height map and the third frame height map, and so on, after obtaining the fusion height map of the first N-1 frame height map, the fusion height map of the previous N-2 frame height map and the N-1 frame height map can be deleted. To reduce storage space pressure and improve processing performance.

Among them, taking the fusion height map of the first 2 frames of height map obtained according to the height map of the first frame and the height map of the second frame as an example, the height pixels of the fusion height map of the height map of the first 2 frames and the height map of the second frame The height pixels are the same, and the difference is the height value corresponding to the height pixel. If the height pixel in the height map of the second frame also exists in the height map of the first frame, the height pixel is the fusion of the height map of the first two frames The height value in the height map is the average value of the height value of the height pixel in the second frame height map and the height value of the height pixel in the first frame height map. If the height pixel in the second frame height map If the point does not exist in the height map of the first frame, the height value of the height pixel in the fusion height map of the height map of the previous 2 frames is the height value of the height pixel in the height map of the second frame.

Correspondingly, a possible implementation of the above S3042 is to determine the fused height map after fusion of the height maps of the previous N-1 frames and the average value of the height values of the same height pixels in the height map of the Nth frame as the first N The height value of the pixel at the same height in the fused height map after the frame height map is fused.

Among them, the height pixels of the fusion height map of the first N frames of height map are the same as the height pixels of the Nth frame height map, and the difference is the height value corresponding to the height pixels. If the height pixels of the Nth frame height map It also exists in the height map of the first N-1 frame, and the height value of the height pixel in the fusion height map of the height map of the previous N-1 frame is the same as the height value of the height pixel in the height map of the Nth frame. The average value of the height pixels in the fusion height map of the height map of the previous N-1 frames. If the height pixel in the height map of the Nth frame does not exist in the fusion height map of the height map of the previous N-1 frame, Then, the height value of the height pixel in the fusion height map of the height map of the first N frames is the height value of the height pixel in the height map of the Nth frame.

Therefore, the above S3042 can be expressed by a formula: according to the fusion height map after the fusion of the height maps of the previous N-1 frames, the height map of the Nth frame, formula 1 and formula 2, obtain the fusion height map of the height map of the first N frames;

Formula one:

Formula 2: w _N,j =w _N-1,j +1;

Among them, h _N,j represents the height value of the height pixel j in the fusion height map after the fusion of the height maps of the first N frames, Z _N,j represents the height value of the height pixel j in the height map of the Nth frame, h _{N -1, j} represents the height value of the height pixel j in the fusion height map after the fusion of the previous N-1 frame height maps, w _{N, j} represents the number of times the height pixel j appears in the N frame height map, w _N-1,j represents the number of times the height pixel point j appears in the height map of the N-1 frame.

Refer to Figure 4, which is a schematic diagram of a height map provided by an embodiment of this application. Taking N equals 3 as an example, the solid line box in the upper left corner is, for example, the first frame height map, and the middle solid line box is, for example, the second frame. For the height map, the solid line box in the lower right corner is, for example, the third frame height map. The height map of each frame is translated along the height map plane relative to the world without rotation, and the amount of translation is guaranteed to be an integer number of height pixels. The height map of frame 2 is translated by one height pixel in the horizontal and vertical directions relative to the height map of frame 1, and the height of frame 3 is translated by one pixel in the horizontal and vertical directions relative to the height map of frame 2. Height in pixels.

When the height map of the first frame and the height map of the second frame are fused, the height pixels of the fused height map of the first two frames of height map are the same as the height pixels of the height map of the second frame. Among them, the height map of the second frame The height pixels of the first-to-last column and the first-to-last row do not appear in the height map of the first frame, therefore, the height pixels of the first-to-last column and the first-to-last row of the height map of the second frame are in the height map of the first 2 frames The number of occurrences is 1. The height values of the first-to-last column and the first-to-last row of the height map of the height map of the first two frames are equal to the first-to-last column and the second-to-last of the height pixels of the height map of the second frame. The height value of the height pixels of a row, the height pixels except for the first-to-last column and the first-to-last row in the height map of the second frame appear in the height map of the first frame, therefore, the height map of the second frame Number of occurrences of height pixels other than the first-to-last column and the first-to-last row 2, the height value of the height pixels except the first-to-last column and the first-to-last row in the fusion height map of the height map of the previous 2 frames It is equal to the average value of the height value of the height pixel in the second frame height map and the height value of the first frame height map.

When the fusion height map of the first 2 frame height maps and the third frame height map are fused (that is, the first frame height map, the second frame height map and the third frame height map are fused), the first 3 frame height maps The height pixels of the fusion height map are the same as the height pixels of the height map of the third frame, and the height pixels of the first to last column and the first row of the height map of the third frame are not fused in the height of the first 2 frames The height map (that is, the height map of the first frame and the height map of the second frame) appears. Therefore, the number of times the height pixels of the first-to-last column and the first-to-last row of the height map of the third frame appear in the height map of the first three frames is 1. The height values of the height pixels in the first-to-last column and the first-to-last row of the height map of the first 3 frames are equal to the height of the first-to-last column and the first-to-last row of the height pixels in the height map of the third frame The height value of the pixel. The height pixels of the second-to-last column and the second-to-last row of the height map of the third frame are all in the height map of the second frame but do not appear in the height map of the first frame. Therefore, the height map of the third frame The number of times that the height pixels in the second-to-last column and the second-to-last row of the middle, the height of the second-to-last column and the second-to-last row of the height pixel in the fusion height map of the first 3 frames are equal to the height pixel The average value of the height value of the height map in the second frame and the height value of the height map in the third frame. Except for the first-to-last column, the second-to-last column, the first-to-last row, and the second-to-last row of the height map in the third frame, the height pixels are all in the height map of the first frame, the height map of the second frame, and the height of the third frame It appears in the figure. Therefore, the number of times these height pixels appear is 3. The height value of these height pixels in the fusion height map of the height map of the first 3 frames is equal to the height of the height pixel in the fusion height map of the height map of the first 2 frames The average value of the height value and the height value in the third frame height map (that is, the average value of the height pixel point in the first frame height map, the second frame height map, and the third frame height map).

In some embodiments, after performing S304, the unmanned vehicle further estimates the ground of the ground area according to the fused height map after the fusion of the N frames of height maps. Since the fusion height map integrates the height information of the ground area in the N frames of height maps, the ground area of the ground area is estimated according to a frame of the fusion height map obtained after the fusion of multiple frames of height maps, and the obtained ground estimation result is more accurate.

Optionally, according to the above-mentioned fusion height map after fusion of the N frames of height maps, a possible implementation manner of estimating the ground of the ground area may be: fusion according to the N frames of height map of the ground area After fusing the height map and the geographic location information of the ground area, a ground model of the ground area is obtained by fitting; the ground model is a function of the height of the ground area with respect to the geographic location information of the ground area.

Wherein, the foregoing geographic location information may include longitude and latitude, or the foregoing geographic location information may include instantaneously constructed map location information of an unmanned vehicle, and the present embodiment is not limited to this.

Optionally, after the ground model of the ground area is obtained by fitting, the ground model of the ground area can be obtained according to the ground model of the ground area, and the ground model diagram can be displayed to the user through the display interface, so that the user can intuitively obtain the ground estimate result.

Optionally, the ground model includes: a B-spline surface model or a polynomial surface model.

Taking the B-spline surface model as an example, the ground model of the B-spline surface model can be expressed as:

Among them, B _{a, n} (x) and B _{b, m} (y) represent the geographic location information of the ground area, f (x, y) represents the height of the ground area, Wa _{, b} represent the weight coefficient, and n is the x dimension The order, m is the order of the y dimension.

For B-spline, the higher the order, the more complicated the curve can be expressed, but the higher the risk of overfitting, and the lower the order, the simpler the curve can be expressed, but it is not easy to overfit. At the same time, the number of B-spline control points will also affect the complexity of the curve and the risk of overfitting. Therefore, it is very important to choose the distribution and order (n, m) of control points reasonably. In the ground fitting example, the x direction here is the front and rear direction of the unmanned vehicle, and the y direction is the left and right direction of the unmanned vehicle. In actual scenes, usually the x-direction (front-rear direction) has a long observation distance, and the ground may have up and down slopes; while the y-direction (ie, the left-right direction), because the road width on the ground is usually limited, and the depth sensor's fov is also Limited; therefore, the control point and order can be configured as: the order n of the x dimension is 2, and a control point is set every 20 meters or so, the interval of the forward control point is 0 to 100 meters, and the order of the y dimension is m Set the control points at 20 meters on the left and right respectively, and there are 2 control points in total.

It should be noted that after the unmanned vehicle in this embodiment executes the above S304, other devices may also estimate the ground of the ground area according to the fusion height map.

The embodiment of the present application also provides a computer storage medium, the computer storage medium stores program instructions, and the program execution may include some or all of the steps of the ground information processing method in FIG. 3 and its corresponding embodiments. .

FIG. 5 is a schematic structural diagram of a ground information processing apparatus provided by an embodiment of this application. As shown in FIG. 5, the ground information processing apparatus 500 of this embodiment may include a depth sensor 501 and a processor 502. The aforementioned depth sensor 501 and the processor 502 may be connected via a bus.

The depth sensor 501 is used to collect a depth map.

The processor 502 is configured to obtain N frames of depth maps collected by the depth sensor 501, where N is an integer greater than or equal to 2; perform ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; Obtain the height map of the ground area in the depth map of each frame for the ground area in the depth map of the frame; obtain the fusion height map after N frames of height map fusion according to the height map of the ground area in the depth map of each frame, and the fusion height map is used The ground estimate for the ground area.

In some embodiments, the processor 502 is specifically configured to:

According to the first N-1 frame height map of the N frame height map, obtain the fused height map after the fusion of the first N-1 frame height map;

According to the fusion height map after the fusion of the height maps of the first N-1 frames and the height map of the Nth frame, the fusion height map after the fusion of the height maps of the first N frames is obtained.

In some embodiments, the processor 502 is specifically configured to:

Obtain the fused height map after the fusion of the height map of the previous i-1 frame in the height map of the previous N-1 frames, where i is an integer greater than or equal to 2 and less than or equal to N-1;

Obtain the fusion height map of the previous i frame height map according to the fusion height map after the fusion of the height map of the previous i-1 frame and the height map of the i-th frame;

Update i to be equal to i+1 until i is equal to N-1, so as to obtain the fused height map of the height map of the previous N-1 frames.

In some embodiments, the height value of each height pixel in the fusion height map is the average value of the height value of the height pixel in the N frames of height maps.

In some embodiments, the processor 502 is further configured to obtain a fused height map after the fusion of N frames of height maps according to the height map corresponding to the depth map of each frame, and then compare the ground area according to the fused height map. The ground is estimated.

In some embodiments, the processor 502 is specifically configured to:

According to the fusion height map of the ground area and the geographic location information of the ground area, a ground model of the ground area is obtained by fitting;

The ground model is a function of the height of the ground area with respect to the geographic location information of the ground area.

In some embodiments, the geographic location information includes longitude and latitude.

In some embodiments, the ground model includes: a B-spline surface model, or a polynomial surface model.

In some embodiments, the processor 502 is specifically configured to:

Obtain the point cloud data corresponding to the ground area according to the ground area in each depth map;

According to the point cloud data corresponding to the ground area in the depth map of each frame, the height map of the ground area in the depth map of each frame is obtained.

In some embodiments, the processor 502 is specifically configured to:

Orthogonally project the point cloud data corresponding to the ground area in the depth map of each frame to the top view perspective to obtain the height map in the top view perspective.

In some embodiments, if multiple point cloud data are projected onto the same height pixel in the height map, the height value of the height pixel in the height map is the minimum value of the height values corresponding to the multiple point cloud data .

In some embodiments, the perspective of the top view is a downward direction of world coordinates.

In some embodiments, the ground information processing device 500 is applied to an unmanned vehicle, and the depth sensor 501 is airborne on the unmanned vehicle.

The processor 502 is specifically configured to obtain N frames of depth maps collected by the depth sensor during the movement of the unmanned vehicle.

In some embodiments, the depth sensor 501 includes a binocular camera, a TOF sensor or a lidar.

Optionally, the ground information processing device 500 of this embodiment may further include: a memory (not shown in the figure), the memory is used to store program code, and when the program code is executed, the ground information processing device 500 can implement the above Technical solutions.

The device of this embodiment can be used to implement the technical solutions of the embodiment of FIG. 3 and its corresponding method, and its implementation principles and technical effects are similar, and will not be repeated here.

FIG. 6 is a schematic structural diagram of an unmanned vehicle provided by an embodiment of this application. As shown in FIG. 6, the unmanned vehicle 600 of this embodiment may include a depth sensor 601 and a processor 602. The above-mentioned depth sensor 601 and the processor 602 may be connected through a bus.

The depth sensor 601 is used to collect a depth map.

The processor 602 is configured to obtain N frames of depth maps collected by the depth sensor 601, where N is an integer greater than or equal to 2; perform ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; Obtain the height map of the ground area in the depth map of each frame for the ground area in the depth map of the frame; obtain the fusion height map after N frames of height map fusion according to the height map of the ground area in the depth map of each frame, and the fusion height map is used The ground estimate for the ground area.

In some embodiments, the processor 602 is specifically configured to:

In some embodiments, the processor 602 is further configured to obtain a fused height map after the fusion of the N frames of height maps according to the height map corresponding to the depth map of each frame, and then calculate the ground area according to the fused height map. The ground is estimated.

In some embodiments, the processor 602 is specifically configured to:

Orthogonally project the point cloud data corresponding to the ground area in each frame of the depth map to the top view perspective to obtain the height map in the top view perspective.

In some embodiments, the processor 602 is specifically configured to obtain N frames of depth maps collected by the depth sensor 601 when the unmanned vehicle is moving.

In some embodiments, the depth sensor 601 includes a binocular camera, a TOF sensor or a lidar.

Optionally, the unmanned vehicle 600 of this embodiment may further include: a memory (not shown in the figure), the memory is used to store program code, and when the program code is executed, the unmanned vehicle 600 can implement the foregoing Technical solutions.

The unmanned vehicle in this embodiment can be used to implement the technical solutions of FIG. 3 and its corresponding method embodiments, and its implementation principles and technical effects are similar, and will not be repeated here.

FIG. 7 is a schematic structural diagram of an unmanned vehicle provided by another embodiment of the application. As shown in FIG. 7, the unmanned vehicle 700 of this embodiment may include a vehicle body 701 and a ground information processing device 702.

Wherein, the ground information processing device 702 is installed on the vehicle body 701. The ground information processing device 702 may be a device independent of the vehicle body 701.

Wherein, the ground information processing device 702 can adopt the structure of the device embodiment shown in FIG. 5, and correspondingly, it can implement the technical solutions of FIG. 3 and its corresponding method embodiments. The implementation principles and technical effects are similar, and will not be repeated here. .

A person of ordinary skill in the art can understand that all or part of the steps in the above method embodiments can be implemented by a program instructing relevant hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, it is executed. Including the steps of the foregoing method embodiment; and the foregoing storage medium includes: read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disks or optical disks, etc., which can store program codes Medium.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the application, not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: It is still possible to modify the technical solutions described in the foregoing embodiments, or equivalently replace some or all of the technical features; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the application range.

Claims

A ground information processing method, characterized in that it comprises:

Obtain N frames of depth maps collected by the depth sensor, where N is an integer greater than or equal to 2;

Perform ground segmentation on each depth map to obtain the ground area in each depth map;

According to the ground area in the depth map of each frame, obtain the height map of the ground area in the depth map of each frame;

According to the height map of the ground area in each frame depth map, a fusion height map after N frames of height map fusion is obtained, and the fusion height map is used for ground estimation of the ground area.
The method according to claim 1, wherein the obtaining a fused height map after fusion of N frames of height maps according to the height map corresponding to the ground area in each frame depth map comprises:

According to the first N-1 frame height map of the N frame height map, obtain the fused height map after the fusion of the first N-1 frame height map;

According to the fusion height map after the fusion of the height maps of the first N-1 frames and the height map of the Nth frame, the fusion height map after the fusion of the height maps of the first N frames is obtained.
The method according to claim 2, wherein the obtaining a fused height map after the fusion of the first N-1 frame height maps according to the first N-1 frame height maps of the N frame height map comprises:

Obtain the fused height map after the fusion of the height map of the previous i-1 frame in the height map of the previous N-1 frames, where i is an integer greater than or equal to 2 and less than or equal to N-1;

Obtain the fusion height map of the previous i frame height map according to the fusion height map after the fusion of the height map of the previous i-1 frame and the height map of the i-th frame;

Update i to be equal to i+1 until i is equal to N-1, so as to obtain the fused height map of the height map of the previous N-1 frames.
The method according to any one of claims 1 to 3, wherein the height value of each height pixel in the fusion height map is the average value of the height value of the height pixel in the N frames of height maps.
The method according to any one of claims 1 to 4, wherein after obtaining a fused height map after fusion of N frames of height maps according to the height map corresponding to the ground area in each frame depth map, the method further comprises:

According to the fusion height map, the ground of the ground area is estimated.
The method according to claim 5, wherein the estimating the ground of the ground area according to the fusion height map comprises:

According to the fusion height map of the ground area and the geographic location information of the ground area, a ground model of the ground area is obtained by fitting;

The ground model is a function of the height of the ground area with respect to the geographic location information of the ground area.
The method according to claim 6, wherein the geographic location information includes longitude and latitude.
The method according to claim 6 or 7, wherein the ground model comprises: a B-spline surface model or a polynomial surface model.
The method according to any one of claims 1-8, wherein the obtaining a height map of the ground area in the depth map of each frame according to the ground area in the depth map of each frame comprises:

Obtain the point cloud data corresponding to the ground area according to the ground area in each depth map;

According to the point cloud data corresponding to the ground area in the depth map of each frame, the height map of the ground area in the depth map of each frame is obtained.
The method according to claim 9, wherein the obtaining a height map of the ground area in each frame of the depth map according to the point cloud data corresponding to the ground area in the depth map of each frame comprises:

Orthogonally project the point cloud data corresponding to the ground area in each frame of the depth map to the top view perspective to obtain the height map in the top view perspective.
The method according to claim 10, wherein if multiple point cloud data are projected onto the same height pixel in the height map, the height value of the height pixel in the height map is the multiple point cloud data The minimum value of the corresponding height value.
The method according to claim 10 or 11, wherein the perspective of the top view is a downward direction of world coordinates.
The method according to any one of claims 1-12, wherein the method is applied to an unmanned vehicle, and the depth sensor is airborne on the unmanned vehicle;

The acquiring N frames of depth maps collected by the depth sensor includes:

Acquire N frames of depth maps collected by the depth sensor when the unmanned vehicle is moving.
The method according to any one of claims 1-13, wherein the depth sensor comprises a binocular camera, a time-of-flight TOF sensor or a lidar.
A ground information processing device, characterized by comprising: a depth sensor and a processor;

The depth sensor is used to collect a depth map;

The processor is configured to obtain N frames of depth maps collected by the depth sensor, where N is an integer greater than or equal to 2; perform ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; according to the depth of each frame For the ground area in the figure, obtain the height map of the ground area in the depth map of each frame; according to the height map of the ground area in the depth map of each frame, obtain the fusion height map after N frames of height map fusion, and the fusion height map is used for the ground Ground estimate of the area.
The device according to claim 15, wherein the processor is specifically configured to:

According to the first N-1 frame height map of the N frame height map, obtain the fused height map after the fusion of the first N-1 frame height map;

According to the fusion height map after the fusion of the height maps of the first N-1 frames and the height map of the Nth frame, the fusion height map after the fusion of the height maps of the first N frames is obtained.
The device according to claim 16, wherein the processor is specifically configured to:

Obtain the fused height map after the fusion of the height map of the previous i-1 frame in the height map of the previous N-1 frames, where i is an integer greater than or equal to 2 and less than or equal to N-1;

Obtain the fusion height map of the previous i frame height map according to the fusion height map after the fusion of the height map of the previous i-1 frame and the height map of the i-th frame;

Update i to be equal to i+1 until i is equal to N-1, so as to obtain the fused height map of the height map of the previous N-1 frames.
The device according to any one of claims 15-17, wherein the height value of each height pixel in the fusion height map is the average value of the height value of the height pixel in the N frames of height maps.
The device according to any one of claims 15-18, wherein the processor is further configured to obtain a fused height map after fusion of N frames of height maps according to the height map corresponding to the depth map of each frame, according to The fusion height map estimates the ground of the ground area.
The device according to claim 19, wherein the processor is specifically configured to:

According to the fusion height map of the ground area and the geographic location information of the ground area, a ground model of the ground area is obtained by fitting;

The ground model is a function of the height of the ground area with respect to the geographic location information of the ground area.
The device according to claim 20, wherein the geographic location information includes longitude and latitude.
The device according to claim 20 or 21, wherein the ground model comprises: a B-spline surface model or a polynomial surface model.
The device according to any one of claims 15-22, wherein the processor is specifically configured to:

Obtain the point cloud data corresponding to the ground area according to the ground area in each depth map;

According to the point cloud data corresponding to the ground area in the depth map of each frame, the height map of the ground area in the depth map of each frame is obtained.
The device according to claim 23, wherein the processor is specifically configured to:

Orthogonally project the point cloud data corresponding to the ground area in each frame of the depth map to the top view perspective to obtain the height map in the top view perspective.
The device according to claim 24, wherein if multiple point cloud data are projected onto the same height pixel in the height map, the height of the height pixel in the height map is the value of the multiple point cloud data The minimum value of the corresponding height value.
The device according to claim 24 or 25, wherein the perspective of the top view is a downward direction of world coordinates.
The device according to any one of claims 15-26, wherein the device is applied to an unmanned vehicle, and the depth sensor is mounted on the unmanned vehicle;

The processor is specifically configured to obtain N frames of depth maps collected by the depth sensor during the movement of the unmanned vehicle.
The device according to any one of claims 15-27, wherein the depth sensor comprises a binocular camera, a time-of-flight TOF sensor or a lidar.
An unmanned vehicle, characterized by comprising: a depth sensor and a processor;

The depth sensor is used to collect a depth map;

The processor is configured to obtain N frames of depth maps collected by the depth sensor, where N is an integer greater than or equal to 2; perform ground segmentation on each frame of depth map to obtain the ground area in each frame of depth map; according to the depth of each frame For the ground area in the figure, obtain the height map of the ground area in the depth map of each frame; according to the height map of the ground area in the depth map of each frame, obtain the fusion height map after N frames of height map fusion, and the fusion height map is used for the ground Ground estimate of the area.
The unmanned vehicle according to claim 29, wherein the processor is specifically configured to:

According to the first N-1 frame height map of the N frame height map, obtain the fused height map after the fusion of the first N-1 frame height map;

According to the fusion height map after the fusion of the height maps of the first N-1 frames and the height map of the Nth frame, the fusion height map after the fusion of the height maps of the first N frames is obtained.
The unmanned vehicle according to claim 30, wherein the processor is specifically configured to:

Obtain the fused height map after the fusion of the height map of the previous i-1 frame in the height map of the previous N-1 frames, where i is an integer greater than or equal to 2 and less than or equal to N-1;

Obtain the fusion height map of the previous i frame height map according to the fusion height map after the fusion of the height map of the previous i-1 frame and the height map of the i-th frame;

Update i to be equal to i+1 until i is equal to N-1, so as to obtain the fused height map of the height map of the previous N-1 frames.
The unmanned vehicle according to any one of claims 29-31, wherein the height value of each height pixel in the fusion height map is the value of the height of the height pixel in the N frame height map average value.
The unmanned vehicle according to any one of claims 29-32, wherein the processor is further configured to obtain a fused height map after fusion of N frames of height maps according to the height map corresponding to the depth map of each frame After that, the ground of the ground area is estimated according to the fusion height map.
The unmanned vehicle according to claim 33, wherein the processor is specifically configured to:

According to the fusion height map of the ground area and the geographic location information of the ground area, a ground model of the ground area is obtained by fitting;

The ground model is a function of the height of the ground area with respect to the geographic location information of the ground area.
The unmanned vehicle according to claim 34, wherein the geographic location information includes longitude and latitude.
The unmanned vehicle according to claim 34 or 35, wherein the ground model comprises: a B-spline surface model or a polynomial surface model.
The unmanned vehicle according to any one of claims 29-36, wherein the processor is specifically configured to:

Obtain the point cloud data corresponding to the ground area according to the ground area in each depth map;

According to the point cloud data corresponding to the ground area in each depth map, the height map of the ground area in each depth map is obtained.
The unmanned vehicle according to claim 37, wherein the processor is specifically configured to:

Orthogonally project the point cloud data corresponding to the ground area in each frame of the depth map to the top view perspective to obtain the height map in the top view perspective.
The unmanned vehicle according to claim 38, wherein if multiple point cloud data are projected onto the same height pixel in the height map, the height of the height pixel in the height map is the multiple The minimum height value corresponding to the point cloud data.
The unmanned vehicle according to claim 38 or 39, wherein the perspective of the top view is a downward direction of world coordinates.
The unmanned vehicle according to any one of claims 29-40, wherein the processor is specifically configured to: acquire the depth of N frames collected by the depth sensor during the movement of the unmanned vehicle Figure.
The unmanned vehicle according to any one of claims 29-41, wherein the depth sensor comprises a binocular camera, a time-of-flight TOF sensor or a lidar.
An unmanned vehicle, characterized by comprising: a vehicle body and the ground information processing device according to any one of claims 15-28, wherein the ground information processing device is installed on the vehicle body.
A readable storage medium, characterized in that a computer program is stored on the readable storage medium; when the computer program is executed, the ground information processing method according to any one of claims 1-14 is realized.