CN117292349B

CN117292349B - Method, device, computer equipment and storage medium for determining road surface height

Info

Publication number: CN117292349B
Application number: CN202311559624.2A
Authority: CN
Inventors: 胡兰; 张如高; 虞正华
Original assignee: Magic Vision Intelligent Technology Wuhan Co ltd
Current assignee: Magic Vision Intelligent Technology Wuhan Co ltd
Priority date: 2023-11-22
Filing date: 2023-11-22
Publication date: 2024-04-12
Anticipated expiration: 2043-11-22
Also published as: CN117292349A

Abstract

The invention relates to the technical field of three-dimensional reconstruction, and discloses a method, a device, computer equipment and a storage medium for determining road surface height, wherein the method comprises the following steps: and acquiring multi-frame images and a preset pavement width. Based on the multi-frame images, a target track is acquired. And determining a road surface reconstruction range based on the preset road surface width and the road surface coordinates of each track point in the target track. Based on the preconfigured grid size, gridding the three-dimensional space of the road surface reconstruction range to obtain a plurality of grids of the road surface reconstruction range. The distance between each two adjacent track points is determined based on the road surface coordinates of each track point. And determining the coverage road surface range between every two adjacent track points based on the distance and the preset road surface width. And selecting a plurality of grids covering the road surface range from the plurality of grids covering the road surface range. And determining the road surface height corresponding to the target track point as the road surface height covering each grid vertex covered by the road surface range. The method can improve the accuracy of the constructed pavement three-dimensional model.

Description

Method, device, computer equipment and storage medium for determining road surface height

Technical Field

The invention relates to the technical field of three-dimensional reconstruction, in particular to a method, a device, computer equipment and a storage medium for determining road surface height.

Background

In the technical field of three-dimensional scene reconstruction, when reconstructing the whole scene, a three-dimensional scene reconstruction algorithm is generally decoupled and is divided into ground object reconstruction and ground reconstruction.

In performing a ground reconstruction, the ground reconstruction algorithm will generally assume that the ground is flat. In practice, however, most floors are undulating. Therefore, the ground reconstruction is carried out through the ground reconstruction algorithm, and the accuracy of the obtained three-dimensional model of the road surface is low.

Disclosure of Invention

In view of the above, the present invention provides a method, apparatus, computer device and storage medium for determining road surface height, so as to solve the problem of low accuracy of three-dimensional reconstructed road surface three-dimensional model.

In a first aspect, the present invention provides a method of determining road surface height, the method comprising:

acquiring a multi-frame image shot by a target vehicle in the running process of a target road surface and a preset road surface width;

acquiring a target track based on the multi-frame image, wherein the target track comprises track coordinates corresponding to a plurality of track points respectively and road surface heights corresponding to the track points respectively, and the track coordinates are used for indicating road surface coordinates of the target vehicle in a reference coordinate system corresponding to the target road surface;

Determining a road surface reconstruction range corresponding to the target road surface based on the preset road surface width and the road surface coordinates corresponding to each track point in the target track;

based on the preset grid size, gridding a three-dimensional space corresponding to the pavement reconstruction range to obtain a plurality of grids corresponding to the pavement reconstruction range;

determining the distance between every two adjacent track points based on the road surface coordinates corresponding to each track point in the target track;

determining a coverage road surface range between every two adjacent track points based on the distance between every two adjacent track points and the preset road surface width;

selecting a plurality of grids covered by the coverage pavement range from the plurality of grids corresponding to the pavement reconstruction range;

and determining the road surface height corresponding to the target track point in the first group of two adjacent track points as the road surface height of each grid vertex in a plurality of grids covered by the road surface range corresponding to the first group of two adjacent track points, wherein the first group of two adjacent track points are any group of two adjacent track points in the target track, and the target track point is the first track point in the first group of two adjacent track points.

The invention provides a method for determining the height of a road surface, which has the following advantages:

because the road surface is rough and uneven, if the road surface height of each grid vertex in a plurality of grids corresponding to the road surface is determined to be zero, the constructed road surface three-dimensional reconstruction model is inaccurate. Therefore, a target track is generated from images taken by the target vehicle during the travel of the target road surface, and each point in the target track corresponds to the pose of the target vehicle in the reference coordinate system to which the target road surface belongs, namely, the road surface coordinates and the road surface height. Further, by comparing the plurality of grids corresponding to the road surface reconstruction range with the plurality of grids corresponding to the coverage road surface range, the road surface height of each grid vertex in the plurality of grids corresponding to the road surface reconstruction range can be determined. Therefore, the accuracy of constructing the three-dimensional reconstruction model of the road surface can be improved by considering the fluctuation condition of the road surface.

In an optional embodiment, the determining the road reconstruction range corresponding to the target road based on the preset road width and the road coordinates corresponding to each track point in the target track includes:

obtaining a track point cloud corresponding to the target road surface and the road surface coordinates of each track point in the track point cloud based on the preset road surface width and the road surface coordinates of each track point in the target track;

And determining the road surface reconstruction range based on the road surface coordinates of each track point in the track point cloud.

Specifically, since the overall shape of the road surface may not be regular, if the road surface reconstruction range is directly determined according to the regular shape, the determined road surface reconstruction range may be too large or too small, and when the model reconstruction is performed subsequently due to too large road surface reconstruction range, more processing resources are occupied, the model reconstruction efficiency is low, and part of key information is omitted due to too small road surface reconstruction range. Therefore, the road surface reconstruction range is determined through the road surface coordinates of each track point in the target track and the preset road surface width, the actual condition of the target road surface can be considered, and the accuracy of constructing the road surface three-dimensional reconstruction model can be improved.

In an optional embodiment, each mesh vertex in the plurality of meshes corresponding to the road surface reconstruction range further includes a color value and a semantic class, and the method further includes:

acquiring camera pose and camera configuration information corresponding to each frame of image based on the multi-frame image, wherein each frame of image in the multi-frame image corresponds to one track point in the target track;

for a first frame image in the multi-frame images, determining an azimuth vector of each pixel in the first frame image under the reference coordinate system based on the camera pose corresponding to the first frame image, the camera configuration information and the track point, wherein the first frame image is any frame image in the multi-frame images;

When an intersection point exists between an azimuth vector of a pixel in at least one frame of image under the reference coordinate system and any one of a plurality of grids corresponding to the pavement reconstruction range, determining a predicted color value of the intersection point based on a color value of each grid vertex in the grid corresponding to the intersection point and a preconfigured interpolation algorithm in a current iteration period;

determining the semantic category of the grid vertex closest to the intersection point as the predicted semantic category of the intersection point;

determining a first loss value based on the predicted color value, the predicted semantic class, the true color value and the true semantic class of the pixel corresponding to the intersection point in the image to which the pixel belongs;

updating the color value of each grid vertex in a plurality of grids corresponding to the pavement reconstruction range based on a preset gradient algorithm and the predicted color value, and updating the semantic category of each grid vertex in the plurality of grids corresponding to the pavement reconstruction range based on the preset gradient algorithm and the predicted semantic category;

stopping the iterative process when the first loss value is determined to meet a convergence condition;

or when the first loss value is determined to not meet the convergence condition, entering a next iteration period, and updating the color value and the semantic category of each grid vertex in the multiple grids corresponding to the reconstruction range again.

Specifically, since the image includes color value information of each pixel, and semantic classification of each pixel in the image can also be obtained by performing semantic segmentation on the image. Therefore, through the comparison of the grids and the pixels, the color value and the semantic category of each grid vertex in the multiple grids corresponding to the pavement reconstruction range can be determined. Further, the three-dimensional model reconstruction of the road surface is performed by combining the road surface coordinates, the road surface height, the color values and the semantic categories of each grid vertex, so that a more accurate three-dimensional reconstruction model of the road surface can be obtained.

In an optional implementation manner, the determining, for a first frame image in the multi-frame images, an azimuth vector of each pixel in the first frame image in the reference coordinate system based on the camera pose corresponding to the first frame image, the camera configuration information, and a track point includes:

for a first frame image in the multi-frame images, determining coordinate system conversion parameters corresponding to the first frame image based on the camera pose and track points corresponding to the first frame image;

determining an azimuth vector of each pixel in the first frame image under a camera coordinate system to which the pixel belongs based on camera configuration information corresponding to the first frame image;

And carrying out coordinate conversion on the azimuth vector of each pixel in the first frame image under the camera coordinate system to which the pixel belongs based on the coordinate system conversion parameters corresponding to the first frame image, so as to obtain the azimuth vector of each pixel in the first frame image under the reference coordinate system.

Specifically, since the image corresponds to the camera coordinate system, the plurality of grids corresponding to the road surface reconstruction range are in the reference coordinate system, and the pixels in the image and the grids of the road surface reconstruction range need to be matched in the subsequent processing. Therefore, it is necessary to transform the image into a reference coordinate system, and determine a color value and a semantic class for each mesh vertex. Furthermore, a three-dimensional reconstruction model of the pavement with color information and semantic information can be obtained, and the three-dimensional reconstruction model is more in line with the actual situation and is more accurate.

In an optional implementation manner, the determining the first loss value based on the predicted color value, the predicted semantic class, the true color value and the true semantic class of the pixel corresponding to the intersection point in the image to which the pixel belongs includes:

determining a second loss value based on the predicted color value, the true color value of the pixel corresponding to the intersection point in at least one frame of image and the first loss function;

Determining a third loss value based on the predicted semantic class and the true semantic class of the pixel corresponding to the intersection point in at least one frame of image and a second loss function;

determining a semantic category weight value based on the pre-acquired semantic segmentation quality of each pixel;

a first loss value is determined based on the second loss value, the third loss value, the semantic category weight value, and a third loss function.

Specifically, since the color value of each pixel in the image is the actual color value, the semantic class of each pixel in the image is determined by the semantic segmentation algorithm, that is, the semantic class of the pixel has a certain error with reality. Thus, for the third penalty value, a semantic category weight value may be determined based on the quality of the semantic segmentation. Thus, a more accurate loss value can be determined.

In an alternative embodiment, the third loss function uses the following expression:

L=loss_rgb +w2·loss_sem……（1）

wherein L is the first loss value, loss_rgb is the second loss value, w2 is the semantic category weight value, and loss_semm is the third loss value.

Specifically, since the semantic class of each pixel in the image is determined through a semantic segmentation algorithm, that is, the semantic class of the pixel has a certain error with reality. Therefore, when calculating the loss value, the error is indicated by the semantic category weight value, and the more accurate loss value can be determined.

In an alternative embodiment, the method further comprises:

obtaining a target semantic category;

selecting grid vertexes with semantic categories as the target semantic categories from a plurality of grids corresponding to the pavement reconstruction range to form a target grid vertex set corresponding to first round dense operation;

based on the road surface coordinates and the road surface heights of each grid vertex in the target grid vertex set corresponding to the first round of dense operation, respectively performing dense operation on each grid vertex in the target grid vertex set corresponding to the first round of dense operation to obtain a dense grid vertex set corresponding to the first round of dense operation;

fusing the target grid vertex set corresponding to the first round of dense operation and the dense grid vertex set to obtain a first sampling point set corresponding to the first round of dense operation;

sampling grid vertexes in the first sampling set based on a preset sampling number corresponding to the first round of dense operation to obtain a grid vertex set corresponding to the next round of dense operation;

in the non-first round of dense operation, carrying out dense operation on each grid vertex in the target grid vertex set of the previous round based on the road surface coordinates and the road surface height corresponding to each grid vertex in the target grid vertex set of the previous round to obtain a dense grid vertex set corresponding to the current round;

Fusing the target grid vertex set corresponding to the round of dense operation and the dense grid vertex set corresponding to the round of dense operation to obtain a sampling point set corresponding to the round of dense operation;

when the operation times corresponding to the dense operation of the round is equal to the preset operation times, sampling the sampling point set corresponding to the dense operation of the round to obtain a target grid vertex set;

and merging the target grid vertex set with a plurality of grids corresponding to the pavement reconstruction range, and performing triangularization treatment to obtain a plurality of grids corresponding to the pavement reconstruction range after densification, wherein the plurality of grids corresponding to the pavement reconstruction range after densification form a three-dimensional reconstruction model of the target pavement.

In particular, since some semantic categories require a significant attention (i.e., target semantic categories), the corresponding mesh vertices require more to construct a finer mesh. Thus, by performing multiple rounds of dense operations on mesh vertices of the target semantic class, the mesh of the target semantic class may be refined. Further, the accuracy of the three-dimensional reconstruction model of the constructed pavement is higher.

In a second aspect, the present invention provides an apparatus for determining road surface height, the apparatus comprising:

The acquisition module is used for acquiring multi-frame images shot by the target vehicle in the running process of the target road surface and the preset road surface width; acquiring a target track based on the multi-frame image, wherein the target track comprises track coordinates corresponding to a plurality of track points respectively and road surface heights corresponding to the track points respectively, and the track coordinates are used for indicating road surface coordinates of the target vehicle in a reference coordinate system corresponding to the target road surface;

the determining module is used for determining a road surface reconstruction range corresponding to the target road surface based on the preset road surface width and the road surface coordinates corresponding to each track point in the target track;

the gridding module is used for gridding the three-dimensional space corresponding to the pavement reconstruction range based on the preset grid size to obtain a plurality of grids corresponding to the pavement reconstruction range;

the determining module is used for determining the distance between every two adjacent track points based on the road surface coordinates corresponding to each track point in the target track; determining a coverage road surface range between every two adjacent track points based on the distance between every two adjacent track points and the preset road surface width;

The selecting module is used for selecting a plurality of grids covered by the coverage pavement range from a plurality of grids corresponding to the pavement reconstruction range;

the determining module is configured to determine a road surface height corresponding to a target track point in a first set of two adjacent track points as a road surface height of each grid vertex in a plurality of grids covered by a road surface range corresponding to the first set of two adjacent track points, where the first set of two adjacent track points are any one set of two adjacent track points in the target track, and the target track point is a first track point in the first set of two adjacent track points.

In a third aspect, the present invention provides a computer device comprising: the device comprises a memory and a processor, wherein the memory and the processor are in communication connection, the memory stores computer instructions, and the processor executes the computer instructions so as to execute the method for determining the road surface height according to the first aspect or any implementation mode corresponding to the first aspect.

In a fourth aspect, the present invention provides a computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of determining road surface height of the first aspect or any of its corresponding embodiments.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained based on these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a method of determining road surface elevation in accordance with an embodiment of the present invention;

FIG. 2 is a flow chart of another method of determining road surface elevation in accordance with an embodiment of the present invention;

FIG. 3 is a flow diagram of a method of updating color values and semantic categories according to an embodiment of the present invention;

FIG. 4 is a flow chart of a method of determining an orientation vector of a pixel in accordance with an embodiment of the present invention;

FIG. 5 is a flow chart of a method of determining loss values according to an embodiment of the present invention;

FIG. 6 is a flow diagram of a method of dense mesh vertices in accordance with an embodiment of the invention;

FIG. 7 is a flow chart of a method of constructing a three-dimensional reconstruction model of a target pavement in accordance with an embodiment of the present invention;

FIG. 8 is a block diagram of an apparatus for determining road surface height according to an embodiment of the present invention;

fig. 9 is a schematic diagram of a hardware structure of a computer device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In the automatic driving technology, modules such as sensing, positioning and controlling are generally divided. After the sensing module acquires the environmental data, three-dimensional reconstruction is carried out through the three-dimensional reconstruction network model, and the reconstructed scene three-dimensional model is output to other modules such as positioning, control and the like for subsequent processing. For three-dimensional reconstruction of a scene, the scene is generally divided into an overground object and a ground surface for three-dimensional reconstruction. For example, three-dimensional reconstruction of the road surface on which the vehicle is traveling.

The embodiment of the invention provides a method for determining the height of a road surface, which is used for determining the height of the road surface of grid vertexes through the running track of a vehicle so as to improve the accuracy of a three-dimensional reconstruction model.

In accordance with an embodiment of the present invention, there is provided an embodiment of a method of determining road surface height, it being noted that the steps illustrated in the flowchart of the figures may be performed in a computer system, such as a set of computer executable instructions, and that although a logical sequence is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in a different order than that illustrated herein.

In this embodiment, a method for determining the height of a road surface is provided, and the method can be used for a computer device, where the computer device may be a terminal, a server, etc., for example, a desktop computer, a tablet computer, etc. The computer device may be configured with a three-dimensional reconstruction model for implementing the method. Fig. 1 is a flowchart of a method of determining road surface height according to an embodiment of the present invention, as shown in fig. 1, the flowchart including the steps of:

step S101, acquiring a multi-frame image captured by a target vehicle during a target road surface driving process, and a preset road surface width.

Wherein the target vehicle may be configured with at least one camera, and the at least one camera may include a forward-looking camera, a look-around camera, and the like. Each of the plurality of frame images may be an image photographed by the same one of the at least one camera, for example, a front view camera. Alternatively, each of the plurality of frames of images may be a fused image of images captured by different cameras.

In practice, during the travel of the target vehicle on the target road surface, the camera configured on the target vehicle may capture an image based on the pre-configured instructions and transmit to the computer device. The computer device may store the captured multi-frame images for subsequent processing. When the three-dimensional reconstruction is performed on the target pavement, a technician can estimate the width of the target pavement according to experience, obtain a preset pavement width (the preset pavement width of the target pavement is larger than the actual pavement width), and input the preset pavement width into the computer equipment. For example, the actual road width is 10 meters and the preset road width may be 12 meters.

Step S102, acquiring a target track based on the multi-frame images.

The target track comprises track coordinates corresponding to the track points and road surface heights corresponding to the track points, wherein the track coordinates are used for indicating road surface coordinates of the target vehicle in a reference coordinate system corresponding to the target road surface. The reference coordinate system may be arbitrarily set, and this embodiment is described by taking the following cases as examples: the first track point in the target track may be on the origin of coordinates of a reference coordinate system having an x-axis in the width direction of the target road surface and a y-axis in the extending direction of the target road surface.

In practice, the computer device may input multiple frames of images into a trajectory generation algorithm to obtain a target trajectory. For example, the trajectory generation algorithm may be an instant localization and mapping (Simultaneous Localization and Mapping, SLAM) algorithm.

Step S103, determining a road surface reconstruction range corresponding to the target road surface based on the preset road surface width and the road surface coordinates corresponding to each track point in the target track.

In implementation, the computer device may determine the reconstruction range in the y direction according to the minimum y coordinate value and the maximum y coordinate value of the track point in the target track, determine the reconstruction range in the x direction according to the preset road width and the x coordinate value of each track point, and further determine the road reconstruction range according to the reconstruction range in the x direction and the reconstruction range in the y direction. For example, the road surface reconstruction range may be a range covered by [ x_min, x_max ] and [ y_min, y_max ].

Step S104, based on the preconfigured grid size, gridding the three-dimensional space corresponding to the road surface reconstruction range to obtain a plurality of grids corresponding to the road surface reconstruction range.

Wherein, the preconfigured grid size can be represented by the side length of the grid, and the side length of the grid can be 0.1 meter.

In implementation, the computer device may gridde the three-dimensional space corresponding to the road surface reconstruction range according to the preconfigured grid size, to obtain a plurality of grids corresponding to the road surface reconstruction range. For example, the number of the plurality of grids corresponding to the road surface reconstruction range may be int ((x_max-x_min)/0.1) ·int ((y_max-y_min)/0.1).

Step S105, determining a distance between every two adjacent track points based on the road surface coordinates corresponding to each track point in the target track.

In practice, for each two adjacent track points, the computer device may determine the absolute value of the difference in the y-axis coordinate values in the road coordinates of the two adjacent track points as the distance between the two adjacent track points.

Step S106, determining the coverage road surface range between every two adjacent track points based on the distance between every two adjacent track points and the preset road surface width.

In practice, for each two adjacent track points, the computer device may determine the product of the distance between the two adjacent track points and the preset road width as the coverage road range between the two adjacent track points, and determine the coordinate range corresponding to the coverage road range. For example, the coordinate ranges may be [ x1, x2] and [ y1, y2]. The coverage road surface range between all adjacent track points can constitute the road surface reconstruction range.

Step S107, selecting a plurality of grids covered by the road surface range from the plurality of grids corresponding to the road surface reconstruction range.

In implementations, the computer device may determine mesh vertices having coordinates within a coordinate range from the coordinate range corresponding to the coverage road surface range, the meshes corresponding to the mesh vertices constituting a plurality of meshes covered by the coverage road surface range.

Step S108, determining the road surface height corresponding to the target track point in the first group of two adjacent track points as the road surface height of each grid vertex in a plurality of grids covered by the road surface range corresponding to the first group of two adjacent track points.

The first group of two adjacent track points are any one group of two adjacent track points in the target track, and the target track point is the first track point in the first group of two adjacent track points.

In implementations, for a first set of two adjacent track points, the computer device may determine the target track point road surface height as the road surface height of each grid vertex in a plurality of grids covering the road surface range corresponding to the first set of two adjacent track points. Thus, the road surface height of each mesh vertex in the plurality of meshes corresponding to the road surface reconstruction range can be obtained.

According to the method for determining the height of the pavement, due to the fact that the height of the pavement is uneven, if the height of the pavement at the vertex of each grid in the multiple grids corresponding to the pavement is determined to be zero, the built pavement three-dimensional reconstruction model is inaccurate. Therefore, a target track is generated from images taken by the target vehicle during the travel of the target road surface, and each point in the target track corresponds to the pose of the target vehicle in the reference coordinate system to which the target road surface belongs, namely, the road surface coordinates and the road surface height. Further, by comparing the plurality of grids corresponding to the road surface reconstruction range with the plurality of grids corresponding to the coverage road surface range, the road surface height of each grid vertex in the plurality of grids corresponding to the road surface reconstruction range can be determined. Therefore, the accuracy of constructing the three-dimensional reconstruction model of the road surface can be improved by considering the fluctuation condition of the road surface.

In this embodiment, a method for determining the height of a road surface is provided, and the method can be used for a computer device, where the computer device may be a terminal, a server, etc., for example, a desktop computer, a tablet computer, etc. The computer device may be configured with a three-dimensional reconstruction model for implementing the method. Fig. 2 is a flowchart of a method of determining road surface height according to an embodiment of the present invention, as shown in fig. 2, the flowchart including the steps of:

Step S201, acquiring a multi-frame image captured by the target vehicle during the driving process of the target road surface, and a preset road surface width.

Please refer to step S101 in the embodiment shown in fig. 1 in detail, which is not described herein.

Step S202, acquiring a target track based on the multi-frame images.

Please refer to step S102 in the embodiment shown in fig. 1 in detail, which is not described herein.

Step S203, determining a road surface reconstruction range corresponding to the target road surface based on the preset road surface width and the road surface coordinates corresponding to each track point in the target track.

Specifically, the step S203 includes:

step S2031, obtaining a track point cloud corresponding to the target road surface and a road surface coordinate of each track point in the track point cloud based on the preset road surface width and the road surface coordinate of each track point in the target track.

In an implementation, the computer device may perform the point scattering in the width direction of the target road surface with the preset road surface width as a boundary according to the road surface coordinates of each track point in the target track as a center, so as to obtain a track point cloud corresponding to the target road surface and the road surface coordinates of each track point in the track point cloud. The process will be specifically described by taking a trace point as an example: the x-axis of the reference coordinate system is in the width direction of the target road surface, and the y-axis is in the extending direction of the target road surface. For the road surface coordinates of each track point, a plurality of x coordinate values can be obtained according to the preset road surface width and the x coordinate values of the track point, and a plurality of track point clouds can be obtained according to the plurality of x coordinate values and the y coordinate values of the track point. For example, the x coordinate value of the track point is 0, the y coordinate value is 5, the preset road width is 12 meters, two x coordinate values of-6 and 6 respectively can be obtained, and further, the road coordinates of two track point clouds are (-6, 5) and (6, 5) respectively.

Step S2032, determining a road reconstruction range based on the road coordinates of each track point in the track point cloud.

In practice, the computer device may determine the track points of the target road boundary based on the road coordinates of each track point. The trajectory point connection line of the boundary can form the road surface reconstruction range.

Step S204, based on the preconfigured mesh size, gridding the three-dimensional space corresponding to the road surface reconstruction range to obtain a plurality of meshes corresponding to the road surface reconstruction range.

In step S205, the distance between every two adjacent track points is determined based on the road surface coordinates corresponding to each track point in the target track.

Step S206, determining the coverage road surface range between every two adjacent track points based on the distance between every two adjacent track points and the preset road surface width.

Step S207, selecting a plurality of grids covered by the road surface range from the plurality of grids corresponding to the road surface reconstruction range.

Step S208, determining the road surface height corresponding to the target track point in the first group of two adjacent track points as the road surface height of each grid vertex in a plurality of grids covered by the road surface range corresponding to the first group of two adjacent track points.

Please refer to the steps S104 to S108 in the embodiment shown in fig. 1 in detail in the steps S204 to S208, which will not be described herein.

According to the method for determining the height of the road surface, because the overall shape of the road surface may not be regular, if the road surface reconstruction range is determined directly according to the rule, the determined road surface reconstruction range may be too large or too small, and when model reconstruction is carried out subsequently due to too large road surface reconstruction range, more processing resources are occupied, so that model reconstruction efficiency is low, and part of key information is omitted due to too small road surface reconstruction range. Therefore, the road surface reconstruction range is determined through the road surface coordinates of each track point in the target track and the preset road surface width, the actual condition of the target road surface can be considered, and the accuracy of constructing the road surface three-dimensional reconstruction model can be improved.

In the three-dimensional reconstruction process, the color values and semantic categories may represent key information of the road surface. Thus, in addition to determining the road coordinates and road heights of each mesh vertex in the flow shown in fig. 1, the color value and semantic category thereof may be determined for each mesh vertex in the process of three-dimensional reconstruction. In this embodiment, a method for updating color values and semantic categories is provided, and the method can be used for a computer device, where the computer device can be a terminal, a server, etc., for example, a desktop computer, a tablet computer, etc. The computer device may be configured with a three-dimensional reconstruction model for implementing the method. FIG. 3 is a flow chart of a method of updating color values and semantic categories according to an embodiment of the present invention, as shown in FIG. 3, the flow comprising the steps of:

Step S301, based on the multi-frame images, camera pose and camera configuration information corresponding to each frame of images are obtained.

Wherein each frame of image in the multi-frame images corresponds to one track point in the target track. The camera pose, namely the camera external parameters, can represent the information such as the position, the angle and the like when the camera shoots the image. The camera configuration information, i.e. camera internal parameters, may represent information such as focal length and pixel pitch when the camera captures an image.

In implementations, a computer device may obtain pose and configuration information of a camera when each frame of image is captured. Further, each frame of image can be input into a semantic segmentation model respectively, so that the semantic category and the semantic segmentation quality of each pixel in each frame of image can be obtained. The semantic segmentation model may be a depth network model, e.g., (Visual Geometry Group, VGG) visual geometry group model, (Residual Neural Network, resnet) residual network model, etc.

In the present embodiment, n+1 semantic categories are used, N represents semantic information of interest, such as road surface, lane line, arrow, zebra line, and the like, and 1 represents a background, that is, other scenes (e.g., vehicles in front, and the like) than a road surface-related scene. The semantic category is represented by a binary value of n+1 bits (i.e., "0" or "1"), that is, the technician specifies in advance the semantic category corresponding to each binary value, for example, the first bit corresponds to the road surface, the second bit corresponds to the lane line, the third bit corresponds to the arrow, etc., when the first bit of a certain pixel is "1", the other bits are all "0", which indicates that the pixel is the pixel corresponding to the road surface.

Step S302, for a first frame image in the multi-frame images, determining an azimuth vector of each pixel in the first frame image under the reference coordinate system based on the camera pose, the camera configuration information, and the track point corresponding to the first frame image.

Wherein the first frame image is any frame image in the multi-frame images.

The specific steps of step S302 may be as shown in fig. 4, and include:

for a first frame image in the multi-frame images, determining coordinate system conversion parameters corresponding to the first frame image based on the camera pose and the track point corresponding to the first frame image;

In step S3021, for a first frame image in the multi-frame images, a coordinate system conversion parameter corresponding to the first frame image is determined based on the pose and the locus point of the camera corresponding to the first frame image.

The coordinate system conversion parameter may be a coordinate conversion matrix, and the coordinate conversion matrix may include information such as a rotation angle and a translation distance.

In implementation, since the image corresponds to a camera coordinate system, the multiple grids corresponding to the road surface reconstruction range are under a reference coordinate system, and the color value and the semantic category of each grid vertex in the multiple grids corresponding to the road surface reconstruction range need to be determined according to the image in subsequent processing, so that the image needs to be converted into the reference coordinate system. The pose of the camera may represent the position of the camera relative to the target vehicle, the road surface coordinates of the track point may represent the position of the target vehicle relative to the target road surface, and further, the position (coordinate system conversion parameter) of the camera relative to the target road surface may be determined by the pose of the camera and the road surface coordinates of the track point, that is, the transformation of each frame of image from the camera coordinate system to the reference coordinate system is determined.

In step S3022, based on the camera configuration information corresponding to the first frame image, an azimuth vector of each pixel in the first frame image under the camera coordinate system to which the pixel belongs is determined.

In implementation, the computer device may perform projective transformation on each pixel in each frame of image according to the camera configuration information corresponding to the image to obtain an azimuth vector of each pixel under the camera coordinate system to which the pixel belongs.

In step S3023, based on the coordinate system conversion parameters corresponding to the first frame image, coordinate conversion is performed on the azimuth vector of each pixel in the first frame image under the camera coordinate system to which the pixel belongs, so as to obtain the azimuth vector of each pixel in the first frame image under the reference coordinate system.

In implementation, for each frame of image, the computer device may perform translation and/or rotation transformation on the azimuth vector of each pixel in the image under the camera coordinate system to which the pixel belongs according to the coordinate system transformation parameter corresponding to the image, so as to obtain the azimuth vector of each pixel under the reference coordinate system.

Step S303, when there is an intersection point between the azimuth vector of the pixel in at least one frame of image under the reference coordinate system and any one of a plurality of grids corresponding to the road surface reconstruction range, determining a predicted color value of the intersection point based on the color value of each grid vertex in the grid corresponding to the intersection point and a preconfigured interpolation algorithm in the current iteration period.

The color values may be Red Green Blue (RGB) color values, including Red color values, green color values, and Blue color values.

In an implementation, the computer device may determine whether an intersection point exists between an azimuth vector of a pixel in the at least one frame image under the reference coordinate system and any one of a plurality of grids corresponding to the road surface reconstruction range, and when it is determined that an intersection point exists between an azimuth vector of a pixel in the at least one frame image under the reference coordinate system and any one of a plurality of grids corresponding to the road surface reconstruction range, the pixel corresponding to the intersection point and the grid where the intersection point is located correspond to the same position of the target road surface.

In the current iteration period, the computer equipment can determine the color value and the road surface coordinate of the grid vertex corresponding to the intersection point and the coordinate of the intersection point, and input the color value and the road surface coordinate into a preconfigured interpolation algorithm to obtain the predicted color value of the intersection point.

Step S304, determining the semantic category of the grid vertex closest to the intersection point as the predicted semantic category of the intersection point.

In implementations, the computer device may calculate the distance of the intersection point from each mesh vertex of the mesh in which it is located, and further determine the semantic class of the mesh vertex closest to the intersection point as the predicted semantic class of the intersection point.

Step S305, determining a first loss value based on the predicted color value, the predicted semantic class, the true color value of the pixel corresponding to the intersection point in the image to which the pixel belongs, and the true semantic class.

The specific steps of step S305 may include, as shown in fig. 5:

in step S3051, a second loss value is determined based on the predicted color value, the true color value of the pixel corresponding to the intersection point in at least one frame of image, and the first loss function.

Wherein the first loss function may take the form of an L1 norm or an L2 norm.

In practice, since the image includes the true color value of each pixel in the image, and the same grid corresponding to the road surface may correspond to multiple pixels in different images. Therefore, there are a plurality of true color values corresponding to each intersection point. When only one real color value is corresponding to the intersection point, the absolute value of the difference value between the real color value and the predicted color value can be determined as the loss value corresponding to the intersection point. When there are multiple real color values corresponding to the intersection point, the absolute value of the difference value between each real color value and the predicted color value can be determined, and further, the average value of the absolute values of the difference values is calculated and determined as the loss value corresponding to the intersection point. And finally, summing or averaging the loss values corresponding to each intersection point to obtain a second loss value.

Step S3052, determining a third loss value based on the true semantic class of the pixel corresponding to the predicted semantic class and the intersection point in the at least one frame of image and the second loss function.

Wherein the second loss function may employ a binary cross entropy (Binary Cross Entropy) loss function.

In practice, since the image includes the true semantic class of each pixel in the image, and one grid in the road reconstruction range may correspond to multiple pixels in different images. Thus, there are multiple true semantic categories for each intersection. When only one real semantic category corresponds to the intersection point, the loss value corresponding to the intersection point can be determined according to the difference between the real semantic category and the predicted semantic category. When there are multiple real color values corresponding to the intersection point, the loss value corresponding to the intersection point can be determined according to the difference between each real semantic category and the predicted semantic category. And finally, summing or averaging the loss values corresponding to each intersection point to obtain a third loss value.

Step S3053, determining a semantic category weight value based on the pre-acquired semantic segmentation quality of each pixel.

In implementations, the computer device may determine an overall semantic segmentation quality based on the semantic segmentation quality of each pixel. The semantic segmentation quality is proportional to the semantic category weight value.

Step S3054, determining a first loss value based on the second loss value, the third loss value, the semantic category weight value, and the third loss function.

The third loss function uses the following expression:

L=loss_rgb +w2•loss_sem……（1）

wherein L is a first loss value, loss_rgb is a second loss value, w2 is a semantic category weight value, and loss_semm is a third loss value.

Step S306, updating the color value of each grid vertex in the grids corresponding to the road surface reconstruction range based on a preset gradient algorithm and a predicted color value, and updating the semantic category of each grid vertex in the grids corresponding to the road surface reconstruction range based on the preset gradient algorithm and the predicted semantic category.

In an implementation, the computer device may update the color value of each mesh vertex in the plurality of meshes corresponding to the road reconstruction range according to a preset gradient algorithm and the predicted color value corresponding to each intersection point. The computer equipment can update the semantic category of each grid vertex in the multiple grids corresponding to the road surface reconstruction range according to a preset gradient algorithm and the predicted semantic category corresponding to each intersection point.

In step S307, when it is determined that the first loss value satisfies the convergence condition, the iterative process is stopped.

In practice, the convergence condition is determined to be satisfied after the computer device determines that the variation of the first loss value output by the continuous n number of iterations is within the preset range (i.e., the difference between the true value and the predicted value tends to stabilize). Wherein n is a preset number of times. Or when the computer equipment determines that the first loss value output by the iterative process of the round is smaller than or equal to the preset threshold value, determining that the convergence condition is met.

Step S308, when the first loss value is determined to not meet the convergence condition, entering a next iteration period, and updating the color value and the semantic category of each grid vertex in the multiple grids corresponding to the reconstruction range again.

In implementation, when the computer device determines that the first loss value output by the present iteration process is greater than the preset threshold, the next iteration cycle is entered, and steps S303 to S308 are repeated.

According to the method for updating the color value and the semantic category, the image comprises the color value information of each pixel, and the semantic category of each pixel in the image can be obtained by carrying out semantic segmentation on the image. Therefore, through the comparison of the grids and the pixels, the color value and the semantic category of each grid vertex in the multiple grids corresponding to the pavement reconstruction range can be determined. Further, the three-dimensional model reconstruction of the road surface is performed by combining the road surface coordinates, the road surface height, the color values and the semantic categories of each grid vertex, so that a more accurate three-dimensional reconstruction model of the road surface can be obtained.

After updating the color value and semantic class of each grid vertex, the grid vertices can be further subjected to dense operation to obtain finer pavement grids. Thus, in the present embodiment, a method for dense mesh vertices is provided, which may be used for a computer device, such as a terminal, a server, etc., for example, a desktop computer, a tablet computer, etc. The computer device may be configured with a three-dimensional reconstruction model for implementing the method. FIG. 6 is a flow chart of a method of dense mesh vertices according to an embodiment of the invention, as shown in FIG. 6, the flow comprising the steps of:

step S601, obtaining a target semantic category.

In implementations, a technician can enter a target semantic category on a computer device. When the computer equipment detects the operation instruction corresponding to the output operation, the target semantic category can be obtained from the operation instruction. For example, the target semantic category may be an arrow.

Step S602, selecting grid vertexes with semantic categories as target semantic categories from a plurality of grids corresponding to the pavement reconstruction range based on the target semantic categories, and forming a target grid vertex set corresponding to the first round of dense operation.

In an implementation, the computer device may compare the semantic class of each mesh vertex in the plurality of meshes corresponding to the road surface reconstruction range with the target semantic class, and determine that the semantic class is the mesh vertex of the target semantic class. Further, the computer device determines the grid vertices with the semantic classes being target semantic classes as a target grid vertex set corresponding to the first round of dense operation.

Step S603, based on the road surface coordinates and the road surface height of each grid vertex in the target grid vertex set corresponding to the first round of dense operation, performing dense operation on each grid vertex in the target grid vertex set corresponding to the first round of dense operation, and obtaining a dense grid vertex set corresponding to the first round of dense operation.

In implementation, for each grid vertex in the target grid vertices, the computer device may determine, according to the road surface height of the grid vertex, a tangent plane corresponding to the grid vertex, and on the tangent plane, add a preset disturbance to the road surface coordinates of the grid vertex, so as to obtain a new coordinate point. These new coordinate points constitute a dense set of mesh vertices. For example, the road surface coordinates of the grid vertices are (x, y), the preset perturbations may include Δx and Δy, the new coordinate points may be (x+Δx, y), (x, y+Δy), (x- Δx, y), (x, y- Δy), and so on. In addition, the color value of the new coordinate point may be the same as the color value of the mesh vertex, and the semantic category of the new coordinate point may be the same as the semantic category of the mesh vertex.

Step S604, fusing the target grid vertex set and the dense grid vertex set corresponding to the first round of dense operation to obtain a first sampling point set corresponding to the first round of dense operation.

Step S605, based on the preset sampling number corresponding to the first round of dense operation, sampling the grid vertexes in the first sampling set to obtain a grid vertex set corresponding to the next round of dense operation.

In implementation, the computer device samples grid vertices with preset sampling numbers in the first sampling point set to obtain grid vertices corresponding to the next round of dense operation.

In step S606, in the non-first round of dense operation, based on the road surface coordinates and the road surface height corresponding to each grid vertex in the target grid vertex set of the previous round, performing dense operation on each grid vertex in the target grid vertex set of the previous round, to obtain a dense grid vertex set corresponding to the current round.

This step is similar to the specific process of step S603 described above, and will not be described here again.

Step S607, fusion is performed on the target mesh vertex set corresponding to the present round of dense operation and the dense mesh vertex set corresponding to the present round of dense operation, so as to obtain the sampling point set corresponding to the present round of dense operation.

This step is similar to the specific process of step S604 described above, and will not be described here again.

In step S608, when the number of operations corresponding to the present round of dense operation is equal to the preset number of operations, sampling operation is performed on the set of sampling points corresponding to the present round of dense operation, so as to obtain the set of target grid vertices.

In an implementation, after each time of performing the dense operation, the computer device may add one to the number of dense operations until the number of dense operations is equal to the preset number of operations, and perform sampling operation on the set of sampling points corresponding to the current round of dense operations, to obtain the set of target grid vertices.

Step S609, merging the target grid vertex set and a plurality of grids corresponding to the pavement reconstruction range, and performing triangulation processing to obtain a plurality of grids corresponding to the densified pavement reconstruction range, wherein the grids corresponding to the densified pavement reconstruction range form a three-dimensional reconstruction model of the target pavement.

In implementation, the computer device may combine the grid vertices in the target grid vertex set obtained after the last densification operation with the multiple grids corresponding to the original road surface reconstruction range to obtain multiple combined grids. Further, the computer equipment can triangulate the combined grids through a preset triangularization algorithm to obtain a plurality of grids corresponding to the thickened pavement reconstruction range.

In some possible implementations, after obtaining the multiple grids corresponding to the densified pavement reconstruction range, the processing from step S304 to step S308 may be performed again. As shown in fig. 7, steps S304 to S308 are first cycles, steps S602 to S609 are second cycles, and the first and second cycles constitute third cycles. Thus, by continuously and circularly processing, a more accurate three-dimensional reconstruction model of the target pavement can be obtained.

In the method for dense grid vertices provided in this embodiment, because some semantic categories need to pay attention to (i.e., target semantic categories), corresponding grid vertices need to be more to construct finer grids. Thus, by performing multiple rounds of dense operations on mesh vertices of the target semantic class, the mesh of the target semantic class may be refined. Further, the accuracy of the three-dimensional reconstruction model of the constructed pavement is higher.

In this embodiment, a device for determining the height of the road surface is further provided, and the device is used to implement the foregoing embodiments and preferred embodiments, and will not be described again. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

The present embodiment provides an apparatus for determining a road surface height, as shown in fig. 8, including:

an acquisition module 801, configured to acquire a multi-frame image captured by a target vehicle during a target road surface driving process, and a preset road surface width; acquiring a target track based on a multi-frame image, wherein the target track comprises track coordinates corresponding to a plurality of track points respectively and road surface heights corresponding to the track points respectively, and the track coordinates are used for indicating road surface coordinates of a target vehicle in a reference coordinate system corresponding to a target road surface;

a determining module 802, configured to determine a road reconstruction range corresponding to the target road based on the preset road width and the road coordinates corresponding to each track point in the target track;

the gridding module 803 is configured to gridde the three-dimensional space corresponding to the road surface reconstruction range based on the preconfigured grid size, so as to obtain a plurality of grids corresponding to the road surface reconstruction range;

a determining module 802, configured to determine a distance between every two adjacent track points based on the road coordinates corresponding to each track point in the target track; determining a coverage road surface range between every two adjacent track points based on the distance between every two adjacent track points and a preset road surface width;

A selecting module 804, configured to select a plurality of grids covered by the road surface range from a plurality of grids corresponding to the road surface reconstruction range;

the determining module 802 is configured to determine a road surface height corresponding to a target track point in the first set of two adjacent track points as a road surface height of each grid vertex in a plurality of grids covered by a road surface range corresponding to the first set of two adjacent track points, where the first set of two adjacent track points are any one set of two adjacent track points in the target track, and the target track point is a first track point in the first set of two adjacent track points.

In an alternative embodiment, determination module 802 is configured to:

and determining a road surface reconstruction range based on the road surface coordinates of each track point in the track point cloud.

In an optional implementation manner, each grid vertex in the multiple grids corresponding to the pavement reconstruction range further includes a color value and a semantic category, and the obtaining module 801 is further configured to obtain, based on multiple frame images, a camera pose and camera configuration information corresponding to each frame image, where each frame image in the multiple frame images corresponds to one track point in the target track;

The determining module 802 is further configured to determine, for a first frame image in the multiple frame images, an azimuth vector of each pixel in the first frame image under a reference coordinate system based on a camera pose, camera configuration information, and a track point corresponding to the first frame image, where the first frame image is any frame image in the multiple frame images; when an intersection point exists between an azimuth vector of a pixel in at least one frame of image under a reference coordinate system and any one of a plurality of grids corresponding to a road surface reconstruction range, determining a predicted color value of the intersection point based on a color value of each grid vertex in the grid corresponding to the intersection point and a preconfigured interpolation algorithm in a current iteration period; determining the semantic category of the grid vertex closest to the intersection point as the predicted semantic category of the intersection point; determining a first loss value based on the predicted color value, the predicted semantic class, the true color value of the pixel corresponding to the intersection point in the image to which the pixel belongs, and the true semantic class; updating the color value of each grid vertex in the multiple grids corresponding to the road reconstruction range based on a preset gradient algorithm and a predicted color value, and updating the semantic category of each grid vertex in the multiple grids corresponding to the road reconstruction range based on the preset gradient algorithm and a predicted semantic category; stopping the iterative process when the first loss value is determined to meet the convergence condition; or when the first loss value is determined to not meet the convergence condition, entering a next iteration period, and updating the color value and the semantic category of each grid vertex in the multiple grids corresponding to the reconstruction range again.

In an alternative embodiment, determination module 802 is configured to:

determining a third loss value based on the predicted semantic class and the true semantic class of the pixel corresponding to the intersection point in at least one frame of image and the second loss function;

the first penalty value is determined based on the second penalty value, the third penalty value, the semantic category weight value, and the third penalty function.

L=loss_rgb +w2•loss_sem……（1）

In an alternative embodiment, the apparatus further comprises a densification module 805:

an obtaining module 801, configured to obtain a target semantic category;

the dense module 805 is configured to select, from a plurality of grids corresponding to the pavement reconstruction range, a grid vertex whose semantic class is a target semantic class, and form a target grid vertex set corresponding to a first round of dense operation; the method comprises the steps of performing first-round dense operation on each grid vertex in a target grid vertex set corresponding to the first-round dense operation on the basis of the road surface coordinates and the road surface heights of each grid vertex in the target grid vertex set corresponding to the first-round dense operation, and obtaining a dense grid vertex set corresponding to the first-round dense operation; fusing the target grid vertex set and the dense grid vertex set corresponding to the first round of dense operation to obtain a first sampling point set corresponding to the first round of dense operation; sampling grid vertexes in a first sampling set based on a preset sampling number corresponding to the first round of dense operation to obtain a grid vertex set corresponding to the next round of dense operation; in the non-first round of dense operation, carrying out dense operation on each grid vertex in the target grid vertex set of the previous round based on the road surface coordinates and the road surface height corresponding to each grid vertex in the target grid vertex set of the previous round to obtain a dense grid vertex set corresponding to the current round; the method comprises the steps of fusing a target grid vertex set corresponding to a round of dense operation and a dense grid vertex set corresponding to the round of dense operation to obtain a sampling point set corresponding to the round of dense operation; when the operation times corresponding to the dense operation of the round is equal to the preset operation times, sampling the sampling point set corresponding to the dense operation of the round to obtain a target grid vertex set; combining the target grid vertex set with a plurality of grids corresponding to the pavement reconstruction range, and performing triangularization treatment to obtain a plurality of grids corresponding to the densified pavement reconstruction range, wherein the plurality of grids corresponding to the densified pavement reconstruction range form a three-dimensional reconstruction model of the target pavement.

Further functional descriptions of the above respective modules and units are the same as those of the above corresponding embodiments, and are not repeated here.

The means for determining road surface height in this embodiment is presented in the form of functional units, herein referred to as ASIC (Application Specific Integrated Circuit ) circuits, processors and memories executing one or more software or firmware programs, and/or other devices capable of providing the above described functionality.

The embodiment of the invention also provides computer equipment, which is provided with the device for determining the road surface height shown in the figure 8.

Referring to fig. 9, fig. 9 is a schematic structural diagram of a computer device according to an alternative embodiment of the present invention, as shown in fig. 9, the computer device includes: one or more processors 10, memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are communicatively coupled to each other using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the computer device, including instructions stored in or on memory to display graphical information of the GUI on an external input/output device, such as a display device coupled to the interface. In some alternative embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple computer devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 10 is illustrated in fig. 9.

The processor 10 may be a central processor, a network processor, or a combination thereof. The processor 10 may further include a hardware chip, among others. The hardware chip may be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general-purpose array logic, or any combination thereof.

Wherein the memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods shown in implementing the above embodiments.

The memory 20 may include a storage program area that may store an operating system, at least one application program required for functions, and a storage data area; the storage data area may store data created based on the use of the computer device, etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, memory 20 may optionally include memory located remotely from processor 10, which may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

Memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid state disk; the memory 20 may also comprise a combination of the above types of memories.

The computer device further comprises input means 30 and output means 40. The processor 10, memory 20, input device 30, and output device 40 may be connected by a bus or other means, for example by a bus connection in fig. 9.

The input device 30 may receive input numeric or character information and generate key signal inputs related to user settings and function control of the computer apparatus, such as a touch screen, a keypad, a mouse, a trackpad, a touch pad, a pointer stick, one or more mouse buttons, and the like. The output means 40 may comprise a display device or the like. Such display devices include, but are not limited to, liquid crystal displays, light emitting diodes, displays and plasma displays. In some alternative implementations, the display device may be a touch screen.

The embodiments of the present invention also provide a computer readable storage medium, and the method according to the embodiments of the present invention described above may be implemented in hardware, firmware, or as a computer code which may be recorded on a storage medium, or may be originally stored in a remote storage medium or a non-transitory machine-readable storage medium and to be stored in a local storage medium, downloaded through a network, so that the method described herein may be stored on such software processes on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, a solid state disk or the like; further, the storage medium may also comprise a combination of memories of the kind described above. It will be appreciated that a computer, processor, microprocessor controller or programmable hardware includes a storage element that can store or receive software or computer code that, when accessed and executed by the computer, processor or hardware, implements the methods illustrated by the above embodiments.

Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope of the invention as defined by the appended claims.

Claims

1. A method of determining road surface elevation, the method comprising:

determining the road surface height corresponding to a target track point in a first group of two adjacent track points as the road surface height of each grid vertex in a plurality of grids covered by the road surface range corresponding to the first group of two adjacent track points, wherein the first group of two adjacent track points are any group of two adjacent track points in the target track, and the target track point is the first track point in the first group of two adjacent track points;

obtaining a target semantic category;

2. The method according to claim 1, wherein the determining the road surface reconstruction range corresponding to the target road surface based on the preset road surface width and the road surface coordinates corresponding to each track point in the target track includes:

3. The method of claim 2, wherein each mesh vertex of the plurality of meshes corresponding to the road reconstruction range further comprises a color value and a semantic category, the method further comprising:

4. A method according to claim 3, wherein said determining, for a first frame image of said plurality of frame images, an orientation vector of each pixel of said first frame image in said reference frame based on said camera pose corresponding to said first frame image, said camera configuration information, and a locus point, comprises:

5. A method according to claim 3, wherein said determining a first loss value based on the predicted color value, the predicted semantic class, the true color value and the true semantic class of the pixel corresponding to the intersection in the image to which it belongs comprises:

6. The method of claim 5, wherein the third loss function uses the expression:

L=loss_rgb +w2·loss_sem（1）

7. An apparatus for determining road surface elevation, the apparatus comprising:

the determining module is configured to determine a road surface height corresponding to a target track point in a first set of two adjacent track points as a road surface height of each grid vertex in a plurality of grids covered by a road surface range corresponding to the first set of two adjacent track points, where the first set of two adjacent track points are any one set of two adjacent track points in the target track, and the target track point is a first track point in the first set of two adjacent track points;

The acquisition module is used for acquiring the target semantic category;

the selecting module is used for selecting grid vertexes with semantic categories being the target semantic categories from a plurality of grids corresponding to the pavement reconstruction range to form a target grid vertex set corresponding to first round dense operation;

the dense module is used for respectively carrying out dense operation on each grid vertex in the target grid vertex set corresponding to the first round of dense operation based on the road surface coordinates and the road surface height of each grid vertex in the target grid vertex set corresponding to the first round of dense operation to obtain a dense grid vertex set corresponding to the first round of dense operation;

the fusion module is used for fusing the target grid vertex set corresponding to the first round of dense operation and the dense grid vertex set to obtain a first sampling point set corresponding to the first round of dense operation;

the sampling module is used for sampling grid vertexes in the first sampling set based on the preset sampling quantity corresponding to the first round of dense operation to obtain a grid vertex set corresponding to the next round of dense operation;

the dense module is used for carrying out dense operation on each grid vertex in the target grid vertex set of the previous round based on the road surface coordinates and the road surface height corresponding to each grid vertex in the target grid vertex set of the previous round in the non-first round dense operation to obtain a dense grid vertex set corresponding to the current round;

the sampling module is used for sampling the sampling point set corresponding to the round of dense operation when the operation times corresponding to the round of dense operation are equal to the preset operation times, and obtaining a target grid vertex set;

and the merging module is used for merging the target grid vertex set with a plurality of grids corresponding to the pavement reconstruction range, and performing triangularization treatment to obtain a plurality of grids corresponding to the pavement reconstruction range after densification, wherein the plurality of grids corresponding to the pavement reconstruction range after densification form a three-dimensional reconstruction model of the target pavement.

8. A computer device, comprising:

a memory and a processor in communication with each other, the memory having stored therein computer instructions which, upon execution, perform the method of determining road surface height of any of claims 1 to 6.

9. A computer-readable storage medium having stored thereon computer instructions for causing a computer to perform the method of determining road surface height according to any one of claims 1 to 6.