CN117197766A

CN117197766A - Method and device for detecting pavement pits, processor and vehicle

Info

Publication number: CN117197766A
Application number: CN202311276901.9A
Authority: CN
Inventors: 刘润华; 马昌训; 侯力玮; 喻逊
Original assignee: Hunan Zoomlion Intelligent Aerial Work Machinery Co Ltd
Current assignee: Hunan Zoomlion Intelligent Aerial Work Machinery Co Ltd
Priority date: 2023-09-28
Filing date: 2023-09-28
Publication date: 2023-12-08

Abstract

The embodiment of the invention provides a method and a device for detecting a pavement pit, a processor and a vehicle, and belongs to the technical field of vehicles. The vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the method for detecting the pavement pits comprises the following steps: acquiring color images and point cloud data corresponding to the road surface to be detected, which are acquired by an image acquisition device and a point cloud data acquisition device respectively; detecting the color image through a pre-trained hole detection model to obtain position information of an external rectangular area corresponding to the hole, wherein the hole detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network; determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area; and determining the length and the width of the pit according to the point cloud data corresponding to the position information. The embodiment of the invention can accurately detect the pits.

Description

Method and device for detecting pavement pits, processor and vehicle

Technical Field

The invention relates to the technical field of vehicles, in particular to a method and a device for detecting a pavement pit, a processor and a vehicle.

Background

In the running process of the vehicle, if the road is uneven, the running of the vehicle is difficult, if a pit with a deeper depth is encountered on the running route of the vehicle, the vehicle can flameout and cannot normally run, and the safety of personnel in the vehicle can be influenced. In order to detect a hole in a travel route, a distance measuring sensor is usually installed on a vehicle, and it is determined whether or not a hole exists in front of the vehicle by changing the distance of a road on the travel route. However, the detection of the pit by the distance measuring sensor is affected by the distance between the vehicle and the pit, and the dimensional information such as the depth of the pit cannot be accurately detected when the vehicle is far from the pit, so that a more accurate detection of the pit is required.

Disclosure of Invention

The embodiment of the invention aims to provide a method and a device for detecting a pavement pit, a method and a device for early warning of the pavement pit, a processor, a vehicle and a storage medium, so as to solve the problems in the prior art.

In order to achieve the above object, a first aspect of an embodiment of the present invention provides a method for detecting a pit on a road surface, which is applied to a vehicle, wherein the vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the method includes:

Acquiring color images and point cloud data corresponding to the road surface to be detected, which are acquired by an image acquisition device and a point cloud data acquisition device respectively;

detecting the color image through a pre-trained hole detection model to obtain position information of an external rectangular area corresponding to the hole, wherein the hole detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network;

determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area;

and determining the length and the width of the pit according to the point cloud data corresponding to the position information.

In an embodiment of the invention, the lightweight convolutional neural network includes an ECA attention mechanism.

In the embodiment of the invention, the path aggregation network in the neck network of the improved YOLOv5 model is replaced by a bidirectional feature pyramid network.

In the embodiment of the invention, the position information comprises angular point position information and center position information; detecting the color image through a pre-trained pit detection model to obtain position information of an external rectangular area corresponding to the pit, wherein the method comprises the following steps: detecting the color image through a pre-trained pit detection model to obtain angular point position information of an external rectangular area corresponding to the pit; and determining the central position information of the circumscribed rectangular area according to the angular point position information.

In the embodiment of the invention, determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area comprises the following steps: acquiring a first height value in the point cloud data corresponding to the central position information and a second height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area; and determining the absolute value of the difference between the first height value and the second height value to obtain the depth of the pit.

A second aspect of the embodiment of the present invention provides a method for early warning a hole in a road surface, which is applied to a vehicle, and the method includes:

acquiring size information of the pit, wherein the size information comprises depth, length and width, and the size information is determined by the method for detecting the road surface pit;

determining that the type of the pit is a type of the non-passable pit according to the size information and a preset size threshold;

determining the distance between the pit and the vehicle according to the point cloud data corresponding to the position information;

determining a corresponding target early warning strategy according to the distance;

and controlling the engineering machinery to execute corresponding early warning actions according to the target early warning strategy.

In the embodiment of the invention, determining that the type of the pit is a type of the non-passable pit according to the size information and the preset size threshold comprises the following steps: and determining that the type of the pit is a non-passable pit type under the condition that the depth, the length and the width of the pit are larger than the corresponding preset size threshold values.

In the embodiment of the invention, the corresponding target early warning strategy is determined according to the distance, and the method comprises the following steps: determining a distance interval in which the distance is located; and determining an early warning strategy corresponding to the distance interval as a target early warning strategy.

In the embodiment of the invention, the distance interval comprises a first distance interval, a second distance interval and a third distance interval which are far and near, the early warning strategy corresponding to the first distance interval comprises sending prompt information indicating that the front road surface has an unviable pit, the early warning strategy corresponding to the second distance interval comprises sending prompt information indicating that the vehicle approaches the unviable pit of the front road surface and the running track of the vehicle needs to be changed, and the early warning strategy corresponding to the third distance interval comprises controlling the vehicle to decelerate or stop.

A third aspect of the embodiment of the present invention provides a processor configured to perform the method for detecting a road hole according to the above-described embodiment or the method for early warning a road hole according to the above-described embodiment.

A fourth aspect of the present invention provides a device for detecting a road hole, applied to a vehicle, where the vehicle is provided with an image acquisition device and a point cloud data acquisition device, including:

The data acquisition module is used for acquiring color images and point cloud data corresponding to the road surface to be detected, which are acquired by the image acquisition device and the point cloud data acquisition device respectively;

the pit detection module is used for detecting the color image through a pre-trained pit detection model so as to obtain the position information of the circumscribed rectangular area corresponding to the pit; the pit detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network;

the size determining module is used for determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area; and determining the length and the width of the pit according to the point cloud data corresponding to the position information.

A fifth aspect of the embodiment of the present invention provides a device for pre-warning a pit on a road surface, which is applied to a vehicle, and includes:

the size acquisition module is used for acquiring size information of the pits, wherein the size information comprises depth, length and width, and the size information is determined by the method for detecting the pavement pits;

the type determining module is used for determining that the type of the pit is a type of the non-passable pit according to the size information and a preset size threshold;

The distance determining module is used for determining the distance between the pit and the vehicle according to the point cloud data corresponding to the position information;

the strategy determining module is used for determining a corresponding target early warning strategy according to the distance;

and the strategy execution module is used for controlling the engineering machinery to execute corresponding early warning actions according to the target early warning strategy.

A sixth aspect of an embodiment of the present invention provides a vehicle, including: an image acquisition device; the point cloud data acquisition device; and the device for detecting the pavement pits or the device for early warning the pavement pits.

A seventh aspect of the embodiments of the present invention provides a machine-readable storage medium, on which a program or an instruction is stored, which when executed by a processor, implements a method for detecting a road hole according to the above or a method for early warning a road hole according to the above.

According to the technical scheme, through the color image and the point cloud data which are respectively acquired by the image acquisition device and the point cloud data acquisition device and correspond to the road surface to be detected, the color image is detected through the pre-trained improved YOLOv5 model, the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced by the lightweight convolutional neural network, so that the position information of the external rectangular area corresponding to the pit is obtained, the depth of the pit is determined according to the point cloud data corresponding to the position information of the external rectangular area and the point cloud data corresponding to the peripheral area of the external rectangular area, the length and the width of the pit are determined according to the position information of the external rectangular area, the size information such as the depth of the pit can be accurately detected, the pit of the road surface is detected through the improved YOLOv5 model, the model is light, the number of network parameters and the number of floating point calculation times can be reduced, the calculation amount of model parameters and the calculation cost can be reduced, and the detection speed and the real-time performance is improved.

Additional features and advantages of embodiments of the invention will be set forth in the detailed description which follows.

Drawings

The accompanying drawings are included to provide a further understanding of embodiments of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain, without limitation, the embodiments of the invention. In the drawings:

FIG. 1 schematically illustrates a flow chart of a method for detecting a roadway hole in an embodiment of the present invention;

FIG. 2 schematically illustrates a flow chart of a method for roadway pit warning in an embodiment of the present invention;

FIG. 3 schematically illustrates a flow chart of a method for detecting and warning a roadway hole in accordance with another embodiment of the present invention;

FIG. 4 schematically illustrates a structural diagram of a lightweight convolutional neural network in an embodiment of the present invention;

FIG. 5 schematically illustrates a schematic structure of a lightweight convolutional neural network in another embodiment of the present invention;

FIG. 6 schematically illustrates a schematic diagram of the mechanism of attention of an ECA in an embodiment of the present invention;

FIG. 7 schematically illustrates a structural diagram of an improved YOLOv5 model in accordance with one embodiment of the present invention;

FIG. 8 schematically illustrates a pothole and circumscribed rectangular area in an embodiment of the invention;

FIG. 9 is a schematic block diagram schematically showing an apparatus for detecting a road surface pit in an embodiment of the present invention;

fig. 10 schematically shows a block diagram of an apparatus for pre-warning a pit on a road surface according to an embodiment of the present invention.

Detailed Description

The following describes the detailed implementation of the embodiments of the present invention with reference to the drawings. It should be understood that the detailed description and specific examples, while indicating and illustrating the invention, are not intended to limit the invention.

It should be noted that, if directional indications (such as up, down, left, right, front, and rear … …) are included in the embodiments of the present invention, the directional indications are merely used to explain the relative positional relationship, movement conditions, etc. between the components in a specific posture (as shown in the drawings), and if the specific posture is changed, the directional indications are correspondingly changed.

In addition, if there is a description of "first", "second", etc. in the embodiments of the present invention, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.

Fig. 1 schematically shows a flow chart of a method for detecting a road surface pit in an embodiment of the invention. As shown in fig. 1, in an embodiment of the present invention, a method for detecting a pit on a road surface is provided, and the method is applied to a vehicle, where the vehicle is provided with an image acquisition device and a point cloud data acquisition device, and the method is applied to a processor for explanation, and the method may include the following steps:

step S102, color images and point cloud data corresponding to the road surface to be detected, which are simultaneously and respectively acquired by the image acquisition device and the point cloud data acquisition device, are acquired.

Step S104, detecting the color image through a pre-trained hole detection model to obtain the position information of the circumscribed rectangular area corresponding to the hole, wherein the hole detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network.

And S106, determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area.

Step S108, determining the length and the width of the pit according to the point cloud data corresponding to the position information.

It will be appreciated that the image acquisition device is used to acquire images of a road surface and may for example comprise a camera or the like. The point cloud data acquisition device is used for acquiring point cloud data corresponding to a road surface, and can comprise a laser radar and the like. The road surface to be detected is the road surface needing to be detected. The circumscribed rectangular area corresponding to the pit is an area where the circumscribed rectangle surrounding the pit is located, for example, the circumscribed rectangular area can be a minimum circumscribed rectangular frame surrounding the pit, and the circumscribed rectangular area can also be called a two-dimensional boundary rectangular frame. The position information of the circumscribed rectangular area refers to two-dimensional position information of the circumscribed rectangular area, and specifically may include angular point position information and/or center position information and/or rectangular boundary position information of the circumscribed rectangular area. The point cloud data corresponding to the position information is three-dimensional position information, i.e., three-dimensional coordinates, of which the two-dimensional position information is projected in a three-dimensional coordinate system (for example, a laser radar coordinate system). The peripheral area of the circumscribed rectangular area is a nearby area located outside the circumscribed rectangular area, for example, a rectangular area obtained by enlarging the circumscribed rectangular area by 1.1 times or 1.2 times belongs to the peripheral area of the circumscribed rectangular area.

The pre-trained pothole detection model is a pre-trained model for detecting potholes on a pavement in a color image, the pothole detection model is an improved YOLOv5 model, a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network, specifically, the lightweight convolutional neural network shufflenet V2 can comprise a Shufflev2-Block-a unit and a Shufflev2-Block-b unit, in the Shufflev2-Block-a unit, an input feature map of the number of C channels can be divided into two paths to be subjected to convolution operation respectively, concat operation is used for fusion after the convolution operation is finished, and a feature map of the number of 2C channels is output after channel shuffling, so that the network width is expanded on the premise of increasing fewer parameters. In the Shufflev2-Block-b unit, the input characteristic diagram of the number of C channels is divided into two parts, namely the characteristic diagrams of the number of C1 channels and the number of C2 channels are not changed, the characteristic diagram of the number of C2 channels passes through three convolution layers with the same input and output channels, wherein the convolution layers comprise two common convolution Conv2D of 1X1 and 1 depth separable convolution DWConv of 3X3, after two branches are finished, concat operation serial channels are carried out, the initial C channel characteristic diagrams are fused, and channel shuffling is utilized for information interaction.

Specifically, the processor may acquire the color image corresponding to the road surface to be detected and the point cloud data corresponding to the road surface to be detected, which are acquired by the image acquisition device and the point cloud data acquisition device respectively, so that the color image and the point cloud data are aligned in time. And inputting the color image into a pre-trained hole detection model, namely extracting features of the color image through the hole detection model to obtain position information of an external rectangular area corresponding to the hole output by the hole detection model on the color image, wherein the hole detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network. The processor may further compare a height value in the point cloud data corresponding to the position information of the circumscribed rectangular area corresponding to the pit with a height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area corresponding to the pit, for example, calculate a difference between the height value in the point cloud data corresponding to the position information of the circumscribed rectangular area corresponding to the pit and the height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area, and determine the maximum height difference as a depth of the pit.

According to the method for detecting the pavement pits, the color image and the point cloud data corresponding to the pavement to be detected are acquired through the image acquisition device and the point cloud data acquisition device respectively, pit detection is carried out on the color image through the pre-trained improved YOLOv5 model, the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced by the lightweight convolutional neural network, so that the position information of the circumscribed rectangular area corresponding to the pits is obtained, the depth of the pits is determined according to the point cloud data corresponding to the position information of the circumscribed rectangular area and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area, the length and the width of the pits are determined according to the position information of the circumscribed rectangular area, the size information such as the depth of the pits can be accurately detected, the light pits of the pavement can be detected through the improved YOLOv5 model, the number of network parameters and the number of floating point calculation times can be reduced, the calculation amount of model parameters and the calculation cost can be reduced, and the detection speed and the real-time performance of detection can be improved.

In one embodiment, the lightweight convolutional neural network may include an ECA attention mechanism.

It can be understood that the C3 feature extraction module in the backbone network of the existing YOLOv5 model comprises an SE attention mechanism, and the lightweight convolutional neural network in the backbone network of the improved YOLOv5 model in the embodiment of the invention adopts an ECA attention mechanism, so that the calculation amount and calculation cost of model parameters can be further reduced on the basis of ensuring the detection accuracy of an algorithm, and further the detection speed and the detection instantaneity are improved.

Specifically, adding an attention mechanism (Efficient Channel Attention, ECA) to the lightweight convolutional neural network ShuffleNetV2 creates a new feature extraction network ShuffleECA-Net, i.e., an ECA attention mechanism-based lightweight convolutional neural network, comprising: the ECA attention mechanism is respectively combined with a Sheffev 2-Block-a unit and a Sheffev 2-Block-b unit to be improved, a SheffeECAModule_1 module and a SheffeECAModule_2 module are formed, and the SheffeECA-Net backbone network is constructed by using the SheffeECAModule_1 module and the SheffeECAModule_2 module.

Compared with SE attention adopted by the original YOLOv5 model, the ECA attention mechanism can well adjust feature weights while avoiding frequent dimension reduction operation. The ECA attention mechanism acquires the interaction information of each channel and its k neighbors after global averaging pooling with a fast 1D convolution of convolution kernel k, where the convolution kernel k is adaptively acquired in direct proportion to the channel dimension, further reducing the parameter complexity by sharing the weights of all channels. In this calculation, w containing k×c parameters is used _k Array to learn attention, channel y _i Weight omega of (2) _i Only y is considered in the calculation process _i Interaction information between k neighbors is calculated as follows:

wherein the method comprises the steps ofRepresenting y _i σ is the Sigmoid activation function.

And carrying out weight sharing after information interaction, wherein the calculation formula is as follows:

the calculation process is realized by means of fast 1D convolution with a convolution kernel k (the size of k is self-adaptive) in an ECA structure, and the formula is as follows:

ω＝σ(C1D _k (y))

the C1D is the fast 1D convolution, so that the attention of the cross-channel information interaction can be effectively improved on the premise of not adding additional calculation parameters.

In one embodiment, the path aggregation network in the neck network of the modified YOLOv5 model is replaced with a bi-directional feature pyramid network.

Understandably, using a bi-directional feature pyramid network (bi-directional feature pyramid network, biFPN) instead of the path aggregation network (Path Aggregation Network, PANet) in the original YOLOv5 model, PANet adds essentially only different features, which would result in unequal weights of different sized features of the same type in the fused output features. Because the contribution weights of different input features to the output feature map should be different at each node in the feature fusion process, in the road pit detection, the shape of the pit is long and discontinuous, and the network is required to have stronger feature extraction capability. The BiFPN performs deep feature fusion from top to bottom and shallow feature fusion from bottom to top through bidirectional connection and weighted feature fusion, skips over certain intermediate layers, connects and fuses feature layers with different scales, and introduces learning weights to learn the importance of different input features, so that the feature extraction capability of a network is enhanced. The BiFPN gives different weight information to each feature map, a weighting mode of rapid normalization fusion is provided, the non-negative weight of each weight is ensured through a ReLU activation function, and a small epsilon (epsilon=0.0001) is added, so that the stability of data is ensured.

In one embodiment, the location information includes corner location information and center location information; detecting the color image through a pre-trained pit detection model to obtain position information of an external rectangular area corresponding to the pit, wherein the method comprises the following steps: detecting the color image through a pre-trained pit detection model to obtain angular point position information of an external rectangular area corresponding to the pit; and determining the central position information of the circumscribed rectangular area according to the angular point position information.

Specifically, the processor may detect the color image through a pre-trained pit detection model to obtain angular point position information of an external rectangular area corresponding to the pit, that is, position information of four angular points of the external rectangular area under image coordinates, and calculate central position information of a center of the external rectangular area according to the position information of the four angular points, for example, coordinates of the four angular points of the external rectangular area are A, B, C and D respectively, where (u 1, v 1) represents a position of an upper left corner a of the external rectangular area in an image coordinate system of the color image, and (u 2, v 2) represents a position of a lower right corner D of the external rectangular area in the image coordinate system of the color image, and further calculate coordinates of a central point E of the external rectangular area

In one embodiment, determining the depth of the pit according to the point cloud data corresponding to the position information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area includes: acquiring a first height value in the point cloud data corresponding to the central position information and a second height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area; and determining the absolute value of the difference between the first height value and the second height value to obtain the depth of the pit.

It may be understood that the point cloud data corresponding to the peripheral area of the circumscribed rectangular area may be point cloud data corresponding to any one point in the peripheral area of the circumscribed rectangular area, for example, the point cloud data corresponding to the peripheral area of the circumscribed rectangular area may be point cloud data corresponding to any one point in the rectangular area obtained by expanding the circumscribed rectangular area by 1.1 times, at this time, the second height value is a height value corresponding to the point, in addition, the point cloud data corresponding to the peripheral area of the circumscribed rectangular area may also be point cloud data corresponding to a plurality of points in the peripheral area of the circumscribed rectangular area, for example, the point cloud data corresponding to the peripheral area of the circumscribed rectangular area may be point cloud data corresponding to four corners in the rectangular area obtained by expanding the circumscribed rectangular area by 1.1 times, at this time, the second height value is an average value of the height values in the point cloud data corresponding to the four corners. The first height value is the height value of the central position information of the circumscribed rectangular area under the three-dimensional coordinates.

Specifically, the processor may obtain a first height value in the point cloud data corresponding to the central position information and a second height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area, calculate a difference value between the first height value and the second height value, and take an absolute value for the difference value, so as to obtain the depth of the pit.

The pit on the road surface seriously affects the driving comfort and safety of the vehicle, a driver of the vehicle has a visual field blind area on the road surface in front in the driving process, and if the vehicle encounters the pit during running or operation, the situations of machine overturning, instability, imbalance and the like are easily caused, and even safety accidents are caused. Therefore, a way to early warn of road pits is needed.

Fig. 2 schematically shows a flow chart of a method for pre-warning a pit in a road surface according to an embodiment of the invention. As shown in fig. 2, an embodiment of the present invention provides a method for pre-warning a pit on a road surface, which is applied to a vehicle, and the method is described by taking an application of the method to a processor as an example, and the method may include the following steps:

step S202, acquiring size information of pits.

Step S204, determining that the type of the pit is a type of the failed pit according to the size information and the preset size threshold.

Step S206, determining the distance between the pit and the vehicle according to the point cloud data corresponding to the position information.

Step S208, corresponding target early warning strategies are determined according to the distances.

Step S210, the engineering machinery is controlled to execute corresponding early warning actions according to the target early warning strategy.

It will be appreciated that the size information may include depth, length and width, and that the size information of the pits is determined by the method for detecting the pavement pits according to the above-described embodiment. The preset size threshold is a preset size threshold, and specifically may include a preset depth threshold, a preset length threshold, and a preset width threshold. The non-passable pit type is a type of pit through which the vehicle cannot pass. The target early warning strategy is a vehicle early warning mode determined based on the currently acquired size information of the pavement pits.

Specifically, the processor may determine the size information (including depth, length, and width) of the hole according to the method for detecting a road surface hole in the above embodiment, and determine the type of the hole as a non-passable hole type according to the size information and a preset size threshold, for example, may determine the type of the hole as a non-passable hole type when the depth in the size information is greater than a preset depth threshold, or may determine the type of the hole as a non-passable hole type when the length in the size information is greater than a preset length threshold, or may determine the type of the hole as a non-passable hole type when the width in the size information is greater than a preset width threshold. After determining that the type of the pit is an impenetrable pit type, the processor may determine a distance between the pit and the vehicle according to point cloud data corresponding to position information of an circumscribed rectangular area corresponding to the pit, for example, obtain projection coordinates (x_lidar, y_lidar, z_lidar) in a neighborhood of n×n (n is 3) of boundary coordinates of the circumscribed rectangular area near the vehicle in a laser radar coordinate system, and find an average value of y_lidar as the distance between the pit and the vehicle. The processor may further select a corresponding target early warning strategy according to the corresponding relation between the distance and the early warning strategy, and control the engineering machinery to execute a corresponding early warning action according to the target early warning strategy, for example, when the distance is 1 meter, the corresponding early warning strategy is parking, and when the distance is 2 meters, the corresponding early warning strategy is decelerating and gives out an alarm sound.

According to the method for pre-warning the pavement pits, the sizes of the pits are obtained, the types of the pits are determined to be the types of the non-passable pits according to the sizes and the preset size threshold, the distance between the pits and the vehicle is determined according to the point cloud data corresponding to the position information, and the corresponding target pre-warning strategy is determined according to the distance, so that the engineering machinery is controlled to execute the corresponding pre-warning action according to the target pre-warning strategy. According to the technical scheme, the judgment is made on the trafficability of the pits, different target early warning strategies are determined according to the types of the pits which cannot pass through and the distance between the vehicle and the pits, early warning operation is performed based on the target early warning strategies, the situation of the blind area of the road surface visual field in front of the vehicle running of the driver is reminded in real time, the driver is reminded of reminding of slowing down and avoiding in time, the situations of vehicle overturning, instability, imbalance and the like caused by the fact that the vehicle is driven into the pits are avoided, safety accidents are avoided, and comfortableness and safety in the vehicle running process are improved.

In one embodiment, determining that the type of pit is a non-passable pit type based on the size information and a preset size threshold includes: and determining that the type of the pit is a non-passable pit type under the condition that the depth, the length and the width of the pit are larger than the corresponding preset size threshold values.

Specifically, when the processor determines that the depth of the hole is greater than the preset depth threshold, the length of the hole is greater than the preset length threshold, and the width of the hole is greater than the preset width threshold, the processor may determine that the type of the hole is a non-passable hole type.

In one embodiment, determining a corresponding target early warning strategy according to the distance includes: determining a distance interval in which the distance is located; and determining an early warning strategy corresponding to the distance interval as a target early warning strategy.

Specifically, the processor may determine a distance interval in which the distance is located, and determine a corresponding pre-warning strategy, that is, a target pre-warning strategy, according to the determined distance interval based on a corresponding relationship between the predetermined distance interval and the pre-warning strategy.

In one embodiment, the distance interval includes a first distance interval, a second distance interval and a third distance interval, the first distance interval corresponds to an early warning strategy including sending a prompt message indicating that an impenetrable pit exists on a road surface in front, the second distance interval corresponds to an early warning strategy including sending a prompt message indicating that a vehicle approaches the impenetrable pit on the road surface in front and that a vehicle driving track needs to be changed, and the third distance interval corresponds to an early warning strategy including controlling the vehicle to slow down or stop.

It can be understood that the prompt information indicating that the front road surface has the non-passable pits can be realized by means of flashing an indicator lamp, audible alarm of a buzzer, text prompt and the like. The prompt information for changing the running track of the vehicle is usually distinguished from the prompt information for indicating that the front road surface has an impenetrable pit, and can be realized by means of flashing an indicator lamp, audible alarm of a buzzer, text prompt and the like.

Specifically, three distance intervals can be set from far to near according to the distance between the pit and the vehicle, namely a first distance interval, a second distance interval and a third distance interval, when the distance between the pit and the vehicle is located in the first distance interval, the processor can send out prompt information indicating that the front road surface is not passable, when the distance between the pit and the vehicle is located in the second distance interval, the processor can send out prompt information indicating that the vehicle is approaching the non-passable pit on the front road surface and the vehicle running track needs to be changed, and when the distance between the pit and the vehicle is located in the third distance interval, the processor can control the vehicle to be decelerated or stopped.

In order to effectively acquire the situation of a road surface pit in front of an overhead working vehicle in real time and improve the driving safety of the overhead working vehicle, a specific embodiment of the invention provides a method for detecting the road surface pit and early warning of the road surface pit, a multi-sensor fusion detection technology of a camera and a laser radar is adopted to extract comprehensive characteristics such as the width, the length, the pit depth of the road surface pit area, the nearest distance between the pit and the vehicle and the like, and comprehensive early warning strategies of different grades are made according to the width, the length, the pit depth of the road surface pit area and the nearest distance between the pit and the vehicle; and executing the early warning operation corresponding to the early warning level based on the early warning strategy.

The camera has the advantages of low cost, can provide rich color and texture information, is beneficial to the identification and classification of targets, but is difficult to acquire accurate three-dimensional information. The laser radar has the advantages that the detection distance is far, the laser radar can provide accurate distance and shape information of a target, and in addition, the laser radar is quite high in stability and good in robustness. By fusing the data of the camera and the laser radar, the advantages of the camera and the laser radar can be comprehensively utilized, more accurate and robust target identification is realized, reliable pit depth and distance information is obtained, and the robustness and reliability of the sensing system are improved.

Specifically, as shown in fig. 3, the method for detecting and early warning a road hole according to an embodiment of the present invention may include the following steps:

step one: and installing a 2D camera and a 3D laser radar on the aerial work vehicle and carrying out joint calibration and calibration.

Step two: and the 2D camera and the 3D laser radar respectively shoot a road surface in front of the vehicle to obtain RGB images and 3D point cloud data, and the RGB images and the 3D point cloud data are sent to the vehicle-mounted server for data calculation and processing.

Step three: preprocessing the RGB image and leading the RGB image into a pre-trained target detection neural network to detect pits.

Step four: and if no pit is detected, repeating the first step, and if the pit is detected, detecting and obtaining the position information of a two-dimensional boundary rectangular box (bounding box) of the pit in the RGB image according to the target detection neural network.

Step five: and associating the visually detected two-dimensional boundary rectangular frame of the pit and the pavement coordinates around the pit to 3D point cloud data under a radar coordinate system.

Step six: and acquiring positions of the pit point cloud and the road point cloud around the pit under the radar coordinate system, acquiring the distance between the pit and the vehicle, and obtaining the depth of the pit by comparing the position height difference of the pit and the surrounding flat road surface under the radar coordinate system.

Step seven: and calculating the minimum circumscribed rectangle of the pit according to the position information of the two-dimensional boundary rectangle frame of the pit, and obtaining the width and the length of the pit area of the pavement.

Step eight: and (5) making different grades of early warning strategies according to comprehensive characteristics of the width, the length, the pit depth and the nearest distance between the pit and the vehicle of the pavement pit area.

Step nine: and executing the early warning operation corresponding to the early warning level according to the early warning strategy in the step eight.

Specifically, after mapping a two-dimensional rectangular frame with detected pits and a pit peripheral area on an RGB image to 3D point cloud data of a laser radar, 3D points (x, y, z) of the laser radar corresponding to coordinates (u, v) on the RGB image are acquired.

In the first step, after the aerial working vehicle is provided with the fixed 2D camera and the laser radar, the pixels in the data of the calibration plate area acquired by the 2D camera and the laser radar are in one-to-one correspondence, so that the joint calibration and the calibration are completed. The joint calibration and calibration algorithm is as follows: assuming that RGB image data acquired by a 2D camera is represented by (u, v), and 3D points acquired by a lidar are represented by (x, y, z), the relationship between the 3D points (x, y, z) and the 2D points (u, v) is:

wherein the matrix (f _u ，f _v ，u ₀ ，v ₀ ) The internal reference of the 2D camera can be obtained according to a Zhang Zhengyou calibration method, R is a rotation matrix, T is a translation vector, and a specific value can be obtained by adopting an Autoware and other combined calibration and calibration tools.

In the second step, the laser radar is responsible for collecting 3D point cloud data, the 2D camera is an RGB camera and is responsible for collecting RGB image data, and the laser radar and the 2D camera respectively collect the 3D point cloud data and the RGB image data in real time and uniformly send the data to the vehicle-mounted server for real-time data calculation and processing. The sampling rates of the 2D camera and the 3D laser radar are different, the sampling rate of the camera is higher than that of the laser radar, the time stamp for collecting the 3D point cloud data and the RGB image data is recorded, the RGB image data with the closest time stamp is selected by taking the time stamp of the 3D point cloud data as a reference, and the synchronous alignment of the laser radar data and the camera data in time is realized. Further, the on-board server pre-processes the obtained RGB image including filtering to reduce noise and resizing to the input size required for the target detection neural network model.

In the third step, the RGB image is subjected to pit detection by adopting a pit detection neural network model, and a construction method of a lightweight pit target detection neural network model (figure 2) based on YOLOv5 is provided, which comprises the following steps:

a. the C3 module in the Backbone network (Backbone) of the Yolov5 is replaced by the lightweight convolutional neural network, the model parameters and the calculation cost are greatly reduced, and an ECA ((Efficient Channel Attention) attention mechanism) is added into the shuffleNetV2 to construct the shuffleNetV2-ECA Backbone network.

b. The path aggregation network PANet (Path Aggregation Network) layer in the neck network neg in YOLOv5 is changed into a bidirectional feature pyramid network BiFPN.

c. And replacing the echo loss function EIoU of the Yolov5 back-end network lifting Box prediction frame with the LEIOU loss function.

Specifically, a C3 module in the original backbone network of the YOLOv5 is replaced by a SheffeNet V2 module, wherein the SheffeNet V2 module is shown in FIG. 4 and comprises a Sheffev 2-Block-a unit and a Sheffev 2-Block-b unit; wherein:

in the Shufflev2-Block-a unit, the input characteristic diagram of the number of C channels is divided into two paths to respectively carry out convolution operation, after the convolution operation is finished, concat operation is used for fusion, after channel shuffling, the characteristic diagram of the number of 2C channels is output, and the network width is expanded on the premise of increasing fewer parameters.

In the Shufflev2-Block-b unit, an input characteristic diagram of the number of C channels is divided into two parts, namely, the characteristic diagrams of the number of C1 channels and C2 channels are not changed, and the characteristic diagram of the number of C2 channels passes through three convolution layers with the same input and output channels, wherein the convolution layers comprise two common convolutions Conv2D of 1X1 and depth separable convolutions DWConv of 1X 3; after the two branches are finished, concat operation is carried out to connect channels in series, an initial C channel characteristic diagram is fused, and information interaction is carried out by channel shuffling.

Further, adding an ECA attention mechanism to the ShuffleNetV2 module creates a new feature extraction network ShuffleECA-Net, comprising: the ECA attention mechanism is respectively combined with a Shufflev2-Block-a unit and a Shufflev2-Block-b unit to be improved, so that a shuffleECAMmodule_1 module and a shuffleECAMmodule_2 module are formed as shown in the following figure 5; the SheffeECAModule_1 module and the SheffeECAModule_2 module are utilized to construct the SheffeECA-Net backbone network, so that the model is light, the number of network parameters and the number of floating point number operation times are reduced, the detection speed is improved, and the target detection neural network can be transplanted to the terminal equipment on the basis of ensuring the detection accuracy of an algorithm.

The overall structure of the ECA (Efficient Channel Attention) attention mechanism is shown in fig. 6 below, and ECA attention can be well adjusted while avoiding frequent dimension reduction operations, compared with SE attention adopted by original YOLOv 5. The ECA attention mechanism acquires the interaction information of each channel and its k neighbors after global averaging pooling with a fast 1D convolution of convolution kernel k, where the convolution kernel k is adaptively acquired in direct proportion to the channel dimension, further reducing the parameter complexity by sharing the weights of all channels. In this calculation, w containing k×c parameters is used _k Array to learn attention, channel y _i Weight omega of (2) _i Only y is considered in the calculation process _i Interaction information between k neighbors is calculated as follows:

ω＝σ(C1D _k (y))

Further, using a bi-directional feature pyramid network BiFPN instead of the PANet of original YOLOv5, PANet is essentially just adding different features, which would result in unequal weights of different sized features of the same type in the fused output features. Because the contribution weights of different input features to the output feature map should be different at each node in the feature fusion process, in the road pit detection, the shape of the pit is long and discontinuous, and the network is required to have stronger feature extraction capability. The BiFPN performs deep feature fusion from top to bottom and shallow feature fusion from bottom to top through bidirectional connection and weighted feature fusion, skips over certain intermediate layers, connects and fuses feature layers with different scales, and introduces learning weights to learn the importance of different input features, so that the feature extraction capability of a network is enhanced. The BiFPN gives different weight information to each feature map, a weighting mode of rapid normalization fusion is provided, the non-negative weight of each weight is ensured through a ReLU activation function, and a small epsilon (epsilon=0.0001) is added, so that the stability of data is ensured, and the following formula is shown:

finally, in the model training stage, optimizing the loss function and the prediction frame screening method. The YOLOv5 model mainly uses the CIOU loss, which considers the distance between the center point of the prediction frame and the center line of the real bounding frame and the aspect ratio of the prediction frame to the real bounding frame, and it is difficult to accurately predict the real bounding frame through the CIOU loss due to the variety of types of pavement pits. In the method, an EIoU loss function is adopted as a prediction frame regression loss function CIOU for improving the YOLOv5 algorithm, the EIOU loss refers to a calculation method of overlapping loss and center distance loss in the CIOU, but the width and height losses use the minimum value of the difference between the width and the height of the prediction frame and the real boundary frame, so that the model converges faster, and higher precision is obtained.

Wherein IoU is the intersection ratio of the prediction boundary box and the real box, ρ ² (b,b ^gt ) Is the distance between the center point of the prediction box and the center point of the ground truth bounding box. C is the diagonal length of the smallest bounding box covering the prediction box and the ground truth bounding box, C _w And C _h Is the width and height of the smallest bounding box that covers the prediction box and the ground truth bounding box. However, the training data set for a road hole has the problem of data sample imbalance, which will result in a far smaller number of high quality samples in the image with smaller regression errors than lower quality samples with larger errors. Samples of poor quality can create large gradients and impact the training process. Thus, focal Loss was introduced to solve the problem of sample imbalance in regression, using the focus Loss L _Focal-EIOU To increase the precision loss, L _Focal-EIOU ＝IOUγL _EIOU Where γ is a focus parameter indicating the degree of suppression of an outlier.

In the third step, a hole detection model is used for detecting holes in the RGB image, and the hole detection model is an improved YOLOv5 model, as shown in fig. 7, and includes a main process of inputting the RGB image into the improved YOLOv5 model for training and detection, wherein the main process includes the following steps:

1) The RGB image is input into a backbone network, and the backbone network performs the following processing:

the RGB images sequentially pass through a Focus slicing operation, a CBL operation, a CSP1_1 operation, a CBL operation, a CSP1_3 operation, a SheffeECAModule_2 operation and a SheffeECAModule_1 operation, and a first-layer feature map of the input image is output; the first layer characteristic diagram of the input image is sequentially operated by the SheffeECAModule_2 and the SheffeECAModule_1, and then a second layer characteristic diagram of the input image is output; the second layer characteristic diagram of the input image is sequentially operated by the SheffeECAModule_2 and the SheffeECAModule_1, and the third layer characteristic diagram of the input image is output;

2) In the above-mentioned Neck network (negk):

the Neck module uses the third layer characteristic diagram of the input image to output a fourth layer characteristic diagram of the input image after being spliced by the Upsamples up-sampling operation and the second layer characteristic diagram Concat and then being sequentially subjected to C3_3 and CBL operations; the fourth layer of feature images are subjected to Concat splicing through the up-sampling of the Upsample and the first layer of feature images of the input image, and a fifth layer of feature images are obtained after C3-3 operation;

performing Concat splicing operation on the fifth layer feature map after CBL operation and the fourth layer feature map, and obtaining a sixth layer feature map of the input image after C3_3 operation;

After CBL operation is carried out on the sixth layer of feature images and Concat splicing is carried out on the third layer of feature images, a seventh layer of feature images of the input images are obtained after C3_3 operation is carried out on the sixth layer of feature images;

and performing detection processing on the fifth layer characteristic diagram, the sixth layer characteristic diagram and the seventh layer characteristic diagram after Conv2d convolution operation respectively to obtain an eighth layer characteristic diagram of the image, wherein detection of a target in the input image and framing of the target are finished.

3) In the Head network (Head) described above:

performing loss function calculation of a target Boundingbox on the eighth layer of feature map, and adopting L _Focal-EIOU Performing NMS non-maximum value inhibition operation in a mode, outputting feature graphs with the size of 3 different sizes, wherein the depth of the feature graphs is 3 (N+1+4), (N is the number of types in a data set, 1 represents a confidence value, 4 represents the framed center point coordinate and the width and height of the center point coordinate), and the three feature layers with different scales output by an output end give a regression boundary box and confidence; filtering out repeated boundary frames higher than a set threshold by adopting a non-maximum value inhibition method to obtain a predicted frame, comparing the predicted frame with a marked frame, and adopting L _Focal-EIOU And calculating the loss of the boundary box, performing back propagation according to the loss function, and adjusting the weight of the improved YOLOv5 model in the training process.

The embodiment of the invention also provides a construction method of the lightweight pit target detection neural network based on the YOLOv5, which comprises the following training steps:

and S1, acquiring video data sets of pavement pits in different scenes and different time periods through cameras fixed on the aerial working vehicle.

S2, converting the video into a frame picture, and performing data enhancement pretreatment by using a Mosaic; the enhancement data set is divided into a training set and a testing set by a certain proportion, after the data set is divided, the pictures are manually marked by Labelimg software, positions of pits are marked by rectangular frames, and an XML format file is generated.

And S3, constructing the lightweight hole target detection neural network model based on the YOLOv5 based on the training set and the testing set, wherein the hole target detection neural network can perform feature extraction on feature information on an image through a convolution network.

And S4, inputting the image data to be detected into the optimized YOLOv5 pit target detection model to obtain a detection result.

Specifically, solving for the depth of the pit may include: the RGB image is subjected to pit detection by adopting the pit detection neural network model, if the existence of a pit on the road surface is detected, the position information of a two-dimensional boundary rectangular frame (bounding box) of the pit in the RGB image is represented as (u 1, v1, u2, v 2), four coordinates of the rectangular frame are represented as A, B, C and D, as shown in figure 8, wherein (u 1, v 1) represents the position of a predicted upper left corner A of the rectangular frame in an RGB image coordinate system, and (u 2, v 2) represents the position of a predicted lower right corner D of the rectangular frame in the RGB image coordinate system, and the center point E coordinate of the pit rectangular frame is further calculated Correlating n (n can take values of 3, 5 and 7) of the central point coordinates (u_mid, v_mid) of the rectangular frame of the pit detected by the RGB image with projection coordinates (x_lidar, y_lidar, z_lidar) under a laser radar coordinate system in the neighborhood, and obtaining a z_lidar average value as the height of the pit under the laser radar coordinate system; enlarging the rectangular frame of the pit by 1.1 times to obtain a rectangular frame enveloping the pit and the pavement around the pit and four coordinates A ', B', C ', D' of the rectangular frame, which can be considered as the periphery of the pitThe road surface on the side is flat, the heights of the four coordinates of A ', B', C ', D' under the laser radar coordinate system are obtained by the same process, and the average value is obtained to be used as the height of the road surface on the periphery of the pit hole under the laser radar coordinate system; further obtaining the height between the pit and the pavement around the pit, namely the depth H of the pit.

Obtaining the nearest distance between the pit and the vehicle, and obtaining the boundary coordinates (u) of the rectangular frame of the pit, detected by the RGB image, approaching the vehicle ₁ ＜u＜u ₂ ，v＝v ₂ ) N (n is 3) in the neighborhood of the projected coordinates (x_lidar, y_lidar, z_lidar) in the lidar coordinate system, and the average value of y_lidar is obtained as the distance DIS between the pit and the vehicle.

Calculating the minimum circumscribed rectangle of the pit: and according to the relation of the pit two-dimensional boundary rectangular frame to the 3D point cloud data set under the radar coordinate system, projecting the coordinates under the laser radar coordinate system onto the ground plane. Assuming that the ground plane is an X-Y plane, z_lidar may be set to 0, resulting in projection coordinates (x_proj, y_proj, 0) of the rectangular box of the pit under the lidar coordinate system. The projection coordinates (x_proj, y_proj, 0) are regarded as a point cloud data set. The minimum bounding rectangle is calculated using RANSAC, the detailed steps are described below:

Initializing parameters of RANSAC: setting iteration times (iterations) and sample number (sample_size), and defining a distance threshold (distance_threshold) for judging whether the point belongs to the minimum circumscribed rectangle.

The iterative process:

a. the sample size points are randomly selected as a sample set for iteration.

b. The centroid coordinates (x_center, y_center, z_center) of the sample set are calculated.

c. For each point, a polar representation of it relative to the centroid is calculated:

calculating the distance from the point to the centroid: ρ_i=sqrt ((x_i-x_center)/(2+ (y_i-y_center)/(2))

Calculating the angle of the point: θ_i=atan2 (y_i-y_center, x_i-x_center)

d. Under polar representation, rotation angle estimation was performed using RANSAC:

two points were randomly selected and their angular difference was calculated: θ_diff=θ_2- θ_1

Repeating the process for several times, selecting two points with the largest angle difference as optimal samples

Calculating an angle average value of the optimal sample as a rotation angle: θ_rot= (θ_1+θ_2)/2

e. According to the estimated rotation angle, the polar coordinates of each point are subjected to rotation transformation:

angle after rotation: θ_i' =θ_i- θ_rot

f. In the rotated polar coordinate set, selecting a point belonging to the smallest circumscribed rectangle according to a distance threshold value:

Find the minimum distance in the polar coordinate set: ρ_min '=min (ρ_i')

Find the maximum distance in the polar coordinate set: ρ_max '=max (ρ_i')

g. Converting the polar coordinates (ρ_min ', ρ_max') of the minimum bounding rectangle back to the Cartesian coordinate system to obtain the corner coordinates (x_min ', y_min') and (x_max ', y_max') of the rectangular frame:

x_min'＝ρ_min'*cos(θ_rot)

y_min'＝ρ_min'*sin(θ_rot)

x_max'＝ρ_max'*cos(θ_rot)

y_max'＝ρ_max'*sin(θ_rot)

h. repeating the iterative process for a plurality of times, and selecting a result with the minimum circumscribed rectangle area as a final pit circumscribed rectangle. Finally, corner coordinates (x_min, y_min) and (x_max, y_max) of the pit circumscribing rectangle under the laser radar coordinate system and a rotation angle theta_rot are obtained.

According to the minimum circumscribed rectangle of the pit, the width W=x_max-x_min of the pavement pit and the length L=y_max-y_min of the pavement pit are obtained.

And comparing the depth, width and length of the comprehensive pits with set thresholds to judge the trafficability of the pits: when H > H_limit, L > L_limit or W > W_limit is judged as a non-passable pit type, other conditions are regarded as passable pit types, and an early warning strategy is determined for the non-passable pit types; the early warning strategy is divided into different early warning levels according to the distance between the vehicle and the pit, and the driver is prompted about the urgency degree; the early warning level comprises primary early warning, secondary early warning and tertiary early warning, if the situation that an impenetrable pit exists in front of a road surface is detected, the primary early warning is set if the distance from the front of the overhead working truck is 1.5-2.5 meters; if the distance between the pit and the front of the overhead working truck is between 0.5 and 1.5 meters, the two-stage early warning is set, and if the distance between the pit and the front of the overhead working truck is less than 0.5 meter, the three-stage early warning is set.

And executing the early warning operation corresponding to the early warning level based on the early warning strategy, wherein each level of early warning corresponds to different early warning operation. If the primary early warning is only indicated by the flashing of the indicator lamp, the vehicle is used for prompting the existence of an impenetrable pit on the road surface in front of the driver, and when the secondary early warning is achieved, the corresponding indicator lamp flashes and the buzzer alarms, and is used for prompting the driver that the vehicle approaches the road surface in front and has the impenetrable pit, and the running track of the vehicle needs to be changed in time; the early warning operation of the three-stage early warning comprises the step of actively intervening to control the vehicle to slow down or even stop besides the flashing of the indicator lamp and the alarm of the buzzer.

According to the technical scheme provided by the embodiment of the application, the multi-sensor fusion detection technology of the camera and the laser radar is adopted to extract comprehensive characteristics such as the width, the length, the pit depth and the nearest distance between the pit and the vehicle of the pit area of the road surface, and the comprehensive early warning strategies of different grades are made according to the width, the length, the pit depth and the nearest distance between the pit and the vehicle of the pit area of the road surface; and executing the early warning operation corresponding to the early warning level based on the early warning strategy. Specifically, the improved YOLOv5 convolutional neural network is adopted, so that the calculated amount of network parameters is reduced, and the real-time performance of detection is improved. In addition, the technical scheme can detect the pits on the road surface of the running area in front of the overhead working vehicle in real time, judge the trafficability of the pits, make different grades of early warning strategies according to the distances between vehicles and the pits for the types of the non-trafficable pits, execute early warning operation based on the early warning strategies, remind the driver of the situation of the visual field blind area of the road surface in front of the vehicle running in real time, remind the driver of slowing down and avoiding in time, avoid the situations of vehicle overturn, instability, imbalance and the like caused by the condition that the vehicle runs into the pits, avoid the occurrence of safety accidents, and improve the comfort and the safety in the running process of the overhead working vehicle.

An embodiment of the present invention provides a processor configured to perform the method for detecting a road hole according to the above-described embodiment.

The embodiment of the invention provides a processor configured to execute the method for pre-warning the pits on the pavement according to the embodiment.

As shown in fig. 9, an embodiment of the present invention provides an apparatus 900 for detecting a pit on a road surface, which is applied to a vehicle, and the vehicle is provided with an image acquisition apparatus and a point cloud data acquisition apparatus, and the apparatus includes:

the data acquisition module 910 is configured to acquire color images and point cloud data corresponding to the road surface to be detected, which are acquired by the image acquisition device and the point cloud data acquisition device simultaneously and respectively.

The pit detection module 920 is configured to detect the color image through a pre-trained pit detection model, so as to obtain position information of an external rectangular area corresponding to the pit; the pit detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network.

The size determining module 930 is configured to determine a depth of the pit according to the point cloud data corresponding to the location information and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area; and determining the length and the width of the pit according to the point cloud data corresponding to the position information.

According to the device for detecting the pavement pits, the image acquisition device and the point cloud data acquisition device are used for acquiring the color image and the point cloud data corresponding to the pavement to be detected, the color image is detected through the pre-trained improved YOLOv5 model, the C3 feature extraction module in the backbone network of the improved YOLOv5 model is replaced by the lightweight convolutional neural network, so that the position information of the circumscribed rectangular area corresponding to the pits is obtained, the depth of the pits is determined according to the point cloud data corresponding to the position information of the circumscribed rectangular area and the point cloud data corresponding to the peripheral area of the circumscribed rectangular area, the length and the width of the pits are determined according to the position information of the circumscribed rectangular area, the size information such as the depth of the pits can be accurately detected, the light pits of the pavement are detected through the improved YOLOv5 model, the model weight is reduced, the number of network parameters and the number of floating point calculation times can be reduced, the calculation amount of model parameters and the calculation cost can be reduced, and the detection speed and the real-time performance of detection can be improved.

In one embodiment, the lightweight convolutional neural network includes an ECA attention mechanism.

In one embodiment, the pothole detection module 920 is further configured to: detecting the color image through a pre-trained pit detection model to obtain angular point position information of an external rectangular area corresponding to the pit; and determining the central position information of the circumscribed rectangular area according to the angular point position information.

In one embodiment, the pothole detection module 920 is further configured to: acquiring a first height value in the point cloud data corresponding to the central position information and a second height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area; and determining the absolute value of the difference between the first height value and the second height value to obtain the depth of the pit.

As shown in fig. 10, an embodiment of the present invention provides a device 1000 for pre-warning a pit on a road, which is applied to a vehicle and includes:

a size obtaining module 1010, configured to obtain size information of the pit, where the size information includes depth, length, and width, and the size information is determined by the method for detecting the pit on the road surface according to the above embodiment.

The type determining module 1020 is configured to determine, according to the size information and the preset size threshold, that the type of the pit is a type of the non-passable pit.

The distance determining module 1030 is configured to determine a distance between the pit and the vehicle according to the point cloud data corresponding to the location information.

The policy determining module 1040 is configured to determine a corresponding target early warning policy according to the distance.

The policy execution module 1050 is configured to control the engineering machine to execute a corresponding early warning action according to the target early warning policy.

According to the device for pre-warning the pavement pits, the type of the pits is determined to be the type of the non-passable pits according to the size information and the preset size threshold value, the distance between the pits and the vehicle is determined according to the point cloud data corresponding to the position information, and the corresponding target pre-warning strategy is determined according to the distance, so that the engineering machinery is controlled to execute the corresponding pre-warning action according to the target pre-warning strategy. According to the technical scheme, the judgment is made on the trafficability of the pits, different target early warning strategies are determined according to the types of the pits which cannot pass through and the distance between the vehicle and the pits, early warning operation is performed based on the target early warning strategies, the situation of the blind area of the road surface visual field in front of the vehicle running of the driver is reminded in real time, the driver is reminded of reminding of slowing down and avoiding in time, the situations of vehicle overturning, instability, imbalance and the like caused by the fact that the vehicle is driven into the pits are avoided, safety accidents are avoided, and comfortableness and safety in the vehicle running process are improved.

In one embodiment, the type determination module 1020 is further configured to: and determining that the type of the pit is a non-passable pit type under the condition that the depth, the length and the width of the pit are larger than the corresponding preset size threshold values.

In one embodiment, the policy determination module 1040 is further to: determining a distance interval in which the distance is located; and determining an early warning strategy corresponding to the distance interval as a target early warning strategy.

An embodiment of the present invention provides a vehicle including: an image acquisition device; the point cloud data acquisition device; and the device for detecting the pavement pits according to the above embodiment.

An embodiment of the present invention provides a vehicle including: an image acquisition device; the point cloud data acquisition device; and the device for pre-warning the pits on the pavement according to the embodiment.

Understandably, the vehicle may include, but is not limited to, an engineering vehicle such as an overhead working vehicle, the image acquisition device may include, but is not limited to, a 2D camera, and the point cloud data acquisition device may include, but is not limited to, a lidar. Further, the vehicle may further include an indicator light and a buzzer, and an in-vehicle server. In addition, the vehicle may include a power source, an overhead working vehicle control unit, and a road hole detection and warning system of the overhead working vehicle, the power source may be a 12V/24V dc low voltage power source, and the power source may supply power to the road hole detection and warning system of the overhead working vehicle to enable the system to perform the above-described method for detecting a road hole and/or method for warning a road hole.

In other embodiments, the calculation unit of the road pit detection and early warning system of the aerial work vehicle is connected with the controller of the aerial work vehicle through CAN communication, and CAN send a deceleration or stop signal to the controller of the aerial work vehicle to assist in controlling the vehicle to decelerate or brake.

The laser radar module is used for acquiring point cloud data of a road surface in front of the vehicle, and the 2D camera is used for acquiring RGB image data of the road surface in front of the vehicle. The pilot lamp and the buzzer are used for executing early warning operation, and the pilot lamp can flash with different frequencies and the buzzer can send out the alarm sound of different frequencies to this warning level of suggestion driver. The laser radar module, the camera module, the indicator light and the buzzer are connected with a vehicle-mounted server, and the vehicle-mounted server comprises: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of road hole detection and warning for an overhead working vehicle.

The laser radar is used for collecting point cloud data of a road surface in front of the vehicle, the laser radar is directly powered by the vehicle, the collected point cloud data is sent to the vehicle-mounted server for further processing, the laser radar is installed in the running direction of the overhead working vehicle, the distance from the laser radar to the ground is about 30cm, and the installation angle is adjusted so that the laser radar can collect the point cloud data of a road surface area of 0.3 m-5 m in the running direction of the vehicle.

The 2D camera is used for collecting RGB image data of a road surface in front of the vehicle, the camera is directly powered by the vehicle, the collected RGB image data is sent to the vehicle-mounted server for further processing, the camera is arranged in the running direction of the overhead working vehicle, the height from the ground is about 50cm, and the installation angle is adjusted so that the camera can collect image data of a road surface area of 0.3 m-5 m in the running direction of the vehicle;

the pilot lamp and the buzzer can receive control signals of the vehicle-mounted server, and when the aerial work vehicle is in early warning, the pilot lamp is executed to flash and sound early warning according to an early warning strategy, so that a driver is timely reminded of slowing down and avoiding, and the travelling comfort and safety of the aerial work vehicle are improved. The indicator light and the buzzer can be arranged in a working column of the overhead working vehicle, and the driver can conveniently see or hear the position. The driver can know in time that the vehicle is going ahead road surface and has the hole condition through the suggestion of pilot lamp and bee calling organ to and the distance between vehicle and the hole, so as to avoid leading to unnecessary incident because the visual field blind area causes the overhead working truck to drive into the hole.

Embodiments of the present application provide a machine-readable storage medium on which a program or instructions is stored which, when executed by a processor, implement a method for detecting a road surface pit according to the above-described embodiments.

The embodiment of the application provides a machine-readable storage medium, on which a program or instructions are stored, which when executed by a processor, implement the method for pre-warning a road hole according to the above embodiment.

It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, etc., such as Read Only Memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises an element.

The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and variations of the present application will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. which come within the spirit and principles of the application are to be included in the scope of the claims of the present application.

Claims

1. A method for detecting a road surface pit, applied to a vehicle, characterized in that the vehicle is provided with an image acquisition device and a point cloud data acquisition device, the method comprising:

acquiring color images and point cloud data corresponding to the road surface to be detected, which are acquired by the image acquisition device and the point cloud data acquisition device respectively;

detecting the color image through a pre-trained hole detection model to obtain position information of an external rectangular area corresponding to a hole, wherein the hole detection model is an improved YOLOv5 model, and a C3 feature extraction module in a backbone network of the improved YOLOv5 model is replaced by a lightweight convolutional neural network;

2. The method of claim 1, wherein the lightweight convolutional neural network comprises an ECA attention mechanism.

3. The method of claim 1, wherein the path aggregation network in the neck network of the modified YOLOv5 model is replaced with a bi-directional feature pyramid network.

4. The method according to claim 1, wherein the position information comprises corner position information and center position information; the method for detecting the color image through the pre-trained pit detection model to obtain the position information of the circumscribed rectangular area corresponding to the pit comprises the following steps:

detecting the color image through a pre-trained pit detection model to obtain angular point position information of an external rectangular area corresponding to the pit;

and determining the central position information of the circumscribed rectangular area according to the angular point position information.

5. The method of claim 4, wherein determining the depth of the pit from the point cloud data corresponding to the location information and the point cloud data corresponding to the peripheral region of the circumscribed rectangular region comprises:

Acquiring a first height value in the point cloud data corresponding to the central position information and a second height value in the point cloud data corresponding to the peripheral area of the circumscribed rectangular area;

and determining the absolute value of the difference value between the first height value and the second height value to obtain the depth of the pit.

6. A method for pre-warning a pit on a road surface, which is applied to a vehicle, and is characterized by comprising the following steps:

acquiring size information of a pit, wherein the size information comprises depth, length and width, and the size information is determined by the method for detecting a pavement pit according to any one of claims 1 to 5;

7. The method of claim 6, wherein the determining that the type of pit is a non-passable pit type based on the size information and a preset size threshold comprises:

And determining that the type of the pit is a non-passable pit type under the condition that the depth, the length and the width of the pit are larger than the corresponding preset size threshold.

8. The method of claim 6, wherein the determining a corresponding target pre-warning strategy based on the distance comprises:

determining a distance interval in which the distance is located;

and determining the pre-warning strategy corresponding to the distance interval as the target pre-warning strategy.

9. The method of claim 8, wherein the distance intervals comprise a first distance interval, a second distance interval and a third distance interval, wherein the first distance interval corresponds to an early warning strategy comprising sending a prompt message indicating that an impenetrable pit exists on a road surface ahead, the second distance interval corresponds to an early warning strategy comprising sending a prompt message indicating that a vehicle approaches an impenetrable pit on the road surface ahead and that a vehicle driving track needs to be changed, and the third distance interval corresponds to an early warning strategy comprising controlling the vehicle to slow down or stop.

10. A processor, characterized by being configured to perform the method for detecting a road hole according to any one of claims 1 to 5 or the method for road hole warning according to any one of claims 6 to 9.

11. A device for detecting a road surface pit applied to a vehicle, characterized in that the vehicle is provided with an image acquisition device and a point cloud data acquisition device, comprising:

12. A device for pre-warning a road hole, applied to a vehicle, comprising:

a size acquisition module for acquiring size information of the pit, wherein the size information includes depth, length, and width, the size information being determined by the method for detecting a road surface pit according to any one of claims 1 to 5;

13. A vehicle, characterized by comprising:

an image acquisition device;

the point cloud data acquisition device; and

a device for detecting a road hole according to claim 11 or a device for road hole warning according to claim 12.

14. A machine-readable storage medium having stored thereon a program or instructions, which when executed by a processor, implements a method for detecting a road hole according to any one of claims 1 to 5 or a method for road hole warning according to any one of claims 6 to 9.