CN115892068A - Vehicle control method, device, equipment, medium and vehicle - Google Patents

Abstract

The present disclosure relates to a vehicle control method, device, device, medium, and vehicle, capable of performing label matching on real-time point cloud data corresponding to the vehicle driving area based on the semantic segmentation result of the real-time image data corresponding to the vehicle driving area, and determining the real-time point cloud The target point cloud points in the data, and remove the target point cloud points from the real-time point cloud data to obtain the target point cloud data, and then control the vehicle based on the target point cloud data. Therefore, the vehicle is controlled based on the point cloud data. Before driving control, point cloud points related to obstacles that do not affect driving can be eliminated, and then when the vehicle is controlled based on point cloud data, obstacles that do not affect driving on the lanes in the port can be avoided. Interference, so as to avoid the self-driving work vehicles in the port from stopping due to obstacles that do not affect the driving, and improve the operating efficiency of the self-driving work vehicles.

Description

Vehicle control method, device, equipment, medium and vehicle

technical field

The present disclosure relates to the technical field of automatic driving, and in particular to a vehicle control method, device, equipment, medium and vehicle.

Background technique

In the field of autonomous driving, it is generally possible to identify pedestrians, roads, cars, obstacles and other objects in the driving of the autonomous vehicle through point cloud data, so that the autonomous vehicle can drive safely on the road.

However, for the port, the lanes in the port are often invaded by obstacles such as weeds and birds that do not affect driving, and when autonomous driving vehicles perform object recognition on point cloud data based on related technologies, they will Obstacles that affect driving are uniformly identified as obstacles, so that the self-driving work vehicle stops driving because of such obstacles that do not affect driving, reducing the operating efficiency of the self-driving work vehicle.

Contents of the invention

In order to solve the above technical problems, the present disclosure provides a vehicle control method, device, equipment, medium and vehicle.

In a first aspect, the present disclosure provides a vehicle control method, including:

Obtain real-time image data and real-time point cloud data corresponding to the vehicle driving area;

Perform semantic segmentation on the real-time image data to obtain the semantic segmentation result, which includes the pixel semantic label of each pixel in the real-time image data;

Match the real-time point cloud data with the real-time image data to determine the target point cloud point in the real-time point cloud data. The point cloud semantic label of the target point cloud point is the target object, and the target object is an obstacle that does not affect driving;

Eliminate the target point cloud points in the real-time point cloud data to obtain the target point cloud data;

Based on the target point cloud data, the vehicle is controlled.

In a second aspect, the present disclosure provides a vehicle control device, including:

The data acquisition module is used to acquire real-time image data and real-time point cloud data corresponding to the vehicle driving area;

The semantic segmentation module is used to perform semantic segmentation on the real-time image data to obtain the semantic segmentation result, the semantic segmentation result includes the pixel semantic label of each pixel in the real-time image data;

The tag matching module is used to tag match the real-time point cloud data with the real-time image data, and determine the target point cloud point in the real-time point cloud data, the point cloud semantic label of the target point cloud point is the target object, and the target object is the obstacles;

The data elimination module is used to eliminate the target point cloud points in the real-time point cloud data to obtain the target point cloud data;

The driving control module is used to control the driving of the vehicle based on the target point cloud data.

In a third aspect, the present disclosure provides a vehicle control device, including:

processor;

memory for storing executable instructions;

Wherein, the processor is used for reading executable instructions from the memory, and executing the executable instructions to implement the vehicle control method of the first aspect above.

In a fourth aspect, the present disclosure provides a computer-readable storage medium, where a computer program is stored in the storage medium, and when the computer program is executed by a processor, the processor is made to implement the vehicle control method in the first aspect above.

In a fifth aspect, the present disclosure provides a vehicle, including at least one of the following:

The vehicle control device of the second aspect above;

The vehicle control device of the third aspect above;

The computer storage medium of the fourth aspect above.

Compared with the prior art, the technical solutions provided by the embodiments of the present disclosure have the following advantages:

The vehicle control method, device, equipment, medium, and vehicle of the embodiments of the present disclosure can perform label matching on the real-time point cloud data corresponding to the vehicle driving area based on the semantic segmentation result of the real-time image data corresponding to the vehicle driving area, and determine the real-time point cloud The target point cloud points in the data, and remove the target point cloud points from the real-time point cloud data to obtain the target point cloud data, and then control the vehicle based on the target point cloud data, because the point cloud semantic tags of the target point cloud points is the target object and the target object is an obstacle that does not affect driving, therefore, before controlling the vehicle based on point cloud data, the point cloud points related to obstacles that do not affect driving can be eliminated first, and then based on the point cloud When the data is used to control the driving of the vehicle, it can avoid the interference of obstacles on the lanes in the port that do not affect the driving, thereby preventing the automatic driving operation vehicles in the port from stopping due to obstacles that do not affect the driving, and improving the automatic driving. The operating efficiency of the work vehicle.

Description of drawings

The above and other features, advantages and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

FIG. 1 is a schematic flowchart of a vehicle control method provided by an embodiment of the present disclosure;

FIG. 2 is a schematic flowchart of another vehicle control method provided by an embodiment of the present disclosure;

FIG. 3 is a schematic flowchart of another vehicle control method provided by an embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of a vehicle control device provided by an embodiment of the present disclosure;

Fig. 5 is a schematic structural diagram of a vehicle control device provided by an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein; A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method implementations of the present disclosure may be executed in different orders, and/or executed in parallel. Additionally, method embodiments may include additional steps and/or omit performing illustrated steps. The scope of the present disclosure is not limited in this respect.

As used herein, the term "comprise" and its variations are open-ended, ie "including but not limited to". The term "based on" is "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one further embodiment"; the term "some embodiments" means "at least some embodiments." Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the sequence of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "one" and "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more" multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are used for illustrative purposes only, and are not used to limit the scope of these messages or information.

For ports, the lanes in the port are often invaded by obstacles such as weeds and birds that do not affect driving, and when autonomous driving vehicles perform object recognition on point cloud data based on related technologies, due to the fact that only based on point cloud data Semantic information is difficult to carry out accurate object recognition. Such obstacles that do not affect driving are often identified as obstacles, and then the autonomous driving vehicle stops driving because of such obstacles that do not affect driving, reducing the number of autonomous driving operations. The operating efficiency of the vehicle.

Take the driveway invaded by weeds as an example. Since the use of herbicides will affect the food safety in the container, considering the safety of the containers in the port, weeds cannot be completely wiped out with herbicides. Manual weeding is not only inefficient but also often recurs. Therefore, it is impossible to prevent the automatic driving operation vehicle from stopping driving by eliminating the intrusion of obstacles in the lane that do not affect driving.

In order to solve the above problems, embodiments of the present disclosure provide a vehicle control method, device, equipment, medium, and vehicle. The vehicle control method provided by the embodiment of the present disclosure will be firstly described below.

Fig. 1 shows a schematic flowchart of a vehicle control method provided by an embodiment of the present disclosure. The vehicle control method shown in FIG. 1 can be executed by a vehicle control device. Wherein, the vehicle control device may be an on-board device mounted on the vehicle or a controller of the vehicle, and the vehicle may be an automatic driving work vehicle.

As shown in Fig. 1, the vehicle control method may include the following steps.

S110. Acquire real-time image data and real-time point cloud data corresponding to the driving area of the vehicle.

In the embodiment of the present disclosure, when the vehicle needs to drive during the operation, the vehicle control device on the vehicle can obtain the real-time image data and real-time point cloud data corresponding to the driving area of the vehicle, to The vehicle performs driving control.

Wherein, the vehicle driving area may be determined according to the pre-planned driving route, and the road area to be driven in front of the driving direction of the vehicle.

Specifically, the image acquisition device installed on the vehicle for collecting the driving environment in front of the vehicle's driving direction can collect real-time image data of the driving environment including the driving area of the vehicle in real time, and the image acquisition device installed on the vehicle for collecting the driving environment in front of the vehicle's driving direction The lidar device of the driving environment can collect real-time point cloud data of the driving environment including the driving area of the vehicle in real time.

Therefore, the vehicle control device can acquire the real-time image data collected by the image collection device and the real-time point cloud data collected by the lidar device.

It should be noted that the real-time image data acquired by the vehicle control device and the real-time point cloud data need to be collected at the same time, so as to ensure the reliability of vehicle driving control based on the real-time image data and real-time point cloud data.

S120. Perform semantic segmentation on the real-time image data to obtain a semantic segmentation result, where the semantic segmentation result includes a pixel semantic label of each pixel in the real-time image data.

In the embodiment of the present disclosure, after acquiring the real-time image data and real-time point cloud data, the vehicle control device may perform semantic segmentation on the real-time image data to determine the semantic information of each pixel in the real-time image data and obtain a semantic segmentation result.

Among them, the pixel semantic label may include various objects that may appear in the driving environment of the vehicle, such as weeds, birds, cones, pedestrians, backgrounds other than roads, and so on.

Further, these objects can be divided into target objects corresponding to obstacles that do not affect driving, non-target objects corresponding to obstacles that affect driving, and irrelevant objects that have nothing to do with obstacles that have nothing to do with driving, according to their influence on driving. category. For example, target objects may include objects such as weeds and birds, and these objects will not actually affect the normal driving of the vehicle during the actual driving process of the vehicle, that is, the possibility of the vehicle colliding with them is relatively small; non-target objects may include Objects such as cones and pedestrians, these objects may actually affect the normal driving of the vehicle during the actual driving process, that is, the vehicle is more likely to collide with it; irrelevant objects can include backgrounds other than roads such as buildings and sky These objects have nothing to do with the road during the actual driving process of the vehicle, so they have nothing to do with the driving obstacles.

Specifically, the vehicle control device can input real-time image data into a pre-trained semantic segmentation model capable of semantically segmenting various objects that may appear in the vehicle driving environment, and obtain the semantic segmentation result output by the semantic segmentation model. The semantic segmentation result The pixel semantic label of each pixel in the real-time image data may be included, and the pixel semantic label is the semantic information representing the object to which the pixel belongs.

Furthermore, the semantic segmentation model can be obtained by offline training based on pre-marked training samples of various objects that may appear in the vehicle driving environment.

S130. Perform label matching on the real-time point cloud data and the real-time image data, and determine target point cloud points in the real-time point cloud data. The point cloud semantic labels of the target point cloud points are target objects, and the target objects are obstacles that do not affect driving.

In the embodiment of the present disclosure, after obtaining the pixel semantic labels of each pixel in the real-time image data, the vehicle control device can perform label matching on the real-time point cloud data and the real-time image data, and determine the point cloud semantic labels in the real-time point cloud data is the target point cloud point of the target object.

Specifically, the vehicle control device can first perform label matching on the real-time point cloud data and real-time image data to obtain the point cloud semantic labels of each real-time point cloud point in the real-time point cloud data, and then judge the point cloud for each real-time point cloud point The object category to which the cloud semantic label belongs, and then the target point cloud point with the point cloud semantic label as the target object is obtained.

Wherein, the objects included in the point cloud semantic tags and the object categories classified by the point cloud semantic tags are the same as the pixel semantic tags, which are not limited here.

S140. Eliminate target point cloud points in the real-time point cloud data to obtain target point cloud data.

In the embodiment of the present disclosure, after determining the target point cloud points in the real-time point cloud data, the vehicle control device can eliminate the target point cloud points in the real-time point cloud data to obtain the target without the influence of obstacles that do not affect driving. point cloud data.

S150. Perform driving control on the vehicle based on the target point cloud data.

In the embodiment of the present disclosure, after obtaining the target point cloud data, the vehicle control device may control the vehicle based on the target point cloud data.

Specifically, the vehicle control device can perform semantic segmentation on the target point cloud data, and then obtain an object recognition result in the vehicle driving area, and control the vehicle based on the object recognition result. For example, if the object recognition result contains an obstacle, the vehicle is controlled to stop moving; if the object recognition result does not contain an obstacle, the vehicle is controlled to continue moving.

In the embodiment of the present disclosure, based on the semantic segmentation result of the real-time image data corresponding to the vehicle driving area, tag matching can be performed on the real-time point cloud data corresponding to the vehicle driving area, and the target point cloud point in the real-time point cloud data can be determined, and the The target point cloud point is eliminated from the real-time point cloud data to obtain the target point cloud data, and then based on the target point cloud data, the driving control of the vehicle is performed. Since the point cloud semantic label of the target point cloud point is the target object and the target object does not affect driving Therefore, before controlling the vehicle based on the point cloud data, the point cloud points related to the obstacles that do not affect the driving can be eliminated, and then when the vehicle is controlled based on the point cloud data, it can avoid The obstacles on the lanes in the port that do not affect the driving interfere with the driving, thereby preventing the self-driving work vehicles in the port from stopping due to obstacles that do not affect the driving, and improving the operating efficiency of the self-driving work vehicles.

In another embodiment of the present disclosure, the vehicle control device can realize coordinate system conversion of real-time point cloud data by mapping real-time point cloud data to real-time image data, and then real-time image data in the same coordinate system and real-time Tag matching is performed on the point cloud data to determine the target point cloud points in the real-time point cloud data, so as to improve the accuracy of tag matching and the determined target point cloud points.

Fig. 2 shows a schematic flowchart of another vehicle control method provided by an embodiment of the present disclosure. As shown in Fig. 2, the vehicle control method may include the following steps.

S210. Acquire real-time image data and real-time point cloud data corresponding to the driving area of the vehicle.

S220. Perform semantic segmentation on the real-time image data to obtain a semantic segmentation result, where the semantic segmentation result includes pixel semantic labels of each pixel in the real-time image data.

Wherein, S210-S220 are similar to S110-S120 in the embodiment shown in FIG. 1 , and will not be repeated here.

S230. Project and image the real-time point cloud data from the radar coordinate system into the image coordinate system corresponding to the real-time image data, and obtain the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data.

In the embodiment of the present disclosure, after obtaining the pixel semantic labels of each pixel in the real-time image data, the vehicle control device can project the real-time point cloud data from the radar coordinate system to the corresponding image of the real-time image data according to the preset projection method In the coordinate system, the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data are obtained, so that the real-time image data and real-time point cloud data are in the same coordinate system, with comparable coordinates and matchable labels .

Wherein, the radar coordinate system is a coordinate system set with the laser radar device as a reference point, for example, a coordinate system set with the laser radar device as a coordinate origin. The image coordinate system is a coordinate system set with the image capture device as a reference point, for example, a coordinate system set with the image capture device as a coordinate origin.

In some examples, projecting and imaging the real-time point cloud data into the image coordinate system corresponding to the real-time image data, and obtaining the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data may specifically include:

According to the external parameters of the lidar equipment and the external parameters of the image acquisition equipment, the real-time point cloud coordinates of each real-time point cloud point in the radar coordinate system are mapped to the coordinate system of the image acquisition equipment to obtain the second point cloud of each real-time point cloud point coordinate;

According to the internal parameters of the image acquisition device, the second point cloud coordinates of each real-time point cloud point are mapped to the image coordinate system to obtain the first point cloud coordinates of each real-time point cloud point.

Wherein, the external parameter of the lidar device is the external parameter from the lidar device to the vehicle body, specifically, it may be a 4×4 matrix generated according to the radar position and the radar angle. Specifically, the radar position may be an offset of the radar relative to the center of the vehicle body, and the radar angle may be a rotation angle of the radar relative to the center of the vehicle body.

The external parameter of the image acquisition device is the external parameter from the vehicle body to the image acquisition device, specifically, it may be a 4×4 matrix generated according to the position of the image acquisition device and the angle of the image acquisition device. Specifically, the position of the image acquisition device may be the offset of the center of the vehicle body relative to the image acquisition device, and the angle of the image acquisition device may be the rotation angle of the center of the vehicle body relative to the image acquisition device.

Furthermore, the external parameters of the lidar device and the external parameters of the image acquisition device can be pre-calibrated, and the internal parameters of the image acquisition device can be the internal parameter matrix set by the image acquisition device at the factory.

Specifically, the vehicle control device can first map the real-time point cloud coordinates of each real-time point cloud point in the radar coordinate system to the image acquisition In the device coordinate system, the second point cloud coordinates of each real-time point cloud point are obtained, and then the second point cloud coordinates of each real-time point cloud point are mapped to the image coordinate system through the second mapping relationship formed based on the internal parameters of the image acquisition device In , the first point cloud coordinates of each real-time point cloud point are obtained.

Taking the coordinates [X _l , Y l , Z l ] of a real-time point cloud point in the radar coordinate system as an example, the coordinates [X l , Y _l , Z _l ] in the coordinate system of the image acquisition device can be obtained by performing coordinate mapping through the first mapping relationship. _c , Y _c , Z _c ], the specific mapping formula is:

Among them, RT _vl is the external parameter of the lidar equipment, RT _cv is the external parameter of the image acquisition equipment, and 1 is the homogeneous coordinate.

Then, coordinate mapping is performed on the coordinates [X _c , Y _c , Z _c ] in the image acquisition device coordinate system through the second mapping relationship, and the coordinates [u, v] in the image coordinate system can be obtained. The specific mapping formula for:

Wherein, K is an internal parameter of the image acquisition device, λ is an intermediate parameter, and λ=Z _c ×K.

Therefore, in the embodiment of the present disclosure, the above formula can be used to realize the mapping and imaging of real-time point cloud data to real-time image data, so that the coordinates of real-time image data and real-time point cloud data are comparable and matchable.

S240. For each real-time point cloud point, based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud point, determine a point cloud semantic label of the real-time point cloud point.

In the embodiment of the present disclosure, after obtaining the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data, the vehicle control device can Classify each real-time point cloud point, and for each real-time point cloud point, the vehicle control device can base on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud point, according to the matching method corresponding to the classification of the real-time point cloud point, Determining point cloud semantic labels for real-time point cloud points.

In some embodiments, based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud point, determining the point cloud semantic label of the real-time point cloud point may specifically include:

When each coordinate component of the first point cloud coordinates is an integer value, the pixel semantic label of the pixel corresponding to the first point cloud coordinate is used as the point cloud semantic label of the real-time point cloud point.

Since each coordinate component of the pixel coordinates of each pixel point of the real-time image data is an integer value, if each coordinate component of the first point cloud coordinates is an integer value, then the first point cloud coordinates have a one-to-one corresponding pixel point Pixel coordinates, that is to say, the real-time point cloud point to which the first point cloud coordinate belongs has a completely matching pixel point at the same coordinate position, therefore, the real-time point cloud coordinate to which each coordinate component is an integer value belongs to Point cloud points are divided into one class.

Specifically, for such real-time point cloud points, the vehicle control device can directly determine the pixel point at the same coordinate position as the real-time point cloud point according to the first point cloud coordinates of the real-time point cloud point, as the first point cloud coordinate corresponding , and use the pixel semantic label of the pixel as the point cloud semantic label of the real-time point cloud point.

In other embodiments, based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud point, determining the point cloud semantic label of the real-time point cloud point may specifically include:

When at least one coordinate component of the first point cloud coordinate is a non-integer value, based on the pixel semantic label of each pixel point surrounding the first point cloud coordinate, the point cloud semantic label of the real-time point cloud point is determined.

Since each coordinate component of the pixel coordinates of each pixel point of the real-time image data is an integer value, if at least one coordinate component of the first point cloud coordinate is a non-integer value, then the first point cloud coordinate does not have a one-to-one corresponding pixel The pixel coordinates of the point, that is to say, the real-time point cloud point to which the first point cloud coordinates belong does not have an exact matching pixel point at the same coordinate position. Therefore, at least one coordinate component can be the first point cloud with a non-integer value The real-time point cloud points to which the coordinates belong are divided into one category.

Specifically, for such real-time point cloud points, the vehicle control device cannot directly use the pixel semantic label of a certain pixel point as the point cloud semantic label of the real-time point cloud point, but needs to use pixel coordinates to surround the first point cloud The pixel semantic label of each pixel point of the coordinate, that is, the pixel semantic label of each pixel surrounding the real-time point cloud point, determines the point cloud semantic label of the real-time point cloud point.

In some examples, the vehicle control device may determine all pixel points whose pixel coordinates surround the first point cloud coordinates according to the first point cloud coordinates of the real-time point cloud points, and select among these pixel points The pixel point whose coordinates are closest to the pixel coordinates belongs to, and then the pixel semantic label of the selected pixel point is used as the point cloud semantic label of the real-time point cloud point.

In other examples, the vehicle control device may determine all pixel points whose pixel coordinates surround the first point cloud coordinates according to the first point cloud coordinates of the real-time point cloud point, and determine the pixel semantic label of each pixel point, and then The pixel semantic labels of these pixels are counted. If the number of pixel semantic labels corresponding to a certain object is larger than the number of pixel semantic labels corresponding to other objects, the pixel semantic label with the largest number is used as the real-time point cloud point. Otherwise, select the pixel point belonging to the pixel coordinate closest to the first point cloud coordinates among these pixels, and then use the pixel semantic label of the selected pixel point as the point of the real-time point cloud point Cloud semantic tags.

Therefore, in the embodiment of the present disclosure, the pixel points related to the real-time point cloud points can be determined according to the first point cloud coordinates of the real-time point cloud points, and the corresponding real-time point cloud point matching can be obtained based on the determined pixel points. Point cloud semantic tags, and then accurately and efficiently realize the analysis of voice information of point cloud points.

S250. Use the real-time point cloud points with the point cloud semantic label as the target object as the target point cloud points respectively.

In the embodiment of the present disclosure, after determining the point cloud semantic tags of all real-time point cloud points, for each real-time point cloud point, it is possible to judge the object category to which the point cloud semantic tag belongs, and set the point cloud semantic tag as the target object The real-time point cloud points are used as the target point cloud points respectively, and then the target point cloud points with the point cloud semantic label as the target object are obtained.

S260. Eliminate target point cloud points in the real-time point cloud data to obtain target point cloud data.

S270. Perform driving control on the vehicle based on the target point cloud data.

Wherein, S260-S270 are similar to S140-S150 in the embodiment shown in FIG. 1 , and will not be repeated here.

In the embodiment of the present disclosure, before the vehicle is controlled based on the point cloud data, the point cloud points related to the obstacles that do not affect the driving can be eliminated, and then when the vehicle is controlled based on the point cloud data, it can Avoid the interference of obstacles that do not affect driving on the lanes in the port to the driving, thereby preventing the self-driving work vehicles in the port from stopping due to obstacles that do not affect driving, and improving the operating efficiency of the self-driving work vehicles. Specifically, with the help of the rich semantic information of the image data, the external parameters of the image acquisition equipment and the participation of the lidar equipment, semantic information can be given to each point cloud point, and then the target point cloud points related to obstacles that do not affect driving can be identified, improving the The recognition accuracy of target point cloud points. To sum up, compared with the method of avoiding vehicle driving interference through manual weeding, the embodiment of the present disclosure saves a lot of human resources and time costs, and compared with the method of avoiding vehicle driving interference through weeding with herbicides, it can ensure port food. Safe, compared with the method of recognizing objects only through semantic segmentation of point clouds, the recognition results are more accurate and efficient.

Fig. 3 shows a schematic flowchart of another vehicle control method provided by an embodiment of the present disclosure. As shown in Fig. 3, the vehicle control method may include the following steps.

S310. Acquire real-time image data and real-time point cloud data corresponding to the driving area of the vehicle.

S320. Perform semantic segmentation on the real-time image data to obtain a semantic segmentation result, where the semantic segmentation result includes a pixel semantic label of each pixel in the real-time image data.

S330. Project and image the real-time point cloud data from the radar coordinate system into the image coordinate system corresponding to the real-time image data, and obtain the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data.

Wherein, S310-S330 are similar to S210-S230 in the embodiment shown in FIG. 2 , and will not be repeated here.

S340. Eliminate invalid point cloud points in the real-time point cloud data, where the invalid point cloud points are point cloud points whose first point cloud coordinates are outside the coordinate range corresponding to the real-time image data.

In the embodiment of the present disclosure, after obtaining the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data, for each real-time point cloud point, it can be determined whether the first point cloud coordinates of the real-time point cloud point fall within within the coordinate range corresponding to the real-time image data, if the first point cloud coordinate of the real-time point cloud point falls within the coordinate range corresponding to the real-time image data, the real-time point cloud point is determined to be a valid point cloud point, and the Valid point cloud points are kept in the real-time point cloud data, otherwise, if the first point cloud coordinates of the real-time point cloud points do not fall within the coordinate range corresponding to the real-time image data, that is, the first point cloud coordinates are located in the corresponding coordinate range of the real-time image data. If it is outside the coordinate range, it is determined that the real-time point cloud point is an invalid point cloud point, and the invalid point cloud point is removed from the real-time point cloud data.

Wherein, the coordinate range corresponding to the real-time image data is the coordinate range formed by the pixel coordinates of all pixel points in the real-time image data in the image coordinate system.

In general, the data acquisition range of the lidar equipment is larger than that of the image acquisition equipment. Therefore, after the real-time point cloud data is projected from the radar coordinate system into the corresponding image coordinate system of the real-time image data, there will be some The point cloud data does not fall into the data acquisition range of the image acquisition equipment, and the semantic analysis of this part of the point cloud data cannot be realized through the semantic segmentation results of real-time image data. Therefore, before label matching, this part of point cloud data can be eliminated to reduce the amount of data processing and improve the efficiency of data processing.

If the first point cloud coordinates of the real-time point cloud point fall within the coordinate range corresponding to the real-time image data, the real-time point cloud point is located in the driving environment reflected by the real-time image data, that is, the real-time point cloud point falls into the image acquisition In the data collection range of the device, it means that the semantic segmentation result of the real-time image data can realize the label matching of the real-time point cloud point. The real-time point cloud point is a valid point cloud point, and subsequent label matching processing can be performed. Otherwise, If the first point cloud coordinates of the real-time point cloud point do not fall within the coordinate range corresponding to the real-time image data, the real-time point cloud point is not located in the driving environment reflected by the real-time image data, that is, the real-time point cloud point does not fall Into the data acquisition range of the image acquisition device, that is to say, the semantic segmentation result of the real-time image data cannot realize the label matching of the real-time point cloud point, the real-time point cloud point is an invalid point cloud point, and no subsequent labeling can be performed match processing.

S350. For each real-time point cloud point, based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud point, determine a point cloud semantic label of the real-time point cloud point.

S360. Taking the real-time point cloud points with the point cloud semantic label as the target object as the target point cloud points respectively.

S370. Eliminate target point cloud points in the real-time point cloud data to obtain target point cloud data.

S380. Perform driving control on the vehicle based on the target point cloud data.

Wherein, S350-S380 are similar to S240-S270 in the embodiment shown in FIG. 2 , and will not be repeated here.

It should be noted that, in order to further reduce the amount of data processing and improve data processing efficiency, the real-time point cloud data described in S370 can be the point cloud data after invalid point cloud points are eliminated in S340; in order to improve the driving control of the vehicle accuracy, the real-time point cloud data described in S370 may also be the point cloud data acquired in S310, which is not limited here.

In the embodiment of the present disclosure, before performing label matching on real-time point cloud data and real-time image data, invalid point cloud points that are not related to the driving environment corresponding to the real-time image data can be eliminated, and then only for the real-time image data The point cloud points related to the corresponding driving environment are tag-matched, which can reduce the amount of data processing and improve the efficiency of data processing, thereby improving the operating efficiency of self-driving work vehicles. Moreover, before the vehicle is controlled based on the point cloud data, the point cloud points related to obstacles that do not affect the driving can be eliminated, and then when the vehicle is controlled based on the point cloud data, the lanes in the port can be avoided. The obstacles on the road that do not affect the driving interfere with the driving, thereby preventing the self-driving work vehicles in the port from stopping due to obstacles that do not affect the driving, and improving the operating efficiency of the self-driving work vehicles.

Fig. 4 shows a schematic structural diagram of a vehicle control device provided by an embodiment of the present disclosure. The vehicle control device 400 shown in FIG. 4 can be applied to a vehicle control apparatus. Wherein, the vehicle control device may be an on-board device mounted on the vehicle or a controller of the vehicle, and the vehicle may be an automatic driving work vehicle.

As shown in FIG. 4 , the vehicle control device 400 may include a data acquisition module 410 , a semantic segmentation module 420 , a label matching module 430 , a data elimination module 440 and a driving control module 450 .

The data acquisition module 410 can be used to acquire real-time image data and real-time point cloud data corresponding to the driving area of the vehicle.

The semantic segmentation module 420 may be used to perform semantic segmentation on real-time image data to obtain a semantic segmentation result, which includes pixel semantic labels of each pixel in the real-time image data.

The tag matching module 430 can be used to tag match the real-time point cloud data with the real-time image data, determine the target point cloud point in the real-time point cloud data, the point cloud semantic tag of the target point cloud point is the target object, and the target object is not Obstacles that affect driving.

The data elimination module 440 can be used to eliminate target point cloud points in real-time point cloud data to obtain target point cloud data.

The driving control module 450 can be used to control the driving of the vehicle based on the target point cloud data.

In the embodiment of the present disclosure, based on the semantic segmentation result of the real-time image data corresponding to the vehicle driving area, tag matching can be performed on the real-time point cloud data corresponding to the vehicle driving area, and the target point cloud point in the real-time point cloud data can be determined, and the The target point cloud point is eliminated from the real-time point cloud data to obtain the target point cloud data, and then based on the target point cloud data, the driving control of the vehicle is performed. Since the point cloud semantic label of the target point cloud point is the target object and the target object does not affect driving Therefore, before controlling the vehicle based on the point cloud data, the point cloud points related to the obstacles that do not affect the driving can be eliminated, and then when the vehicle is controlled based on the point cloud data, it can avoid The obstacles on the lanes in the port that do not affect the driving interfere with the driving, thereby preventing the self-driving work vehicles in the port from stopping due to obstacles that do not affect the driving, and improving the operating efficiency of the self-driving work vehicles.

In some embodiments of the present disclosure, the tag matching module 430 may include a projection imaging unit, a first determination unit and a second determination unit.

The projection imaging unit can be used to project and image the real-time point cloud data from the radar coordinate system to the image coordinate system corresponding to the real-time image data, so as to obtain the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data.

The first determining unit may be used for determining, for each real-time point cloud point, a point cloud semantic label of the real-time point cloud point based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud point.

The second determination unit may be used to use the real-time point cloud points with the point cloud semantic label as the target object as the target point cloud points respectively.

In some embodiments of the present disclosure, the projection imaging unit can be further used to map the real-time point cloud coordinates of each real-time point cloud point in the radar coordinate system to the image acquisition In the device coordinate system, the second point cloud coordinates of each real-time point cloud point are obtained; according to the internal parameters of the image acquisition device, the second point cloud coordinates of each real-time point cloud point are mapped to the image coordinate system, and the coordinates of each real-time point cloud point are obtained The first point cloud coordinates.

In some embodiments of the present disclosure, the first determination unit may be further configured to use the pixel semantic label of the pixel corresponding to the first point cloud coordinate as a real-time point when each coordinate component of the first point cloud coordinate is an integer value Point cloud semantic labeling of cloud points.

In some embodiments of the present disclosure, the first determination unit may be further configured to, when at least one coordinate component of the first point cloud coordinates is a non-integer value, based on the pixel semantic labels of each pixel point surrounding the first point cloud coordinates, Determining point cloud semantic labels for real-time point cloud points.

In some embodiments of the present disclosure, the label matching module 430 may also include a point cloud elimination unit, which is used for each real-time point cloud point based on the semantic segmentation result and the first point of the real-time point cloud point Cloud coordinates, before determining the point cloud semantic tags of real-time point cloud points, remove invalid point cloud points in real-time point cloud data, invalid point cloud points are point clouds whose first point cloud coordinates are outside the coordinate range corresponding to real-time image data point.

It should be noted that the vehicle control device 400 shown in FIG. 4 can execute various steps in the method embodiments shown in FIGS. 1 to 3 , and realize various processes and effects in the method embodiments shown in FIGS. 1 to 3 , I won't go into details here.

Fig. 5 shows a schematic structural diagram of a vehicle control device provided by an embodiment of the present disclosure. The vehicle control device shown in FIG. 5 may be an on-board device mounted on the vehicle or a controller of the vehicle, and the vehicle may be an automatic driving work vehicle.

As shown in FIG. 5 , the vehicle control device may include a processor 501 and a memory 502 storing computer program instructions.

Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or may be configured to implement one or more integrated circuits in the embodiments of the present application.

Memory 502 may include mass storage for information or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (Hard Disk Drive, HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a Universal Serial Bus (Universal Serial Bus, USB) drive or two or more thereof. A combination of the above. Storage 502 may include removable or non-removable (or fixed) media, where appropriate. Memory 502 may be internal or external to the integrated gateway device, where appropriate. In a particular embodiment, memory 502 is a non-volatile solid-state memory. In a particular embodiment, the memory 502 includes a read-only memory (Read-Only Memory, ROM). Where appropriate, the ROM can be mask programmed ROM, programmable ROM (Programmable ROM, PROM), erasable PROM (Electrical Programmable ROM, EPROM), electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM) , Electrically rewritable ROM (Electrically Alterable ROM, EAROM) or flash memory, or a combination of two or more of these.

The processor 501 executes the steps of the vehicle control method provided by the embodiments of the present disclosure by reading and executing the computer program instructions stored in the memory 502 .

In an example, the vehicle control device may further include a transceiver 503 and a bus 504 . Wherein, as shown in FIG. 5 , the processor 501 , the memory 502 and the transceiver 503 are connected through a bus 504 and complete mutual communication.

Bus 504 includes hardware, software, or both. By way of example and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (Hyper Transport, HT) interconnection, Industrial Standard Architecture (Industrial Standard Architecture, ISA) bus, Infinity Bandwidth interconnection, Low Pin Count (Low Pin Count, LPC) bus, memory bus, Micro Channel Architecture (MicroChannel Architecture, MCA ) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (Serial Advanced Technology Attachment, SATA) bus, Video Electronics Standards Association Local Bus, VLB) bus or other suitable bus or a combination of two or more of these. Bus 504 may comprise one or more buses, where appropriate. Although the embodiments of this application describe and illustrate a particular bus, this application contemplates any suitable bus or interconnect.

An embodiment of the present disclosure also provides a computer-readable storage medium, which can store a computer program, and when the computer program is executed by a processor, the processor can implement the vehicle control method provided by the embodiment of the present disclosure.

The above-mentioned storage medium may include, for example, a memory 502 of computer program instructions, and the above-mentioned instructions can be executed by the processor 501 of the vehicle control device to complete the vehicle control method provided by the embodiments of the present disclosure. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be ROM, random access memory (Random Access Memory, RAM), compact disc read-only memory (Compact Disc ROM) , CD-ROM), tapes, floppy disks and optical data storage devices, etc.

An embodiment of the present application also provides a vehicle, which may include at least one of a vehicle control device, a vehicle control device, and a computer storage medium, and the specific vehicle control device, vehicle control device, and computer storage medium involved are the same The description of the content is consistent and will not be repeated here.

It should be noted that in this article, relative terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these No such actual relationship or order exists between entities or operations. Moreover, the term "comprising" is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed, or is also included as such. An element inherent in a process, method, article, or device.

The above descriptions are only specific implementation manners of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A vehicle control method, characterized in that, comprising: Obtain real-time image data and real-time point cloud data corresponding to the vehicle driving area; Semantic segmentation is performed on the real-time image data to obtain a semantic segmentation result, the semantic segmentation result including pixel semantic labels of each pixel in the real-time image data; The real-time point cloud data is tag-matched with the real-time image data, and the target point cloud point in the real-time point cloud data is determined, the point cloud semantic label of the target point cloud point is a target object, and the target object An obstacle that does not affect driving; Removing the target point cloud points in the real-time point cloud data to obtain target point cloud data; Based on the target point cloud data, the driving control of the vehicle is performed. 2. method according to claim 1, is characterized in that, described real-time point cloud data and described real-time image data are carried out label matching, determine the target point cloud point in the described real-time point cloud data, comprising: Projecting and imaging the real-time point cloud data from the radar coordinate system into the corresponding image coordinate system of the real-time image data to obtain the first point cloud coordinates of each real-time point cloud point in the real-time point cloud data; For each of the real-time point cloud points, based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud points, determine the point cloud semantic tags of the real-time point cloud points; The real-time point cloud points of the point cloud semantically labeled as the target object are respectively used as the target point cloud points. 3. method according to claim 2, is characterized in that, described real-time point cloud data projection imaging is described in the corresponding image coordinate system of described real-time image data, obtains each real-time point cloud in described real-time point cloud data The first point cloud coordinates of the point, including: According to the external parameters of the laser radar equipment and the external parameters of the image acquisition equipment, the real-time point cloud coordinates of the various real-time point cloud points in the radar coordinate system are mapped to the coordinate system of the image acquisition equipment to obtain the various real-time point clouds. The second point cloud coordinates of the point; According to the internal reference of the image acquisition device, the second point cloud coordinates of the respective real-time point cloud points are mapped to the image coordinate system to obtain the first point cloud coordinates of the respective real-time point cloud points. 4. method according to claim 2, is characterized in that, described based on described semantic segmentation result and the first point cloud coordinate of described real-time point cloud point, determine the point cloud semantic label of described real-time point cloud point, include: When each coordinate component of the first point cloud coordinates is an integer value, the pixel semantic label of the pixel corresponding to the first point cloud coordinate is used as the point cloud semantic label of the real-time point cloud point. 5. The method according to claim 2, wherein, the first point cloud coordinates based on the semantic segmentation result and the real-time point cloud point determine the point cloud semantic label of the real-time point cloud point, include: When at least one coordinate component of the first point cloud coordinates is a non-integer value, the point cloud semantic tags of the real-time point cloud points are determined based on the pixel semantic tags of each pixel point surrounding the first point cloud coordinates. 6. The method according to claim 2, wherein, at each of the real-time point cloud points, based on the semantic segmentation result and the first point cloud coordinates of the real-time point cloud points, determine the Before the point cloud semantic label of the real-time point cloud point, the method also includes: Eliminate invalid point cloud points in the real-time point cloud data, where the invalid point cloud points are point cloud points whose coordinates of the first point cloud are outside the coordinate range corresponding to the real-time image data. 7. A vehicle control device, characterized in that it comprises: The data acquisition module is used to acquire real-time image data and real-time point cloud data corresponding to the vehicle driving area; A semantic segmentation module, configured to perform semantic segmentation on the real-time image data to obtain a semantic segmentation result, the semantic segmentation result including pixel semantic labels of each pixel in the real-time image data; A tag matching module, configured to tag-match the real-time point cloud data with the real-time image data, determine the target point cloud point in the real-time point cloud data, and the point cloud semantic tag of the target point cloud point is the target object, the target object is an obstacle that does not affect driving; A data elimination module, configured to eliminate the target point cloud points in the real-time point cloud data to obtain target point cloud data; A driving control module, configured to control the driving of the vehicle based on the target point cloud data. 8. A vehicle control device, characterized in that it comprises: processor; memory for storing executable instructions; Wherein, the processor is configured to read the executable instructions from the memory, and execute the executable instructions to implement the vehicle control method described in any one of claims 1-6 above. 9. A computer-readable storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the processor is enabled to implement any one of the preceding claims 1-6. vehicle control method. 10. A vehicle, characterized in that it comprises at least one of the following: A vehicle control device as claimed in claim 7; A vehicle control device as claimed in claim 8; The computer storage medium of claim 9.

Images