WO2024087456A1

WO2024087456A1 - Determination of orientation information and autonomous vehicle

Info

Publication number: WO2024087456A1
Application number: PCT/CN2023/080380
Authority: WO
Inventors: 史磊
Original assignee: 北京三快在线科技有限公司
Priority date: 2022-10-26
Filing date: 2023-03-09
Publication date: 2024-05-02
Also published as: CN117975404A

Abstract

The present application discloses an orientation information determining method, comprising: acquiring point cloud data and image data obtained by photographing a target obstacle by an autonomous vehicle; performing first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle, and performing second feature extraction on the image data of the target obstacle to obtain an appearance feature of the target obstacle; on the basis of the geometric feature of the target obstacle, determining a candidate orientation set of the target obstacle, the candidate orientation set comprising a plurality of pieces of candidate orientation information, and the plurality of pieces of candidate orientation information indicating the orientations of a plurality of surfaces of the target obstacle; on the basis of the appearance feature of the target obstacle, determining first orientation information of the target obstacle; and determining second orientation information from the candidate orientation set as the orientation information of the target obstacle, wherein an orientation indicated by the second orientation information matches an orientation indicated by the first orientation information.

Description

Determining orientation information and autonomous vehicles

Technical Field

The present application relates to the field of autonomous driving technology, and in particular to a method for determining orientation information and an autonomous driving vehicle.

Background technique

In the application scenario of autonomous driving, the orientation information of obstacles has an important impact on the decision-making and path planning of autonomous driving vehicles. Therefore, how to determine the orientation information of obstacles around autonomous driving vehicles has become a problem that needs to be solved urgently.

Currently, the point cloud data of obstacles can be obtained through the radar of the autonomous driving vehicle, and the direction information of the obstacle can be determined based on the point cloud data.

Summary of the invention

The embodiment of the present application provides a method for determining direction information and an autonomous driving vehicle, which can more accurately determine the direction of an obstacle. The technical solution is as follows:

In one aspect, a method for determining orientation information is provided, the method comprising:

Obtain point cloud data and image data taken by the autonomous driving vehicle for target obstacles;

Performing a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle, and performing a second feature extraction on the image data of the target obstacle to obtain an apparent feature of the target obstacle;

Determine, based on the geometric features of the target obstacle, a candidate orientation set of the target obstacle, wherein the candidate orientation set includes a plurality of candidate orientation information, and the plurality of candidate orientation information indicates orientations of a plurality of faces of the target obstacle;

Determining first orientation information of the target obstacle based on the apparent characteristics of the target obstacle;

Second orientation information whose indicated orientation matches the orientation indicated by the first orientation information is determined from the candidate orientation set as the orientation information of the target obstacle.

In one aspect, a device for determining orientation information is provided, the device comprising:

An acquisition module is used to acquire point cloud data and image data taken by the autonomous driving vehicle for the target obstacle;

A feature extraction module, configured to perform a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle, and perform a second feature extraction on the image data of the target obstacle to obtain an apparent feature of the target obstacle;

A first determination module is used to determine a candidate orientation set of the target obstacle based on the geometric features of the target obstacle, wherein the candidate orientation set includes a plurality of candidate orientation information, and the plurality of candidate orientation information indicates orientations of a plurality of faces of the target obstacle;

A second determination module, configured to determine first orientation information of the target obstacle based on the apparent characteristics of the target obstacle;

A third determination module is configured to determine, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the first orientation information as the orientation information of the target obstacle.

In one possible implementation, the orientation information includes an orientation angle; the first determination module is used to determine the orientation angle of the target obstacle based on the geometric features; the orientation angle is increased by 90 degrees, 180 degrees and 270 degrees respectively to obtain multiple expanded orientation angles; the orientation angle and the multiple expanded orientation angles are determined as candidate orientation information to obtain the candidate orientation set.

In a possible implementation, the first determination module is configured to determine orientation information of multiple faces of the target obstacle based on the geometric features; and determine the orientation information of the multiple faces as a candidate orientation set of the target obstacle.

In a possible implementation, the image data is multi-frame image data, and the multi-frame image data is image data obtained by the autonomous driving vehicle photographing the target obstacle at different times;

The feature extraction module is used to perform a third feature extraction on each frame of image data in the multiple frames of image data of the target obstacle to obtain a first appearance feature corresponding to each frame of image data; and perform the following operations on each frame of image data in sequence according to the shooting order of the multiple frames of image data: fusing the first appearance feature of the current frame of image data with the first appearance feature of the previous frame of image data to obtain a second appearance feature of the current frame of image data; and determining the first orientation information of the target obstacle based on the second appearance feature of the last frame of image data.

In a possible implementation, the first orientation information is a first orientation category; the apparent features of the target obstacle and the first orientation category are determined by an orientation classification model, and the orientation classification model is used to determine an orientation category that matches the apparent features of the obstacle from a target number of orientation categories.

In one possible implementation, the target number of orientation categories includes front, right front, right, right rear, rear, left rear, left and left front; or, the target number of orientation categories includes 0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees and 315 degrees.

In one possible implementation, the multiple candidate orientation information indicate the orientation of the target obstacle in the autonomous driving vehicle coordinate system, the first orientation information indicates the orientation of the target obstacle in the observer coordinate system, and the observer coordinate system has a first ray pointing from the autonomous driving vehicle to the target obstacle as the vertical axis and a second ray passing through the center of the target obstacle and perpendicular to the first ray as the horizontal axis.

In one possible implementation, the third determination module is used to determine the angle between the autonomous driving vehicle coordinate system and the observer coordinate system; based on the angle, determine fourth orientation information corresponding to the first orientation information in the autonomous driving vehicle coordinate system, the fourth orientation information indicating the orientation of the target obstacle in the autonomous driving vehicle coordinate system; and determine second orientation information from the candidate orientation set whose indicated orientation matches the orientation indicated by the fourth orientation information as the orientation information of the target obstacle.

In a possible implementation, the third determination module is configured to determine, from the candidate orientation set, second orientation information whose indicated orientation has the smallest difference from the orientation indicated by the fourth orientation information as the orientation information of the target obstacle.

On the one hand, an autonomous driving vehicle is provided, which includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the orientation information determination method in any possible implementation method as described above.

On the one hand, a computer-readable storage medium is provided, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor to implement the operations performed by the orientation information determination method in any possible implementation manner as described above.

On the one hand, a computer program or a computer program product is provided, wherein the computer program or the computer program product comprises: a computer program code, wherein when the computer program code is executed by a computer, the computer implements the operations performed by the method for determining the orientation information in any possible implementation manner as described above.

The method for determining orientation information and the autonomous driving vehicle provided in the embodiments of the present application can be based on the geometry of the target obstacle in the point cloud data. The coarse-grained candidate orientation set is determined by the geometric information in the point cloud data and the appearance information in the image data. One orientation information is selected from the candidate orientation set as the orientation information of the target obstacle according to the appearance information of the target obstacle in the image data. This is equivalent to jointly determining the orientation information of the target obstacle by integrating the geometric information in the point cloud data and the appearance information in the image data, which can more accurately determine the orientation of the target obstacle.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

FIG2 is a flow chart of a method for determining orientation information provided by an embodiment of the present application;

FIG3 is a flow chart of a method for determining orientation information provided by an embodiment of the present application;

FIG4 is a schematic diagram of the structure of a direction classification model provided in an embodiment of the present application;

FIG5 is a schematic diagram of image data provided by an embodiment of the present application;

FIG6 is a schematic diagram of a coordinate system provided in an embodiment of the present application;

FIG7 is a schematic diagram of the structure of a device for determining orientation information provided in an embodiment of the present application;

FIG8 is a schematic diagram of the structure of an autonomous driving vehicle provided in an embodiment of the present application;

FIG. 9 is a schematic diagram of the structure of a server provided in an embodiment of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application more clear, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

It is understood that the terms "first", "second", etc. used in this application can be used in this article to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, a first order can be called a second order, and a second order can be called a first order without departing from the scope of this application.

The terms "at least one", "multiple", "each", and "any" used in this application, at least one includes one, two or more than two, multiple includes two or more than two, and each refers to each of the corresponding multiple, and any refers to any one of the multiple. For example, multiple corner points include 3 corner points, and each refers to each of the 3 corner points, and any refers to any one of the 3 corner points, which can be the first, the second, or the third.

It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image data, point cloud data, etc. involved in this application are all obtained with full authorization. Moreover, the above information and data are processed and used in big data application scenarios, and cannot be identified to any natural person or have a specific association with him.

In some embodiments, the method for determining the orientation information provided in the embodiments of the present application is performed by an autonomous vehicle. The autonomous vehicle is any device with an automatic driving function. In some embodiments, the autonomous vehicle includes a vehicle traveling on the ground (e.g., a car, a truck, a bus, etc.), and may also include a vehicle traveling in the air (e.g., a drone, an airplane, a helicopter, etc.). It may also include vehicles traveling on or in water (e.g., ships, submarines, etc.). The autonomous driving vehicle may or may not accommodate one or more passengers. In addition, the autonomous driving vehicle may be applied to unmanned delivery fields, such as express logistics fields, takeaway food delivery fields, etc.

In other embodiments, the method for determining the orientation information provided in the embodiments of the present application is performed by an autonomous driving vehicle and a server. The server may be a single server, a server cluster consisting of a plurality of servers, or a cloud computing service center.

It should be noted that the embodiment of the present application does not limit the execution subject of the method for determining the orientation information.

FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. As shown in FIG1 , the implementation environment includes an autonomous driving vehicle 101 and a server 102. The autonomous driving vehicle 101 and the server 102 are connected via a wireless or wired network.

In some embodiments, the autonomous driving vehicle 101 takes a photo of the target obstacle to obtain point cloud data and image data of the target obstacle. The autonomous driving vehicle 101 determines the orientation information of the target obstacle based on the point cloud data and image data, and sends the orientation information of the target obstacle to the server 102. The server 102 plans the driving path of the autonomous driving vehicle 101 based on the orientation information of the target obstacle, and sends the driving path to the autonomous driving vehicle 101. The autonomous driving vehicle 101 receives the driving path and drives according to the driving path.

In some embodiments, the autonomous driving vehicle 101 takes a photo of the target obstacle to obtain the point cloud data and image data of the target obstacle, and the autonomous driving vehicle 101 sends the point cloud data and image data of the target obstacle to the server 102. The server 102 determines the orientation information of the target obstacle based on the point cloud data and image data of the target obstacle. The server 102 sends the orientation information of the target obstacle to the autonomous driving vehicle 101, and the autonomous driving vehicle 101 plans the driving path of the autonomous driving vehicle 101 based on the orientation information of the target obstacle; or, the server 102 plans the driving path of the autonomous driving vehicle 101 based on the orientation information of the target obstacle, sends the driving path to the autonomous driving vehicle 101, and the autonomous driving vehicle 101 drives according to the driving path. Of course, the autonomous driving vehicle 101 can also complete the above process together with the server 102, and this application does not limit what the autonomous driving vehicle 101 and the server 102 do specifically.

FIG2 is a flow chart of a method for determining orientation information provided in an embodiment of the present application. The present application embodiment is illustrated by taking an autonomous driving vehicle as an example. The embodiment includes:

201. The autonomous driving vehicle obtains point cloud data and image data taken for the target obstacle.

The autonomous driving vehicle is any device with an automatic driving function. The target obstacle is an object that may interfere with the autonomous driving vehicle during its automatic driving. For example, the target obstacle is other vehicles, pedestrians, etc. around the autonomous driving vehicle or in the lane where the autonomous driving vehicle is located. The embodiments of the present application do not limit the target obstacle.

In some embodiments, the autonomous driving vehicle is equipped with a radar and a camera. The autonomous driving vehicle photographs the target obstacle through the radar to obtain point cloud data of the target obstacle; the autonomous driving vehicle photographs the target obstacle through the camera to obtain image data of the target obstacle.

202. The autonomous driving vehicle performs a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle, and performs a second feature extraction on the image data of the target obstacle to obtain an apparent feature of the target obstacle.

Geometric features are used to describe the geometric shape of the target obstacle. Appearance features are used to describe the texture, color and other information of the target obstacle.

203. The autonomous driving vehicle determines a set of candidate orientations of the target obstacle based on the geometric features of the target obstacle. The set includes a plurality of candidate orientation information, where the plurality of candidate orientation information indicates orientations of a plurality of faces of the target obstacle.

Unless otherwise specified, the orientation of the target obstacle refers to the orientation of the front of the target obstacle. Taking the target obstacle as a vehicle as an example, the orientation of the vehicle is the orientation of the front of the vehicle. Taking the target obstacle as a pedestrian as an example, the orientation of the pedestrian is the orientation of the face. However, the geometric features of the front and the geometric features of the back or other faces in the point cloud data of some target obstacles are similar. When the orientation information of the target obstacle is determined based on the geometric features of the target obstacle, it is very likely that the determined orientation information is the orientation information of the back or other faces of the target obstacle, resulting in inaccurate orientation information. For example, the target obstacle is a large vehicle (for example, a bus, a coach, a truck, etc.), and the geometric shapes of the front and rear of the vehicle are similar. When the orientation information of the vehicle is determined based on the geometric features of the vehicle, the rear of the vehicle may be misjudged as the front of the vehicle, and the orientation information of the rear of the vehicle is determined as the orientation information of the vehicle, resulting in the determined orientation information being the opposite orientation information.

It should be noted that when determining the orientation information based on the geometric features of the target obstacle, the precise orientation angle can be determined, but the determined orientation angle may not be the orientation angle corresponding to the front side but the orientation angle corresponding to the back side or other sides.

To this end, in an embodiment of the present application, when the autonomous driving vehicle determines the orientation information of the target obstacle based on the geometric features of the target obstacle, it determines the orientation information of multiple faces of the target obstacle and uses the orientation information of the multiple faces as candidates for the target obstacle orientation information.

204. The autonomous driving vehicle determines first orientation information of the target obstacle based on the apparent characteristics of the target obstacle.

It should be noted that the resolution of the image data is low, and it is difficult to determine accurate orientation information based on the apparent characteristics of the target obstacle. However, based on the apparent characteristics of the target obstacle, the front of the target obstacle can be clearly determined. Therefore, the first orientation information of the target obstacle determined based on the apparent characteristics of the target obstacle has a certain reference value.

For example, based on the apparent features of the vehicle, it can be clearly determined that the front of the vehicle is roughly facing east, but it is impossible to accurately determine whether the front of the vehicle is facing due east, or whether there is a certain angle between the front of the vehicle and due east, and how large the angle is.

205. The autonomous driving vehicle determines, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the first orientation information as the orientation information of the target obstacle.

When determining the orientation information based on the geometric features of the target obstacle, the accurate orientation angle can be determined, but the determined orientation angle may not be the orientation angle corresponding to the front side but the orientation angle corresponding to the back side or other sides. Based on the apparent features of the target obstacle, the front side of the target obstacle can be clearly determined. Therefore, the first orientation information can be referenced to select an orientation information from the candidate orientation set as the orientation information of the target obstacle.

Although the first orientation information is not accurate enough, the approximate direction is correct. Therefore, the second orientation information whose indicated orientation matches the orientation indicated by the first orientation information can be determined from the candidate orientation set as the orientation information of the target obstacle. That is, the second orientation information that is most similar to the first orientation information can be determined from the candidate orientation set as the orientation information of the target obstacle.

The method for determining orientation information provided in the embodiment of the present application can determine a coarse-grained candidate orientation set based on the geometric information of the target obstacle in the point cloud data, and select one orientation information from the candidate orientation set as the orientation information of the target obstacle according to the apparent information of the target obstacle in the image data. This is equivalent to jointly determining the orientation information of the target obstacle by integrating the geometric information in the point cloud data and the apparent information in the image data, and can more accurately determine the orientation of the target obstacle.

FIG3 is a flow chart of a method for determining orientation information provided in an embodiment of the present application. The present application embodiment is illustrated by taking an autonomous driving vehicle as an example. The embodiment includes:

301. The autonomous driving vehicle obtains point cloud data and image data of the target obstacle.

In some embodiments, the point cloud data and the image data are acquired at the same time, ensuring that the orientations of the target obstacles in the point cloud data and the image data are the same.

In some embodiments, the autonomous driving vehicle identifies obstacles from the surrounding environment during driving, and executes the method steps provided in the embodiments of the present application for any identified obstacle. The obstacle can be any movable object, so determining the direction of the obstacle can assist in determining the state of the obstacle at the next moment, which is conducive to the autonomous driving vehicle making more accurate decisions.

302. The autonomous driving vehicle performs a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle.

In the embodiments of the present application, the autonomous driving vehicle may use any feature extraction algorithm to perform a first feature extraction on the point cloud data of the target obstacle to obtain the geometric features of the target obstacle. In some embodiments, the autonomous driving vehicle performs a first feature extraction on the point cloud data of the target obstacle through a point cloud deep learning model to obtain the geometric features of the target obstacle. The point cloud deep learning model is a model specifically used to process point clouds.

Optionally, the point cloud deep learning model includes a first feature extraction layer, and the autonomous driving vehicle performs a first feature extraction on the point cloud data of the target obstacle through the first feature extraction layer to obtain the geometric features of the target obstacle.

303. The autonomous driving vehicle determines a candidate orientation set of the target obstacle based on the geometric features of the target obstacle, where the candidate orientation set includes multiple candidate orientation information indicating orientations of multiple faces of the target obstacle in the coordinate system of the autonomous driving vehicle.

In the embodiment of the present application, when the autonomous driving vehicle determines the candidate orientation set of the target obstacle based on the geometric features of the target obstacle, the orientation information of the target obstacle can be determined based on the geometric features of the target obstacle, and the orientation information of multiple faces of the target obstacle can be expanded based on the orientation information of the target obstacle; or the orientation information of multiple faces of the target obstacle can be directly determined. The embodiment of the present application does not limit the method of determining the candidate orientation set.

In a first possible implementation manner, the autonomous driving vehicle determines the orientation information of the target obstacle based on the geometric features of the target obstacle, and based on the orientation information of the target obstacle, expands the orientation information of multiple faces of the target obstacle.

Optionally, the orientation information includes an orientation angle. The autonomous driving vehicle determines a candidate orientation set of the target obstacle based on the geometric features of the target obstacle, including: the autonomous driving vehicle determines the orientation angle of the target obstacle based on the geometric features; increases the orientation angle by 90 degrees, 180 degrees, and 270 degrees respectively to obtain a plurality of expanded orientation angles; determines the orientation angle and the plurality of expanded orientation angles as candidate orientation information to obtain the candidate orientation set.

Among them, most of the target obstacles are rectangular parallelepipeds. Since the orientation angle of the target obstacle is the orientation angle of the front side of the target obstacle, after determining the orientation angle of the target obstacle, the orientation angle can be increased by 90 degrees, 180 degrees and 270 degrees respectively to obtain the orientation angles of other faces.

Optionally, the autonomous driving vehicle processes the geometric features of the target obstacle through a point cloud deep learning model to obtain a candidate orientation set of the target obstacle. In some embodiments, the autonomous driving vehicle determines a candidate orientation set of the target obstacle based on the geometric features of the target obstacle, including: the autonomous driving vehicle processes the geometric features of the target obstacle through a point cloud deep learning model to obtain the orientation angle of the target obstacle; increases the orientation angle by 90 degrees, 180 degrees, and 270 degrees respectively to obtain multiple expanded orientation angles; determines the orientation angle and the multiple expanded orientation angles as candidate orientation information to obtain the candidate orientation set.

In some embodiments, the point cloud deep learning model includes a direction determination layer, and the autonomous driving vehicle processes the geometric features of the target obstacle through the point cloud deep learning model to obtain the direction angle of the target obstacle, including: the autonomous driving vehicle determines the direction The geometric features of the target obstacle are processed in the first layer to obtain the orientation angle of the target obstacle.

Among them, the point cloud deep learning model can be obtained through sample data training, and the sample data includes sample point cloud data and sample orientation information of sample obstacles. The point cloud deep learning model is trained through the sample data, so that the point cloud deep learning model processes the sample point cloud data, and the error between the obtained orientation information and the sample orientation information converges.

In a second possible implementation, the autonomous driving vehicle directly determines the orientation information of multiple faces of the target obstacle based on the geometric features of the target obstacle. The autonomous driving vehicle determines the candidate orientation set of the target obstacle based on the geometric features of the target obstacle, including: the autonomous driving vehicle determines the orientation information of multiple faces of the target obstacle based on the geometric features; and determines the orientation information of the multiple faces as the candidate orientation set of the target obstacle.

Optionally, the autonomous driving vehicle processes the geometric features of the target obstacle through a point cloud deep learning model to obtain a candidate orientation set of the target obstacle. In some embodiments, the autonomous driving vehicle determines a candidate orientation set of the target obstacle based on the geometric features of the target obstacle, including: the autonomous driving vehicle processes the geometric features of the target obstacle through a point cloud deep learning model to obtain orientation information of multiple faces of the target obstacle.

In some embodiments, the point cloud deep learning model includes an orientation determination layer, and the autonomous driving vehicle processes the geometric features of the target obstacle through the point cloud deep learning model to obtain the orientation information of multiple faces of the target obstacle, including: the autonomous driving vehicle processes the geometric features of the target obstacle through the orientation determination layer to obtain the orientation information of multiple faces of the target obstacle.

Among them, the point cloud deep learning model can be obtained through training with sample data, and the sample data includes sample point cloud data of sample obstacles and sample orientation information of multiple faces of the sample obstacles. The point cloud deep learning model is trained with the sample data, so that the point cloud deep learning model processes the sample point cloud data, and the error between the orientation information of multiple faces obtained and the sample orientation information of the multiple faces converges.

In a possible implementation, the autonomous driving vehicle obtains multiple frames of point cloud data for the target obstacle during driving, and the autonomous driving vehicle determines the orientation information of the target obstacle based on the multiple frames of point cloud data. Optionally, in order to make the obtained geometric features more complete, the autonomous driving vehicle can fuse the geometric features of the target obstacle in the multiple frames of point cloud data. In some embodiments, the point cloud data is multiple frames of point cloud data, and the multiple frames of point cloud data are image data obtained by the autonomous driving vehicle shooting the target obstacle at different times. The autonomous driving vehicle extracts the first feature of the point cloud data of the target obstacle to obtain the geometric features of the target obstacle, including: extracting the first feature of each frame of point cloud data in the multiple frames of point cloud data of the target obstacle to obtain the first geometric feature corresponding to each frame of point cloud data; and performing the following operations on each frame of point cloud data in sequence according to the shooting order of the multiple frames of point cloud data: fusing the first geometric feature of the current frame of point cloud data with the first geometric feature of the previous frame of point cloud data to obtain the second geometric feature of the current frame of point cloud data. The autonomous driving vehicle determines the candidate orientation set of the target obstacle based on the geometric features of the target obstacle, including: the autonomous driving vehicle determines the candidate orientation set of the target obstacle based on the second geometric feature of the last frame of point cloud data.

304. The autonomous driving vehicle performs a second feature extraction on the image data of the target obstacle to obtain an apparent feature of the target obstacle; based on the apparent feature of the target obstacle, first orientation information of the target obstacle is determined, where the first orientation information indicates an orientation of the target obstacle in the observer coordinate system.

In the embodiment of the present application, the image data of the target obstacle is obtained by shooting the autonomous driving vehicle, so the autonomous driving vehicle is the observer of the target obstacle. The observer coordinate system takes the first ray pointed by the autonomous driving vehicle to the target obstacle as the vertical axis and the second ray passing through the center of the target obstacle and perpendicular to the first ray as the horizontal axis.

In the embodiment of the present application, the autonomous driving vehicle may use any algorithm to extract the second feature of the image data of the target obstacle to obtain the apparent feature of the target obstacle, and determine the first orientation information of the target obstacle based on the apparent feature of the target obstacle. The embodiment of the present application is not limited to this.

In a possible implementation, the image data is a plurality of frames of image data, and the plurality of frames of image data is image data obtained by the autonomous driving vehicle shooting the target obstacle at different times. The autonomous driving vehicle performs a second feature extraction on the image data of the target obstacle to obtain the apparent features of the target obstacle, including: performing a second feature extraction on each frame of image data in the plurality of frames of image data of the target obstacle to obtain the first apparent features corresponding to each frame of image data; and performing the following steps on each frame of image data in sequence according to the shooting order of the plurality of frames of image data: fusing the first apparent features of the current frame of image data with the first apparent features of the previous frame of image data to obtain the second apparent features of the current frame of image data. The autonomous driving vehicle determines the first orientation information of the target obstacle based on the apparent features of the target obstacle, including: determining the first orientation information of the target obstacle based on the second apparent features of the last frame of image data.

In order to avoid the appearance features of the target obstacle in one frame of image data being not rich enough, the embodiment of the present application can refer to the appearance features in multiple frames of image data to determine the orientation information of the target obstacle, so that the determined orientation information is more accurate.

In some embodiments, the first orientation information is a first orientation category; the apparent characteristics of the target obstacle and the first orientation category are determined by an orientation classification model, which is used to determine an orientation category that matches the apparent characteristics of the obstacle from a target number of orientation categories.

The target number can be any number, and the embodiment of the present application does not limit the target number. In some embodiments, the target number is 8. The target number of orientation categories includes front, right front, right, right rear, rear, left rear, left and left front; or, the target number of orientation categories includes 0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees and 315 degrees.

For example, the embodiment of the present application is modeled with a pinhole imaging model, with the observer coordinate system as the reference coordinate system, and the obstacle orientation is divided into eight regions: front, right front, right, right rear, rear, left rear, left, and left front, corresponding to 0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees, and 315 degrees, respectively. The labeling personnel label the sample image data according to the standard, accumulate the sample data, and train the orientation classification model.

In some embodiments, as shown in FIG4 , the orientation classification model includes a feature extraction layer, a fusion layer, and an orientation classification layer. The autonomous driving vehicle extracts features from multiple frames of image data of the target obstacle through the feature extraction layer to obtain the appearance features corresponding to each frame of image data; through the fusion layer, in accordance with the shooting order of the multiple frames of image data, each frame of image data is sequentially executed: the first appearance feature of the current frame of image data is fused with the first appearance feature of the previous frame of image data to obtain the second appearance feature of the current frame of image data; through the orientation classification layer, the second appearance feature of the last frame of image data is classified to obtain the first orientation category of the target obstacle.

305. The autonomous driving vehicle determines an angle between the autonomous driving vehicle coordinate system and the observer coordinate system.

Among them, the autonomous driving vehicle coordinate system takes the center of the autonomous driving vehicle as the origin, the longitudinal direction of the autonomous driving vehicle as the longitudinal axis, and the lateral direction of the autonomous driving vehicle as the horizontal axis. The observer coordinate system takes the center of the target obstacle as the origin, the first ray pointed by the autonomous driving vehicle to the target obstacle as the longitudinal axis, and the second ray passing through the center of the target obstacle and perpendicular to the first ray as the horizontal axis. Therefore, based on the position of the center point of the autonomous driving vehicle and the position of the center point of the target obstacle, the angle between the autonomous driving vehicle coordinate system and the observer coordinate system can be determined. The sine function value of the angle is the ratio of the lateral distance between the autonomous driving vehicle and the target obstacle to the distance between the autonomous driving vehicle and the target obstacle. The lateral distance between the autonomous driving vehicle and the target obstacle refers to the distance from the center point of the target obstacle to the autonomous driving vehicle. Therefore, the autonomous driving vehicle determines the angle between the autonomous driving vehicle coordinate system and the observer coordinate system, including: obtaining the position of the center point of the autonomous driving vehicle and the position of the center point of the target obstacle; based on the position of the center point of the autonomous driving vehicle and the position of the center point of the target obstacle, determining the lateral distance between the center point of the autonomous driving vehicle and the center point of the target obstacle, and also determining the distance between the center point of the autonomous driving vehicle and the center point of the target obstacle; obtaining the ratio of the distance and the lateral distance, and determining the angle between the autonomous driving vehicle coordinate system and the observer coordinate system based on the sine function and the ratio.

306. Based on the angle, the autonomous driving vehicle determines fourth orientation information corresponding to the first orientation information in the autonomous driving vehicle coordinate system, where the fourth orientation information indicates the orientation of the target obstacle in the autonomous driving vehicle coordinate system.

Since the first orientation information and the candidate orientation information belong to different coordinate systems, the first orientation information can be mapped to the autonomous driving vehicle coordinate system to obtain the fourth orientation information. In this way, the fourth orientation information determined based on the image data and the candidate orientation information determined based on the point cloud data are in the same coordinate system.

307. The autonomous driving vehicle determines, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the fourth orientation information as the orientation information of the target obstacle.

The indicated orientation matches the orientation indicated by the fourth orientation information, which means that the indicated orientation is similar to or close to the orientation indicated by the fourth orientation information. In some embodiments, the autonomous driving vehicle determines, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the fourth orientation information as the orientation information of the target obstacle, including: determining, from the candidate orientation set, second orientation information whose indicated orientation has the smallest difference with the orientation indicated by the fourth orientation information as the orientation information of the target obstacle.

For example, Figure 5 is an obstacle whose direction needs to be determined. The process of determining the direction of the obstacle can be shown in Figure 6. Based on the point cloud data of the obstacle, the direction angle of the obstacle in the autonomous driving vehicle coordinate system is determined to be α+β, and the direction angle is expanded to obtain a candidate direction set {α+β, α+β+90°, α+β+180°, α+β+270°}; based on the image data of the obstacle, the direction angle of the obstacle in the observer coordinate system is determined to be γ; the angle between the autonomous driving vehicle coordinate system and the observer coordinate system is determined to be β, and the direction angle with the smallest difference from the γ+β angle is determined from the candidate direction set as the direction angle of the autonomous driving vehicle.

It should be noted that the embodiment of the present application is only illustrative of an example in which multiple candidate orientation information indicates the orientation of the target obstacle in the autonomous driving vehicle coordinate system, and the first orientation information indicates the orientation of the target obstacle in the observer coordinate system. In another embodiment, multiple candidate orientation information indicates the orientation of the target obstacle in the world coordinate system, and the first orientation information indicates the orientation of the target obstacle in the world coordinate system. The embodiment of the present application does not limit the coordinate system.

Another point that needs to be explained is that the method provided in the embodiment of the present application can reduce the error rate of vehicle orientation recognition by 20% and increase the accuracy of pedestrian orientation recognition by 50% compared with determining orientation information based only on point cloud data.

FIG. 7 is a schematic diagram of the structure of a device for determining orientation information provided in an embodiment of the present application. Referring to FIG. 7 , the device includes:

An acquisition module 701 is used to acquire point cloud data and image data captured by the autonomous driving vehicle for a target obstacle;

A feature extraction module 702 is used to perform a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle, and perform a second feature extraction on the image data of the target obstacle to obtain an apparent feature of the target obstacle;

A first determination module 703 is configured to determine a candidate orientation set of the target obstacle based on the geometric features of the target obstacle, wherein the candidate orientation set includes a plurality of candidate orientation information, and the plurality of candidate orientation information indicates orientations of a plurality of faces of the target obstacle;

A second determination module 704, configured to determine first orientation information of the target obstacle based on the apparent characteristics of the target obstacle;

The third determination module 705 is configured to determine, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the first orientation information as the orientation information of the target obstacle.

In one possible implementation, the orientation information includes an orientation angle; the first determination module 703 is used to determine the orientation angle of the target obstacle based on the geometric features; the orientation angle is increased by 90 degrees, 180 degrees and 270 degrees respectively to obtain multiple expanded orientation angles; the orientation angle and the multiple expanded orientation angles are determined as candidate orientation information to obtain the candidate orientation set.

In a possible implementation, the first determination module 703 is configured to determine orientation information of multiple faces of the target obstacle based on the geometric features; and determine the orientation information of the multiple faces as a candidate orientation set of the target obstacle.

The feature extraction module 702 is used to extract the third feature of each frame of image data in the multiple frames of image data of the target obstacle to obtain the first appearance feature corresponding to each frame of image data; and perform the following steps on each frame of image data in sequence according to the shooting order of the multiple frames of image data: fusing the first appearance feature of the current frame of image data with the first appearance feature of the previous frame of image data to obtain the second appearance feature of the current frame of image data;

The second determination module 704 is configured to determine first orientation information of the target obstacle based on a second appearance feature of the last frame of image data.

In a possible implementation, the third determination module 705 is used to determine the angle between the autonomous driving vehicle coordinate system and the observer coordinate system; based on the angle, determine fourth orientation information corresponding to the first orientation information in the autonomous driving vehicle coordinate system, the fourth orientation information indicating the orientation of the target obstacle in the autonomous driving vehicle coordinate system; determine second orientation information from the candidate orientation set whose indicated orientation matches the orientation indicated by the fourth orientation information as the target obstacle The direction information of the marked obstacle.

In a possible implementation, the third determination module 705 is configured to determine, from the candidate orientation set, second orientation information whose indicated orientation has the smallest difference from the orientation indicated by the fourth orientation information as the orientation information of the target obstacle.

It should be noted that: the orientation information determination device provided in the above embodiment only uses the division of the above functional modules as an example when calibrating. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the autonomous driving vehicle is divided into different functional modules to complete all or part of the functions described above. In addition, the orientation information determination device provided in the above embodiment and the orientation information determination method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

FIG8 shows a block diagram of an autonomous driving vehicle 800 provided by an exemplary embodiment of the present application. The autonomous driving vehicle 800 includes: a processor 801 and a memory 802 .

The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code, which is executed by the processor 801 to implement the orientation information determination method provided in the method embodiment of the present application.

In some embodiments, the autonomous driving vehicle 800 may also optionally include: a peripheral device interface 803 and at least one peripheral device. The processor 801, the memory 802 and the peripheral device interface 803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808 and a power supply 809.

The peripheral device interface 803 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The RF circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with the communication network and other communication devices through electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, The RF circuit 804 may include a device, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 804 may communicate with other autonomous driving vehicles through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and/or a WiFi (Wireless Fidelity) network. In some embodiments, the RF circuit 804 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

The display screen 805 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 805 is a touch display screen, the display screen 805 also has the ability to collect touch signals on the surface or above the surface of the display screen 805. The touch signal may be input as a control signal to the processor 801 for processing. At this time, the display screen 805 may also be used to provide virtual buttons and/or virtual keyboards, also known as soft buttons and/or soft keyboards. In some embodiments, the display screen 805 may be one, and the front panel of the autonomous driving vehicle 800 may be set; in other embodiments, the display screen 805 may be at least two, and they may be set on different surfaces of the autonomous driving vehicle 800 or in a folding design; in still other embodiments, the display screen 805 may be a flexible display screen, and may be set on a curved surface or a folding surface of the autonomous driving vehicle 800. Even more, the display screen 805 may be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 805 may be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. The front camera is set on the front panel of the autonomous driving vehicle, and the rear camera is set on the back of the autonomous driving vehicle. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 801 for processing, or input them into the RF circuit 804 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the autonomous driving vehicle 800. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 801 or the RF circuit 804 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

The positioning component 808 is used to locate the current geographic location of the autonomous driving vehicle 800 to achieve navigation or LBS (Location Based Service). The positioning component 808 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system, or the European Union's Galileo system.

The power supply 809 is used to power various components in the autonomous driving vehicle 800. The power supply 809 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 809 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

In some embodiments, the autonomous driving vehicle 800 further includes one or more sensors 810. The one or more sensors 810 Including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 and a proximity sensor 816 .

The acceleration sensor 811 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the autonomous driving vehicle 800. For example, the acceleration sensor 811 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 811. The acceleration sensor 811 can also be used to collect motion data of games or users.

The gyro sensor 812 can detect the body direction and rotation angle of the autonomous driving vehicle 800, and the gyro sensor 812 can cooperate with the acceleration sensor 811 to collect the user's 3D actions on the autonomous driving vehicle 800. The processor 801 can implement the following functions based on the data collected by the gyro sensor 812: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

The pressure sensor 813 can be set on the side frame of the autonomous driving vehicle 800 and/or the lower layer of the display screen 805. When the pressure sensor 813 is set on the side frame of the autonomous driving vehicle 800, it can detect the user's grip signal of the autonomous driving vehicle 800, and the processor 801 performs left and right hand recognition or quick operation according to the grip signal collected by the pressure sensor 813. When the pressure sensor 813 is set on the lower layer of the display screen 805, the processor 801 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 805. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

The fingerprint sensor 814 is used to collect the user's fingerprint, and the processor 801 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 814, or the fingerprint sensor 814 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 801 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 814 can be set on the front, back, or side of the autonomous driving vehicle 800. When a physical button or manufacturer logo is set on the autonomous driving vehicle 800, the fingerprint sensor 814 can be integrated with the physical button or manufacturer logo.

The optical sensor 815 is used to collect the ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 according to the ambient light intensity collected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is reduced. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 according to the ambient light intensity collected by the optical sensor 815.

The proximity sensor 816, also called a distance sensor, is disposed on the front panel of the autonomous driving vehicle 800. The proximity sensor 816 is used to collect the distance between the user and the front of the autonomous driving vehicle 800. In one embodiment, when the proximity sensor 816 detects that the distance between the user and the front of the autonomous driving vehicle 800 is gradually decreasing, the processor 801 controls the display screen 805 to switch from the screen-on state to the screen-off state; when the proximity sensor 816 detects that the distance between the user and the front of the autonomous driving vehicle 800 is gradually increasing, the processor 801 controls the display screen 805 to switch from the screen-off state to the screen-on state.

Those skilled in the art will appreciate that the structure shown in FIG. 8 does not constitute a limitation on the autonomous driving vehicle 800 , and may include more or fewer components than shown, or combine certain components, or adopt a different component arrangement.

FIG9 is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more processors 901, such as CPUs (Central Processing Units) and one or more memories 902, wherein the memory 902 stores at least one program code. At least one program code is loaded and executed by the processor 901 to implement the methods provided by the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

The server 900 is used to execute the steps executed by the server in the above method embodiment.

In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a program code, and the program code can be executed by a processor in a computer device to complete the orientation information determination method in the above embodiment. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

In an exemplary embodiment, a computer program or a computer program product is further provided. The computer program or the computer program product includes a computer program code. When the computer program code is executed by a computer, the computer implements the method for determining the orientation information in the above embodiment.

A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

A method for determining orientation information, wherein the method comprises:

Obtain point cloud data and image data taken by the autonomous driving vehicle for target obstacles;

Performing a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle, and performing a second feature extraction on the image data of the target obstacle to obtain an apparent feature of the target obstacle;

Determine, based on the geometric features of the target obstacle, a candidate orientation set of the target obstacle, wherein the candidate orientation set includes a plurality of candidate orientation information, and the plurality of candidate orientation information indicates orientations of a plurality of faces of the target obstacle;

Determining first orientation information of the target obstacle based on the apparent characteristics of the target obstacle;

Second orientation information whose indicated orientation matches the orientation indicated by the first orientation information is determined from the candidate orientation set as the orientation information of the target obstacle.
The method according to claim 1, wherein the orientation information includes an orientation angle; and determining a candidate orientation set of the target obstacle based on the geometric features of the target obstacle comprises:

Based on the geometric features, determining the orientation angle of the target obstacle;

The orientation angle is increased by 90 degrees, 180 degrees and 270 degrees respectively to obtain a plurality of expanded orientation angles;

The orientation angle and the multiple expanded orientation angles are determined as candidate orientation information to obtain the candidate orientation set.
The method according to claim 1, wherein determining the candidate orientation set of the target obstacle based on the geometric features of the target obstacle comprises:

Based on the geometric features, determining orientation information of multiple faces of the target obstacle;

The orientation information of the multiple faces is determined as a candidate orientation set of the target obstacle.
The method according to claim 1, wherein the image data is multi-frame image data, and the multi-frame image data is image data obtained by the autonomous driving vehicle photographing the target obstacle at different times; and the performing a second feature extraction on the image data of the target obstacle to obtain the apparent features of the target obstacle comprises:

Performing second feature extraction on each frame of image data in the multiple frames of image data of the target obstacle to obtain a first appearance feature corresponding to each frame of image data;

According to the shooting order of the multiple frames of image data, performing the following steps on each frame of image data in sequence: fusing the first appearance feature of the current frame of image data with the first appearance feature of the previous frame of image data to obtain the second appearance feature of the current frame of image data;

The determining, based on the apparent feature of the target obstacle, the first orientation information of the target obstacle includes:

Based on the second appearance feature of the last frame of image data, first orientation information of the target obstacle is determined.
The method according to claim 1 or 4, wherein the first orientation information is a first orientation category; the apparent characteristics of the target obstacle and the first orientation category are determined by an orientation classification model, and the orientation classification model is used to determine an orientation category that matches the apparent characteristics of the obstacle from a target number of orientation categories.
The method according to claim 5, wherein the target number of orientation categories includes front, right front, right, right rear, rear, left rear, left and left front; or, the target number of orientation categories includes 0 degrees, 45 degrees, 90 degrees, 135 degrees, 180 degrees, 225 degrees, 270 degrees and 315 degrees.
The method according to claim 1, wherein the multiple candidate orientation information indicates the orientation of the target obstacle in the autonomous driving vehicle coordinate system, the first orientation information indicates the orientation of the target obstacle in the observer coordinate system, the observer coordinate system has a first ray pointing from the autonomous driving vehicle to the target obstacle as the vertical axis, and a second ray passing through the center of the target obstacle and perpendicular to the first ray as the horizontal axis, and the autonomous driving vehicle coordinate system has the center of the autonomous driving vehicle as the origin, the longitudinal direction of the autonomous driving vehicle as the vertical axis, and the lateral direction of the autonomous driving vehicle as the horizontal axis.
The method according to claim 7, wherein determining, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the first orientation information as the orientation information of the target obstacle comprises:

Determining an angle between the autonomous driving vehicle coordinate system and the observer coordinate system;

Based on the angle, determining fourth orientation information corresponding to the first orientation information in the autonomous driving vehicle coordinate system, the fourth orientation information indicating the orientation of the target obstacle in the autonomous driving vehicle coordinate system;

Second orientation information whose indicated orientation matches the orientation indicated by the fourth orientation information is determined from the candidate orientation set as the orientation information of the target obstacle.
The method according to claim 8, wherein determining, from the candidate orientation set, second orientation information whose indicated orientation matches the orientation indicated by the fourth orientation information as the orientation information of the target obstacle comprises:

Determine, from the candidate orientation set, second orientation information whose indicated orientation has the smallest difference from the orientation indicated by the fourth orientation information as the orientation information of the target obstacle.
The method according to claim 1, wherein the point cloud data is multi-frame point cloud data, and the multi-frame point cloud data is image data obtained by the autonomous driving vehicle photographing the target obstacle at different times.

Performing a first feature extraction on the point cloud data of the target obstacle to obtain a geometric feature of the target obstacle includes:

Performing a first feature extraction on each frame of point cloud data in the multiple frames of point cloud data of the target obstacle to obtain a first geometric feature corresponding to each frame of point cloud data;

According to the shooting order of the multiple frames of point cloud data, the following are performed on each frame of point cloud data in sequence: fusing the first geometric feature of the current frame of point cloud data with the first geometric feature of the previous frame of point cloud data to obtain the second geometric feature of the current frame of point cloud data;

Determining a candidate orientation set of the target obstacle based on the geometric features of the target obstacle includes:

Based on the second geometric feature of the last frame of point cloud data, a candidate orientation set of the target obstacle is determined.
An autonomous driving vehicle, wherein the autonomous driving vehicle comprises one or more processors and one or more memories, wherein the one or more memories store at least one program code, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the orientation information determination method as described in any one of claims 1 to 10.
A computer-readable storage medium, wherein at least one program code is stored in the storage medium, and the at least one program code is loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 10.
A computer program or a computer program product, wherein the computer program or the computer program product comprises: a computer program code, and when the computer program code is executed by a computer, the computer implements the operations performed by the method according to any one of claims 1 to 10.