WO2022012094A1

WO2022012094A1 - Driving scene reconstruction method and apparatus, system, vehicle, device, and storage medium

Info

Publication number: WO2022012094A1
Application number: PCT/CN2021/085226
Authority: WO
Inventors: 丁磊; 朱兰芹; 何磊; 胡健
Original assignee: 华人运通(上海)自动驾驶科技有限公司
Priority date: 2020-07-16
Filing date: 2021-04-02
Publication date: 2022-01-20
Also published as: CN111880533B; CN111880533A

Abstract

A driving scene reconstruction method and apparatus, a system, a vehicle, an electronic device, and a computer readable storage medium. A driving scene reconstruction method, comprising: acquiring road traffic condition information in sensor information, Internet of Vehicles information, and map information (S101); integrating the road traffic condition information in the sensor information, Internet of Vehicles information, and map information (S102); and, on the basis of the integrated road traffic condition information, reconstructing the driving scenario of a self-driving vehicle (11) (S103). The present driving scenario reconstruction method fuses sensor information, Internet of Vehicles information, and map information, enriching the information sources of the environment around a self-driving vehicle and incorporating road traffic condition information so that the reconstructed driving scenario is closer to the real driving environment, providing the driver with a more effective assistance function and enhancing driving safety.

Description

Driving scene reconstruction method, device, system, vehicle, equipment and storage medium

This application claims the priority of the Chinese patent application filed on July 16, 2020 with the application number 202010685995.5 and the invention titled "Driving Scene Reconstruction Method, Device, System, Vehicle, Equipment and Storage Medium". The entire contents of this application are incorporated by reference.

technical field

The present application relates to the technical field of automatic driving, and in particular, to a driving scene reconstruction method, apparatus, system, vehicle, electronic device, and computer-readable storage medium.

Background technique

Self-driving car, also known as driverless car, computer-driven car or wheeled mobile robot, is a kind of intelligent car that realizes unmanned driving through computer system. As the level of autonomous driving becomes higher and higher, the responsibility of the driver is gradually transformed into that of the monitor. The driving scene reconstruction system can better provide the driver with the surrounding situation of the vehicle, so that the driver can clearly understand the surrounding situation of the vehicle in a relaxed state. At the same time, in the non-autonomous driving mode, the sensors and other components used in autonomous driving work normally, which can realize scene reconstruction and provide driving assistance for the driver.

At present, the driving scene information provided by the driving scene reconstruction system is limited, which reduces the automatic driving experience and has hidden dangers of automatic driving.

SUMMARY OF THE INVENTION

The embodiments of the present application provide a method, device, system, vehicle, electronic device, and computer-readable storage medium for automatic driving reconstruction, so as to solve the problems existing in the related art, and the technical solutions are as follows:

In a first aspect, an embodiment of the present application provides a driving scene reconstruction method, including:

Obtain road traffic information from sensor information, vehicle networking information and map information;

integrating the sensor information, the vehicle networking information and the road traffic situation information in the map information;

Based on the integrated road traffic situation information, the driving scene of the self-vehicle is reconstructed.

In a second aspect, an embodiment of the present application provides a driving scene reconstruction device, including:

The acquisition module is used to acquire the road traffic situation information in the sensor information, the Internet of Vehicles information and the map information;

an integration module, configured to integrate the sensor information, the vehicle networking information and the road traffic situation information in the map information;

The reconstruction module is used to reconstruct the driving scene of the self-vehicle based on the integrated road traffic situation information.

In a third aspect, an embodiment of the present application provides a driving scene reconstruction system, including the driving scene reconstruction device described above, and the system further includes:

a sensor, connected to the driving scene reconstruction device, for collecting and outputting sensor information to the driving scene reconstruction device;

an Internet of Vehicles device, connected to the driving scene reconstruction device, and used for outputting the Internet of Vehicles information to the driving scene reconstruction device;

a map device, connected to the driving scene reconstruction device, for outputting map information to the driving scene reconstruction device;

The display device is connected to the driving scene reconstruction device, and is used for receiving data of the reconstructed driving scene from the driving scene reconstruction device, and displaying the reconstructed driving scene.

In a fourth aspect, an embodiment of the present application provides a vehicle, which includes the above-mentioned driving scene reconstruction device, or includes the above-mentioned driving scene reconstruction system.

In a fifth aspect, an embodiment of the present application provides an electronic device, the electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor , so that at least one processor can execute the above driving scene reconstruction method.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the method in any one of the implementation manners of the above aspects is executed.

The advantages or beneficial effects of the above technical solutions include at least:

The driving scene reconstruction method integrates sensor information, Internet of Vehicles information and map information, enriches the information sources of the surrounding environment of the autonomous driving vehicle, and combines road traffic situation information to make the reconstructed driving scene closer to the real one. The driving environment can provide more effective assistance for the driver and improve driving safety.

The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will become apparent by reference to the drawings and the following detailed description.

Description of drawings

In the drawings, unless stated otherwise, the same reference numbers refer to the same or like parts or elements throughout the several figures. The drawings are not necessarily to scale. It should be understood that these drawings depict only some embodiments disclosed in accordance with the present application and should not be considered as limiting the scope of the present application.

FIG. 1 is a schematic block diagram of a driving scene reconstruction system according to an exemplary embodiment;

Fig. 2 shows the processing procedure of the controller in Fig. 1;

Figure 3 shows the types of information handled by the processor;

4 is a schematic flowchart of a driving scene reconstruction method according to an embodiment of the present application;

5 is a schematic diagram of road traffic situation information in a driving scene reconstruction method according to an embodiment of the present application;

6 is a schematic diagram of target information in a driving scene reconstruction method according to an embodiment of the present application;

7 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to an embodiment of the present application;

8 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application;

9 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application;

10 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application;

11 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application;

12 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application;

13 is a structural block diagram of a driving scene reconstruction device according to an embodiment of the present application;

14 is a structural block diagram of an acquisition module of a driving scene reconstruction device according to an embodiment of the present application;

15 is a structural block diagram of an integration module of a driving scene reconstruction device according to an embodiment of the present application;

16 is a structural block diagram of a driving scene reconstruction system according to an embodiment of the present application;

FIG. 17 is a structural block diagram of an electronic device according to an embodiment of the present application.

detailed description

In the following, only certain exemplary embodiments are briefly described. As those skilled in the art would realize, the described embodiments may be modified in various different ways, all without departing from the spirit or scope of the present application. Accordingly, the drawings and description are to be regarded as illustrative in nature and not restrictive.

Advanced Driver Assist System (ADAS) sensors can collect environmental data inside and outside the car through various on-board sensors, and perform technical processing such as identification, detection and tracking of static and dynamic objects, so that drivers can The fastest time to detect possible dangers and take corresponding measures to improve driving safety.

Sensors include, but are not limited to, one or more image acquisition devices (eg, cameras), inertial measurement units, radars, and the like. The image acquisition device can be used to collect target information, road marking information and lane line information of the surrounding environment of the autonomous vehicle. The inertial measurement unit can sense position and orientation changes of the autonomous vehicle based on inertial acceleration. Radar can use radio signals to sense objects, road signs, within the local environment of an autonomous vehicle. In addition to sensing the target, the radar unit may additionally sense the speed and/or heading of the target. The image capture device may include one or more devices for capturing images of the environment surrounding the autonomous vehicle. For example, the image capture device may be a still camera and/or a video camera. The cameras may include infrared cameras. The camera may be moved mechanically, eg by mounting the camera on a rotating and/or tilting platform.

Sensors may also include, for example, sonar sensors, infrared sensors, steering sensors, accelerator sensors, brake sensors, audio sensors (eg, microphones), and the like. Audio sensors can be configured to pick up sound from the environment surrounding the autonomous vehicle. The steering sensor may be configured to sense the steering angle of the steering wheel, the wheels of the vehicle, or a combination thereof. The accelerator sensor and the brake sensor sense the accelerator position and the brake position of the vehicle, respectively. In some cases, the accelerator sensor and brake sensor may be integrated into an integrated accelerator/brake sensor.

V2X (vehicle to everything, the Internet of Vehicles, or the Internet of Vehicles) is the exchange of information between vehicles and the outside world. V2X communication is an important key technology for realizing environmental perception, information interaction and collaborative control in the Internet of Vehicles. It adopts various communication technologies to realize vehicle-to-vehicle (Vehicle-To-Vehicle, referred to as V2V), vehicle-to-road (Vehicle-To-Infrastructure, referred to as V2I) and vehicle-to-person (Vehicle-To-Person, referred to as V2P) interconnection It can effectively use information such as extraction and sharing on the information network platform to effectively manage and control vehicles and provide comprehensive services. In this way, a series of road traffic information such as real-time road conditions, road sign information, lane line information, and target information can be obtained, thereby improving driving safety, reducing congestion, improving traffic efficiency, and providing in-vehicle entertainment information.

During the automatic driving scene reconstruction process in the embodiment of the present application, not only sensor information from ADAS sensors, but also vehicle networking information from V2X can be obtained, and the finally obtained surrounding environment information is rich and can reflect the real driving environment , thereby improving the safety of autonomous driving and enhancing the autonomous driving experience.

FIG. 1 is a schematic block diagram of a driving scene reconstruction system according to an exemplary embodiment. As shown in Figure 1, after the sensor, V2X and map device collect the information around the vehicle, the controller can receive the raw data from the sensor, and the controller can also receive the road information (also called lane line information), traffic sign information from the V2X (also referred to as road sign information), surrounding vehicles or/and pedestrians and other traffic participants (also referred to as target information), etc., the controller can also receive road information (also referred to as lane line information) from the map device, traffic Sign information (also referred to as road sign information), own vehicle position information, navigation information (also referred to as navigation route planning information), and the like. After the controller receives information from sensor information, vehicle networking information and map information, it obtains road traffic condition information from the received information, integrates the road traffic condition information, and integrates the integrated road traffic condition information such as lane line information. , traffic sign information (also known as road sign information), target information (including target type, direction, location, alarm, etc.) and the vehicle's trajectory, etc. are transmitted to the instrument controller. The instrument controller processes the received information into driving scene data, and transmits the driving scene data to the instrument display device to display the reconstructed driving scene.

In one embodiment, the road traffic condition information may include road sign information, lane line information, road traffic abnormal condition information, congestion condition information, road traffic scene information, navigation path planning information, target information, and the like. The targets include traffic participants such as vehicles and pedestrians around the ego vehicle.

FIG. 2 shows the processing procedure of the controller in FIG. 1 , and FIG. 3 shows the types of information processed by the controller. As shown in FIG. 2 , the sensors may include, for example, radars and image acquisition devices, among others. Radar can use radio signals to sense target information and road marking information in the surrounding environment of autonomous vehicles, and generate point cloud data such as point cloud maps. Image capture devices such as cameras can be used to capture road signs, lane lines, and objects in the surrounding environment to generate video streams. As shown in FIG. 2 , for example, the controller may include a classification processing module and an information fusion module (also referred to as an integration module). The classification processing module may include a target information identification module, a traffic information identification module (also referred to as a road traffic condition information identification module), and a function alarm module. After the controller receives the point cloud image from the radar, the video stream of the camera, the information of V2X and the information of the map device, the target information identification module identifies the target information from the information received by the controller, and transmits the target information to the information fusion module; The traffic information identification module identifies the road traffic condition information from the information received by the controller, and transmits the road traffic condition information to the information fusion module; the function alarm module identifies the function alarm information from the information received by the controller, and sends the function alarm information to the information fusion module. The alarm information is transmitted to the information fusion module (also called the integration module). The information fusion module integrates the received target information, road traffic condition information, and function alarm information respectively, and outputs the integrated target information, road traffic condition information, and function alarm information.

As shown in FIG. 3 , for example, the target identification module can identify information such as the type, coordinates, and orientation of the target. The traffic information recognition module can identify the location and current state of traffic lights, the value and location of speed limit signs, the type and coordinates of lane lines, road traffic scenarios, road traffic anomalies, and optimized road path planning. The function alarm module can include automatic driving level, forward collision warning, emergency braking warning, intersection collision warning, blind spot warning, lane change warning, speed limit warning, lane keeping warning, emergency lane keeping, rear collision warning, rear cross traffic Intersection collision warning, door opening warning, left turn assist, red light running warning, reverse overtaking warning, vehicle loss of control warning, abnormal vehicle warning, vulnerable traffic participant warning, etc.

The information fusion module combines the function alarm information with the target information, the automatic driving level, the lane keeping status and the lane line information, the navigation path planning and the vehicle movement, and outputs the integrated target information and road traffic information. , Function alarm information.

As shown in FIG. 3, exemplarily, the target information may include:

Target status: none, yes, alarm level 1, alarm level 2, alarm level 3, etc.;

Target type: For example, cars, off-road vehicles, SUVs, passenger cars, buses, small trucks, trucks, motorcycles, two-wheelers, adults, children, etc.;

Target orientation: for example, forward, backward, left, right, etc.;

Coordinates: abscissa, ordinate, etc.

As shown in FIG. 3 , for example, the lane line information may include:

Status: None, Yes, Alarm, Adaptive Cruise Control (ACC), Autopilot Level L2, Autopilot Level L3, Autopilot Level L4, etc.;

Type: solid line, dotted line, double yellow line, road edge, road edge, etc.;

Parameters: A0, A1, A2, A3 (lane line equation: y=a0+a1*x+a2*x ² +a3*x ³ ).

Those skilled in the art can understand that A0, A1, A2, and A3 are all polynomial coefficients of the lane line on the left side of the own lane, where A0 represents the lateral distance from the center of the vehicle to the lane line, and a positive value indicates that the lane line is on the left; The heading angle of the vehicle relative to the lane line, a positive value means the lane line is counterclockwise; A2 means the lane line curvature, a positive value means the lane line bends to the left; A3 means the change rate of the lane line curvature, and a positive value means the lane line bends to the left. After the controller obtains the lane line information from the sensor information or V2X information, it can obtain the values of A0, A1, A2, and A3. When the lane line is drawn on the display device, the lane line is drawn according to the lane line equation, so that the lane line consistent with the actual lane line can be displayed in the reconstructed driving scene.

As shown in FIG. 3, exemplarily, the traffic information may include:

Number of lanes: 1, 2, 3, 4, 5, 6, 7, 8, etc.;

The current lane: left 1, left 2, right 1, right 2, etc.;

Path planning direction: none, go straight, turn left, turn right, front left, front right, U-turn, etc.

Speed limit information status: no, yes, alarm

Speed limit value: 5, 10, 15...130;

Traffic light status: none, red light, yellow light, green light, countdown, etc.;

Traffic light coordinates: abscissa, ordinate;

Road traffic scenarios: ramps, intersections, road merging, road bifurcations, T-junctions, etc.

Based on sensor information, Internet of Vehicles information and map information, this application proposes a driving scene reconstruction method based on actual traffic.

FIG. 4 is a schematic flowchart of a driving scene reconstruction method according to an embodiment of the present application. As shown in Figure 1, the driving scene reconstruction method may include:

S101 , acquiring road traffic situation information in sensor information, vehicle networking information and map information;

S102. Integrate the sensor information, the vehicle networking information and the road traffic situation information in the map information;

S103 , reconstruct the driving scene of the self-vehicle based on the integrated road traffic situation information.

The driving scene reconstruction method of the present application integrates sensor information from ADAS sensors, vehicle networking information and map information from V2X, and enriches the information sources of the surrounding environment of autonomous vehicles. Moreover, the driving scene reconstruction method combines Road traffic situation information, thus, the reconstructed driving scene is closer to the real driving environment, more in line with the actual needs of users in the automatic driving state and ordinary driving state, which can provide more effective assistance for the driver and improve the driving experience. safety.

Those skilled in the art can understand that in the driving scene reconstruction method, the sensor information, the Internet of Vehicles information and the road traffic situation information in the map information are acquired in real time, and the sensor information, the Internet of Vehicles information and the map information are The road traffic situation information is integrated; based on the integrated road traffic situation information, the driving scene of the self-vehicle is reconstructed in real time. Therefore, the reconstructed driving scene can display the surrounding environment of the vehicle in real time.

FIG. 5 is a schematic diagram of road traffic situation information in a driving scene reconstruction method according to an embodiment of the present application.

In one embodiment, in S101, acquiring road traffic condition information in the sensor information may include: receiving point cloud data from radar, and analyzing the point cloud data to obtain road sign information.

In one embodiment, in S101, acquiring road traffic condition information in sensor information may include: receiving a video stream from an image acquisition device, and parsing the video stream to obtain road sign information and lane line information.

Exemplarily, as shown in FIG. 5 , the road sign information may include red street light information (also referred to as traffic light information), speed limit sign information, and the like. The traffic light information may include the status of the traffic light, the location of the traffic light, and the like. The speed limit sign information may include the value of the speed limit sign, the position of the speed limit sign, and the like. For example, the value of the speed limit sign on a certain section of an urban road is 50, indicating that the maximum speed limit of this section is 50km/h.

Those skilled in the art can understand that the road sign information is not limited to traffic light information and speed limit sign information, but may also include identifiable signs such as U-turn signs, going straight signs, turn signs and the like.

The lane line information may include at least one of information about the lane where the vehicle is located, information about the lane where surrounding vehicles are located, color information of the lane line, type information of the lane line, and position information of the lane line. For example, the colors of the lane lines include white, yellow, and so on. The types of lane lines include solid lines, dashed lines, double yellow lines, curbs, etc. The position information of the lane line may be the coordinate information of the lane line, which corresponds to the position of the lane line. Those skilled in the art can understand that the lane line can determine the lane, and therefore, the lane line information can reflect the number of lanes, the lane in which the self-vehicle is located, and the lanes in which surrounding vehicles are located.

In one embodiment, in S101, acquiring road traffic situation information in the Internet of Vehicles information may include: acquiring road identification information, lane line information, road traffic abnormality information and/or roadside units from the vehicle-mounted unit and/or the roadside unit At least one of the congestion status information.

In one embodiment, as shown in FIG. 5 , the device for V2X communication may include at least one of an on-board unit (On Board Unit, OBU) and a road side unit (Road Side Unit, RSU). The source of the Internet of Vehicles information is at least one of an on-board unit (OBU) and a roadside unit (RSU). Through V2X communication, at least one of road sign information, lane line information, road traffic abnormality information and congestion condition information can be obtained. The road traffic abnormal situation information may include at least one of road construction information, abnormal vehicle information, and emergency vehicle information. Therefore, through the information of the Internet of Vehicles, the optimal driving route of the self-vehicle can be planned according to the abnormal situation information of road traffic and the information of the congestion situation, so as to reach the destination efficiently.

In one embodiment, in S101, acquiring road traffic situation information in the map information may include: acquiring at least one of road identification information, lane line information, road traffic scene information and navigation path planning information from the map information .

For example, the map information can come from a Beidou navigation system or a GPS navigation system. The map information may include information such as road marking information, lane line information, road traffic scene information, and navigation path planning information. In order to acquire road traffic situation information, at least one of road identification information, lane line information, road traffic scene information and navigation path planning information may be acquired from map information. The road traffic scene information may include road information such as intersection information, road merging information, road bifurcation information, and ramp information. The navigation route planning information includes the travel route information from the origin to the destination. Those skilled in the art can understand that the map information may provide a plurality of travel route information from the origin to the destination, and the navigation route planning information may include a plurality of travel route information from the origin to the destination.

Exemplarily, as shown in FIG. 5 , the road traffic situation information obtained from the map information includes: the lane where the vehicle and surrounding vehicles are located, the type and coordinates of the lane line, the location of the traffic lights, the distance from the vehicle to the ramp, the intersection. The distance to the mouth, etc., navigation path planning information, etc.

There may be duplicate information in sensor information, vehicle networking information, and map information. For example, sensor information, vehicle networking information, and map information all include road sign information. After obtaining road sign information from sensor information, vehicle networking information, and map information, respectively, the road sign information can be integrated, thereby selecting the optimal road sign information for reconstructing the driving scene. In order to obtain the optimal road traffic situation information, exemplarily, as shown in FIG. 5 , in S102, the sensor information, the Internet of Vehicles information and the road traffic situation information in the map information are integrated, which may include one of the following:

Integrate sensor information, vehicle networking information and road sign information in map information;

Integrate sensor information, vehicle networking information and lane line information in map information;

Integrate road traffic scene information in map information;

Screen and integrate the abnormal road traffic information in the Internet of Vehicles information;

The congestion status information in the Internet of Vehicles information and the navigation path planning information in the map information are combined to obtain optimized navigation path planning information.

In sensor information, road sign information, such as the location of speed limit signs, traffic lights, etc., can be parsed from the point cloud data of the radar; road sign information can be parsed from the video stream of the image acquisition device, such as limited The position of the speed sign, the speed limit value, the position of the traffic light, the status of the traffic light (red light or green light or yellow light), etc. The Internet of Vehicles information includes road identification information, such as the position of the speed limit sign, the speed limit value, the position of the traffic light, the status of the traffic light, and so on. The map information includes road identification information, such as the location of traffic lights, speed limit values, and the like.

Those skilled in the art can understand that the accuracy of the information collected by sensor information, vehicle networking information, and map information may be different. For example, the position of the speed limit sign and the position of the traffic light in the point cloud data of the radar is compared with the image The position of the speed limit sign and the position of the traffic light in the video stream of the acquisition device is more accurate; the point cloud data of the radar does not contain the speed limit value, the status of the traffic light, etc.

In one embodiment, integrating sensor information, vehicle networking information, and road marking information in map information may include screening, selecting, and fusing road marking information in sensor information, vehicle networking information, and map information. The output integrated road sign information may include: the position and current state of the traffic lights; the value and position of the speed limit sign, etc.

For the repeated road sign information obtained from sensor information, vehicle networking information and map information, the information with the best accuracy can be selected as the integrated information. For example, the location information of traffic lights can be obtained from sensor information, Internet of Vehicles information, and map information. Then, the location information of traffic lights with the best accuracy is selected as the integrated location information of traffic lights. For example, the location information of the traffic lights in the point cloud data of the radar has the best accuracy. Then, the location information of the traffic lights parsed from the point cloud data of the radar is selected as the integrated traffic light location information.

For only one item of road sign information obtained from sensor information, vehicle networking information and map information, this information can be directly used as the integrated information. For example, in sensor information, vehicle networking information, and map information, the status of traffic lights can only be parsed from the video stream of the image acquisition device. The status of the traffic lights after.

For those skilled in the art, the accuracy of each information in sensor information, vehicle networking information and map information can be known. In the process of information integration, the source of the information can be directly set, or the information can be screened through a model , selection, fusion. After integrating the sensor information, the information of the Internet of Vehicles, and the speed limit sign position, speed limit value, traffic light position, traffic light status and other information in the map information, the position and speed limit value of the integrated speed limit sign are obtained. , the position of the traffic light, the status of the traffic light, and serve as the integrated road sign information.

In the sensor information, the lane line information can be parsed from the video stream of the image acquisition device, for example, the color of the lane line (white or yellow, etc.), the type of the lane line (dotted line or solid line or road edge, etc.), the lane line location, etc. The information of the Internet of Vehicles includes the information of the lane where the vehicle is located, and the information of the lane where the surrounding vehicles are located. The map information includes information about the lane where the vehicle is located, information about the lane where the surrounding vehicles are located, the type of lane line, the location of the lane line, and other information.

In one embodiment, integrating sensor information, vehicle networking information, and lane line information in map information may include screening, selecting, and fusing the sensor information, vehicle networking information, and lane line information in map information. Output the type, coordinates, etc. of the integrated lane lines.

For the repeated lane line information obtained from sensor information, vehicle networking information and map information, the information with the best accuracy can be selected as the integrated information. For example, the type of lane line and the position of the lane line can be obtained from both the sensor information and the map information. Then, the type of lane line and the position of the lane line with the best accuracy are selected as the integrated type of lane line and lane line. s position. For example, the type of lane lines parsed from the video stream of the image acquisition device has the best accuracy, then the type of lane lines parsed from the video stream of the image acquisition device is selected as the type of integrated lane lines; map information The position accuracy of the lane lines obtained from the is the best, then, the position of the lane lines obtained from the map information is selected as the integrated lane line position.

For only one item of road sign information obtained from sensor information, vehicle networking information and map information, this information can be directly used as the integrated information. For example, in sensor information, vehicle networking information and map information, only the color of the lane line can be parsed from the video stream of the image acquisition device, then the color of the lane line parsed from the video stream of the image acquisition device is directly used as the integration Rear lane line color.

For those skilled in the art, the accuracy of each information in sensor information, vehicle networking information and map information can be known. In the process of information integration, the source of the information can be directly set, or the information can be screened through a model , selection, fusion. After integrating the sensor information, the information of the Internet of Vehicles and the information of the lane where the vehicle is located, the information of the lane where the surrounding vehicles are located, the type of lane line, the color of the lane line, the position of the lane line, etc. The information of the lane where the self-vehicle is located, the information of the lane where the surrounding vehicles are located, the type of lane line, the color of the lane line, the position of the lane line and other information are used as the integrated lane line information.

In an embodiment, the road traffic scene information in the map information is integrated, and the road traffic scene information in the map information may be directly used as the integrated road traffic scene information. The road traffic scene information may include road information such as intersection information, road merging information, road bifurcation information, and ramp information. Road information such as intersection information, road merging information, road bifurcation information, and ramp information obtained from the map information can be used as the integrated road traffic scene information.

In one embodiment, screening and integrating the road traffic abnormality information in the Internet of Vehicles information may include: screening, selecting, and merging the road traffic abnormality information from a plurality of roadside units, and obtaining the self-vehicle information. Information about road traffic anomalies that affect driving. Those skilled in the art can understand that the self-vehicle can communicate with multiple roadside units, so that road traffic abnormality information from multiple roadside units can be obtained. After integrating the road traffic abnormal situation information, the road traffic abnormal situation information that affects the driving of the vehicle can be obtained, which is beneficial to predict the driving route of the vehicle. For example, the road traffic abnormal situation information may include road construction information, abnormal vehicle information, emergency vehicle information, and the like. The road construction information, abnormal vehicle information, emergency vehicle information, etc. obtained from the Internet of Vehicles information can be filtered and integrated as the integrated road traffic abnormal situation information. For example, abnormal vehicles may include malfunctioning vehicles, out-of-control vehicles, and the like. Emergency vehicles may include ambulances, fire trucks, police cars, and the like.

In one embodiment, combining the congestion status information in the Internet of Vehicles information with the navigation path planning information in the map information to obtain optimized navigation path planning information, which may include obtaining a plurality of origins from the map information. The navigation path planning information to the destination, combined with the congestion status information in the IoV information, can discard the navigation path planning information affected by the congestion status, thereby obtaining optimized navigation path planning information, which can reach the destination more efficiently.

In one embodiment, in S103, based on the integrated road traffic situation information, reconstructing the driving scene of the self-vehicle may include:

Based on at least one of the integrated road sign information, lane line information, road traffic scene information, road traffic abnormality information and optimized navigation path planning information, the driving scene of the self-vehicle is reconstructed.

The integrated road sign information, lane line information, road traffic scene information, road traffic abnormality information and optimized navigation path planning information are all optimized information, so that the reconstructed driving scene of the vehicle is closer to the real one The driving environment can provide effective assistance for the driver.

Those skilled in the art can understand that after obtaining the road traffic situation information, the driving scene of the self-vehicle can be reconstructed by using the conventional reconstruction technology in the field. A detailed description.

In one embodiment, the driving scene reconstruction method may further include:

Obtain target information from sensor information and vehicle networking information;

Integrate sensor information and target information in IoV information;

Based on the integrated road traffic situation information and the integrated target information, the driving scene of the self-vehicle is reconstructed.

Those skilled in the art can understand that in the surrounding environment of the vehicle, in addition to the surrounding road traffic conditions, there are many objects (vehicles or pedestrians) around the vehicle, and the type, direction, location, etc. of these objects will also affect the driving of the vehicle. . Based on the integrated road traffic situation information and the integrated target information, the driving scene of the self-vehicle is reconstructed, so that the obtained driving scene can not only reflect the road traffic situation, but also reflect the target information around the self-vehicle. The driving scene is more in line with the real driving environment, and the targets around the vehicle can also play an early warning role for the driver, so that the driver can predict the danger of collision and better improve driving safety.

In one embodiment, the target information may include size information of the target, type information of the target (eg, vehicle and vehicle type, or pedestrian and pedestrian type), direction information of the target (eg, forward, backward, left and right), the location information of the target (for example, the abscissa and ordinate of the target relative to the vehicle), etc. Vehicle types may include cars, off-road vehicles, passenger cars, buses, pickup trucks, trucks, motorcycles, two-wheelers, and the like. Pedestrian types may include adults, children, and the like.

In one embodiment, acquiring target information in sensor information may include at least one of the following:

Receive point cloud data from radar, and analyze the point cloud data to obtain target information;

Receive a video stream from an image capture device, and parse the video stream to obtain target information.

FIG. 6 is a schematic diagram of target information in a driving scene reconstruction method according to an embodiment of the present application. In one embodiment, as shown in FIG. 6 , the target information obtained from the point cloud data of the radar may include information such as the size of the target, the position of the target, and the direction of the target. The target information obtained from the video stream of the image capture device may include information such as the size of the target, the type of the target, the position of the target, and the direction of the target.

In one embodiment, acquiring target information in the Internet of Vehicles information may include:

Obtain target information from onboard units and/or roadside units.

The target information obtained from the Internet of Vehicles information may include information such as the type of the target, the location of the target, and the direction of the target.

In one embodiment, integrating the sensor information and the target information in the Internet of Vehicles information may include: screening, corresponding, and selecting the sensor information and the target information in the Internet of Vehicles information. Output the integrated target type, coordinates, orientation (also called direction), etc. For the repeated target information obtained from sensor information and IoV information, the information with the best accuracy can be selected as the integrated information. For example, the location information of the target can be obtained from the point cloud data of the radar, the video stream of the image acquisition device, and the information of the Internet of Vehicles. The position information of the target obtained from the point cloud data of the radar is more accurate, then the position information of the target obtained from the point cloud data of the radar is used as the position information of the integrated target. The direction information of the target obtained from the Internet of Vehicles information is more accurate, then the direction information of the target obtained from the Internet of Vehicles information is used as the direction information of the integrated target.

For those skilled in the art, the accuracy of the sensor information and the target information in the Internet of Vehicles information is known. Therefore, in the process of information integration, the source of the information can be directly set, or the information can be screened through a model. , correspondence, selection, fusion. After integrating the information such as the size of the target, the type of the target, the position of the target, and the direction of the target in the sensor information and the Internet of Vehicles information, the size of the integrated target, the type of the target, the position of the target, and the direction of the target are obtained. information and serve as the integrated target information.

In one embodiment, the driving scene reconstruction method may further include: receiving data of the reconstructed driving scene, and displaying the reconstructed driving scene. In this way, the reconstructed driving scene can be presented.

FIG. 7 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to an embodiment of the present application. In one embodiment, the integrated road traffic situation information includes the lane in which the self-vehicle is located, the left lane, the right lane, the color of the lane line, the type of the lane line, the target located in the lane, and the like. The reconstructed driving scene is shown in Figure 7. It can be seen from Figure 7 that the driving scene includes the vehicle 11, the lane where the vehicle 11 is located, the left lane of the vehicle 11, and the right lane of the vehicle 11. , related objects in the left lane, related objects in the right lane, and lane lines (color, type). In the driving scene, not only the relevant target but also the direction of the target is displayed, for example, the direction of the vehicle 12 and the direction of the vehicle 13 can be clearly obtained from the driving scene.

In one embodiment, the driving scene may only include the most relevant targets of the vehicle. For example, the driving scene may display 3 targets in front of the vehicle, 1 target behind the vehicle, and two targets on the left and right sides of the vehicle. Type, direction and location of 1 target.

In one embodiment, the driving scene may include 3 targets in front of the vehicle, 1 target behind the vehicle, 1 target on the left side and 1 target on the right side of the vehicle, 1 target behind the vehicle on the left side, and 1 target on the left side of the vehicle. Type, direction and location of 1 target in front, 1 target behind the right side of the vehicle, and 1 target in front of the right side of the vehicle.

In this way, the target information most relevant to the self-vehicle can be displayed in the driving scene, avoiding the display of targets that are less relevant to the self-vehicle, making the display screen more concise and clear, and allowing the driver to pay more attention to the most relevant target information. Related goals, improve the driver's experience and avoid display redundancy.

Those skilled in the art can understand that, when the vehicle is driving, sometimes there is no relevant target around the vehicle, for example, there may be no vehicle within a range of several kilometers in front of the vehicle. In order to avoid the displayed scene distance from being too large, in one embodiment, the driving scene may include vehicles and pedestrians located within a certain range of the vehicle. For example, the target may be a vehicle or a pedestrian whose lateral distance from the vehicle is within the range of X1 and the longitudinal distance is within the range of Y1. The values of the horizontal distance X1 and the vertical distance Y1 can be determined according to actual needs.

FIG. 8 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application. The driving scene shown in FIG. 8 includes an intersection, a lane perpendicular to the lane where the vehicle 11 is located,

vehicles

12 and 13 located in the vertical lane, speed limit signs 14 , traffic lights 15 and surrounding areas of the vehicle related goals. In the driving scene shown in FIG. 8 , not only the relevant targets but also the directions of the targets are displayed. For example, the head direction of the vehicle 12 and the head direction of the vehicle 13 can be clearly obtained from the driving scene.

When the self-vehicle is located at a T-junction, the driving scene can include the T-junction, a lane perpendicular to the lane where the vehicle is located, vehicles located in the vertical lane, speed limit signs, traffic lights, and related objects around the vehicle.

FIG. 9 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application. In the driving scenario shown in Figure 9, lane merging is included. There are other vehicles merging into the lane where the self-vehicle 11 is located, and the driving scene shows the lane merging, the merged lane, and the related vehicles in the merged lane.

In one embodiment, when a vehicle enters a forked lane, information about road forks, forked lanes and other target information located in the same lane as the vehicle may be displayed in the driving scene.

Those skilled in the art can understand that a vehicle sometimes drives in the leftmost lane, so the lane on the left side of the vehicle is the opposite lane. In one embodiment, the driving scene may also include opposite lane information. Generally, the object lane does not affect driving, and in a driving scene, the opposite lane may be displayed in a predetermined color such as gray, and the opposite lane target information is not displayed. Those skilled in the art can understand that in actual driving, the target of the target lane usually does not affect driving. Therefore, displaying the opposite lane as a predetermined color and not displaying the target information of the opposite lane is more in line with driving habits and avoids Drivers pay attention to irrelevant road traffic situation information.

In Figures 7, 8 and 9, the targets shown are specific vehicle styles. Those skilled in the art can understand that the contour information of the target can be obtained from sensor information and/or Internet of Vehicles information, and the contour information can be processed so that the specific style of the target can be displayed in the driving scene. In another embodiment, a dedicated target model library can be established for commonly used targets (vehicles or pedestrians), the type of the target can be obtained from sensor information and/or Internet of Vehicles information, and then the corresponding target model library can be called from the target model library. The target model is presented in the driving scene. For example, the target model library includes the model of Mercedes-Benz. When the logo of the Mercedes-Benz is parsed from the image acquisition device, the model of the Mercedes-Benz can be retrieved from the target model library, thereby displaying the model of the Mercedes-Benz in the in the driving scene.

In one embodiment, the sensor information, the Internet of Vehicles information and the road traffic condition information and target information in the map information are acquired in real time, so that the reconstructed driving scene can feed back the surrounding environment conditions of the vehicle in real time, and can display the real-time status of the vehicle in real time. The driving scene is closer to the real driving scene of the vehicle.

In one embodiment, road sign information, lane line information, road traffic scene information, road traffic abnormality information, congestion status information and navigation path planning information can be combined to obtain optimized navigation path planning information, and Fit the best route for the vehicle to travel.

In one embodiment, according to the best route of the vehicle and the lane in which the vehicle is located, the next driving path and movement direction of the vehicle can be displayed in the driving scene, and arrows and other methods can be used in the driving scene. Indicates the next movement direction of the vehicle (for example, left turn, right turn, U-turn, lane change, left front or right front, etc.), making the reconstructed driving scene more realistic and reliable, and providing effective assistance for the driver.

FIG. 10 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application. As shown in FIG. 10 , the next driving path and moving direction (turn left) of the self-vehicle 11 are displayed in the driving scene, and are indicated by means of arrows.

FIG. 11 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application. As shown in FIG. 11 , the next driving path and moving direction (lane change) of the self-vehicle 11 are displayed in the driving scene, and are indicated by means of arrows.

FIG. 12 is a schematic diagram of a driving scene reconstructed by a driving scene reconstruction method according to another embodiment of the present application. As shown in FIG. 12 , the next driving path and moving direction (turning right into the ramp) of the self-vehicle 11 are displayed in the driving scene, and are indicated by means of arrows.

Those skilled in the art can understand that at present, automatic driving can be divided into six automatic driving levels, namely: L0 (no automation), the human driver drives the car with full authority, and can be warned during the driving process; L1 (driving support) , one operation of the steering wheel and acceleration and deceleration are supported through the driving environment, and the rest are operated by humans; L2 (partial automation), multiple operations in the steering wheel and acceleration and deceleration are supported through the driving environment, and the rest are operated by humans; L3 (Conditional automation), all driving operations are completed by the unmanned system, and humans provide appropriate responses according to system requirements; L4 (high automation), all driving operations are completed by the unmanned system, according to system requirements, humans do not All responses must be provided, the road and environmental conditions are limited; L5 (full automation), all driving operations are performed by the driverless system, where possible, the human takes over, the road and environmental conditions are not limited. Different levels of autonomous driving require different levels of driver involvement.

During the driving process, in order to remind the driver of the driving level of the vehicle in real time, in one embodiment, the driving scene reconstruction method may further include:

Extract the autopilot level information and level prompt information of the vehicle;

Based on the integrated road traffic situation information, automatic driving level information and level prompt information, the driving scene of the self-vehicle is reconstructed.

By extracting the self-driving level information and level prompt information, and reconstructing the driving scene of the self-vehicle based on the integrated road traffic condition information, autopilot level information and level prompt information, it is possible to clearly distinguish different driving scenes in the driving scene. Driving level, so that the driver can feel the difference and change of different driving levels more realistically, so that the driver can grasp the driving participation in real time.

In one embodiment, the level prompt information may include at least one of color information, sound information, blinking information, and the like. For example, the level prompt information may be color information, the automatic driving level corresponds to the color information, and one automatic driving level corresponds to one color. Exemplarily, the background color of the driving scene may be displayed as a color corresponding to the automatic driving level. For example, when the automatic driving level is L1, the background color of the driving scene is displayed as light blue; when the automatic driving level is L2, the background color of the driving scene is displayed as light green; when the automatic driving level is L3, the background color of the driving scene is displayed as light green. The background color appears light purple.

In one embodiment, the color of the ego vehicle may be displayed as a color corresponding to the level of autonomous driving.

In one embodiment, the lane line color may be displayed as a color corresponding to the level of autonomous driving.

In one embodiment, the level prompt information may be sound information, the automatic driving level corresponds to the sound information, and one automatic driving level corresponds to one sound.

Those skilled in the art can understand that the display of the level prompt information is not limited to the above methods, as long as the reconstructed driving scene can display the level prompt information so that the driver can feel the driving level of the vehicle, it is within the scope of protection of the present application.

In one embodiment, the level prompt information may be the content displayed in the scene, and different driving levels are reflected by setting different content displayed in the scene. Exemplarily, different driving levels can be embodied by setting the type or number of elements of the road traffic situation information displayed in the driving scene. For example, at the L0 driving level, all real-time road traffic information can be displayed in the reconstructed driving scene; at the L1 driving level, if only the lateral control function is available, only the lane line information can be displayed. Control function, which can only display the target (such as the vehicle) located on the front or rear side of the vehicle; at the L2 driving level, it can display the lane line information, the front side target and the rear side target at the same time; at the L3 driving level, it can be displayed at the same time Display lane line information, front targets, rear targets and road changes; at L4 driving level, the self-vehicle is highlighted, and other elements are simplified; at L5 driving level, only the self-vehicle can be displayed, and at the same time, the driving scene Scenes with entertainment, meetings, games and other links can be added to the system.

Those skilled in the art can understand that automatic driving can be divided into multiple automatic driving functions. For example, the automatic driving function includes no lateral and longitudinal control; Lane Keep Assist (LKA), Lane Center Assist (LCA), only longitudinal control; Highway Assist (HWA) ), that is, there is horizontal and vertical control, and the driver can take off his hands and feet; the traffic jam driving function (Traffic Jam Pilot, TJP), that is, there is horizontal and vertical control, and the driver can take off his hands and feet and eyes.

During the driving process, in order to remind the driver of the driving function adopted by the vehicle in real time, in one embodiment, the driving scene reconstruction method may further include:

Extract the self-driving function information and function prompt information of the vehicle;

Based on the integrated road traffic condition information, automatic driving function information and function prompt information, the driving scene of the self-vehicle is reconstructed.

Such a driving scene reconstruction method can clearly distinguish different driving functions in the driving scene, so that the driver can more truly feel the difference and change of different driving functions, so that the driver can grasp the driving participation in real time.

In one embodiment, the function prompt information may include at least one of cruise speed information, lane line color information, sound information, flashing information, and the like. For example, the cruising speed information can be used as the prompt information of the longitudinal control function, and the color information of the lane line can be used as the prompt information of the lateral control function. The change of cruising speed is used to reflect the longitudinal control function, and the color change of the lane line can be used to reflect the lateral control function. For example, when there is no horizontal and vertical control, the color of the lane line is displayed as white; when the HWA function is used, the color of the lane line is displayed as red; when the TJP function is used, the color of the lane line is displayed as yellow.

Those skilled in the art can understand that the display of the function prompt information is not limited to the above methods, as long as the reconstructed driving scene can display the function prompt information so that the driver can feel the driving function of the vehicle, it is within the scope of protection of the present application.

The driving scene reconstruction method of the present application can acquire sensor information, Internet of Vehicles information and road sign information, lane line information, road traffic scene information, abnormal road traffic situation information, target information and optimized navigation paths in real time in the map information. Planning information, and integrating these information, based on the integrated road sign information, lane line information, road traffic scene information, road traffic abnormal situation information, target information and optimized navigation path planning information, reconstruct the driving scene of the self-vehicle . Therefore, the reconstructed driving scene can adaptively adjust the content of the driving scene according to the actual road environment where the vehicle is located, and display the most relevant information to the user for the automatic driving task of the vehicle. For example, it can be displayed to the user. Exit traffic lights, the status of traffic lights, speed limit signs and values, the lane where the vehicle is located, the lane where surrounding vehicles are located, the color and type of lane lines, etc.; when the road traffic scene changes, you can In the actual situation, the user will be shown the intersection, road merging, road bifurcation or ramp, etc.; when the opposite lane information is displayed in the driving scene, the opposite lane can be displayed in gray, and the target lane target information will not be displayed; When there are road constructions, abnormal vehicles or emergency vehicles that affect driving in the environment, the reconstructed driving scene will display road constructions, abnormal vehicles or emergency vehicles; and the driving path of the self-vehicle is optimal driving path. In a word, the driving scene reconstruction method of this application integrates sensor information from ADAS sensors, vehicle networking information and map information from V2X, and enriches the information sources of the surrounding environment of autonomous vehicles. Moreover, the driving scene reconstruction method, Combining the road traffic situation information and the target information around the vehicle, the reconstructed driving scene can reflect the current real driving environment of the vehicle in real time, which is more in line with the actual needs of users in the automatic driving state and ordinary driving state. The driver provides more effective assistance and improves driving safety.

It should be noted that, although the detailed content of the road traffic situation information has been listed, those skilled in the art can understand that the road traffic situation information is not limited to the listed content. In fact, the user can flexibly set the content of the road traffic situation information obtained from sensor information, IoV information and map information according to personal preferences and/or actual application scenarios, as long as the driving scene reconstruction method of the present application is used to reconstruct the content of the road traffic situation information. The driving scenarios of the self-built vehicle are all within the scope of protection of the present application.

FIG. 13 is a structural block diagram of a driving scene reconstruction apparatus according to an embodiment of the present application. The embodiment of the present application also provides a driving scene reconstruction device. As shown in FIG. 13 , the driving scene reconstruction device may include:

an acquisition module 21, configured to acquire the road traffic situation information in the sensor information, the Internet of Vehicles information and the map information;

The integration module 22, connected with the acquisition module 21, is used to integrate the road traffic situation information in the sensor information, the vehicle networking information and the map information;

The reconstruction module 23 is connected with the integration module 22, and is used for reconstructing the driving scene of the self-vehicle based on the integrated road traffic situation information.

FIG. 14 is a structural block diagram of an acquisition module of a driving scene reconstruction apparatus according to an embodiment of the present application. In one embodiment, as shown in FIG. 14 , the acquisition module 21 may include at least one of the following:

The point cloud data acquisition sub-module 211 is used to receive point cloud data from radar, and analyze the point cloud data to obtain road sign information;

The video stream acquisition sub-module 212 is configured to receive the video stream from the image acquisition device, and parse the video stream to obtain road sign information and lane line information.

In one embodiment, the obtaining module 21 may include:

The vehicle networking information acquisition sub-module 213 is configured to acquire at least one of road sign information, lane line information, road traffic abnormality information and congestion condition information from the vehicle-mounted unit and/or the roadside unit.

In one embodiment, the obtaining module 21 may include:

The map information acquisition sub-module 214 is configured to acquire at least one of road identification information, lane line information, road traffic scene information and navigation path planning information from the map information.

FIG. 15 is a structural block diagram of an integration module of a driving scene reconstruction apparatus according to an embodiment of the present application. In one embodiment, as shown in FIG. 15 , the integration module 22 includes at least one of the following:

a road sign information integration sub-module 221, configured to integrate the sensor information, the vehicle networking information and the road sign information in the map information;

a lane line information integration sub-module 222, configured to integrate the sensor information, the vehicle networking information and the lane line information in the map information;

The scene information integration sub-module 223 is used to integrate the road traffic scene information in the map information;

The abnormal situation information integration sub-module 224 is used to screen and integrate the road traffic abnormal situation information in the Internet of Vehicles information;

The navigation path optimization sub-module 225 is configured to combine the congestion status information in the IoV information with the navigation path planning information in the map information to obtain optimized navigation path planning information.

In one embodiment, the road sign information includes traffic light information and/or speed limit sign information.

In one embodiment, the lane line information includes at least one of information about the lane where the vehicle is located, information about the lane where surrounding vehicles are located, color information of the lane line, type information of the lane line, and position information of the lane line.

In one embodiment, the road traffic scene information includes at least one of intersection information, road merging information, road bifurcation information, and ramp information.

In one embodiment, the road traffic abnormal situation information includes at least one of road construction information, abnormal vehicle information and emergency vehicle information.

In one embodiment, the reconstruction module 23 is configured to reconstruct the reconstruction based on at least one of the integrated road sign information, lane line information, road traffic scene information, road traffic abnormal situation information and optimized navigation path planning information. The driving scene of the car.

In one embodiment, the acquiring module 21 is further configured to acquire the sensor information and the target information in the vehicle networking information; the integration module 22 is further configured to acquire the sensor information and the target information in the vehicle networking information Perform integration; the reconstruction module 23 is used to reconstruct the driving scene of the self-vehicle based on the integrated road traffic situation information and the integrated target information.

In one embodiment, the target information includes at least one of size information, type information, location information and orientation information of the target.

In one embodiment, as shown in FIG. 13 , the driving scene reconstruction device may further include:

The extraction module 24 is used to extract the automatic driving level information and level prompt information of the self-vehicle;

The reconstruction module 23 is configured to reconstruct the driving scene of the self-vehicle based on the integrated road traffic condition information, automatic driving level information and level prompt information.

In one embodiment, the driving scene reconstruction device may further include:

The extraction module 24 is used to extract the automatic driving function information and function prompt information of the self-vehicle;

The reconstruction module 23 is used to reconstruct the driving scene of the self-vehicle based on the integrated road traffic condition information, automatic driving function information and function prompt information.

For the functions of the modules in the embodiments of the present application, reference may be made to the corresponding descriptions in the foregoing methods, and details are not described herein again.

FIG. 16 is a structural block diagram of a driving scene reconstruction system according to an embodiment of the present application. The embodiment of the present application further provides a driving scene reconstruction system. As shown in FIG. 16 , the driving scene reconstruction system includes the driving scene reconstruction device 20 described above. The driving scene reconstruction system may further include:

The sensor 31 is connected to the driving scene reconstruction device 20 for collecting and outputting sensor information to the driving scene reconstruction device 20;

The vehicle networking device 32 is connected to the driving scene reconstruction device 20, and is used for outputting the vehicle networking information to the driving scene reconstruction device;

a map device 33, connected to the driving scene reconstruction device 20, and used for outputting map information to the driving scene reconstruction device;

The display device 34 is connected to the driving scene reconstruction device, and is configured to receive data of the reconstructed driving scene from the driving scene reconstruction device, and display the reconstructed driving scene.

In one embodiment, the sensor 21 , the IoV device 32 , and the map device 33 are all connected to the acquisition module 21 in the driving scene reconstruction device 20 . The display device 34 may be connected to a reconstruction device in the driving scene reconstruction device 20 . Those skilled in the art can understand that a "connection" is an electrical connection, which may be a CAN connection, or a WIFI connection, or a network connection, or the like.

In one embodiment, the driving scene reconstruction device may be a controller, and the controller is integrated with an acquisition module, an integration module, an extraction module and a reconstruction module. The display device may be an instrument controller with a display function in the vehicle. In one embodiment, the controller is integrated with an acquisition module, an integration module, and an extraction module. The display device can be an instrument controller with a display function in the vehicle, and the instrument controller can realize the function of the reconstruction module and the function of display.

For the functions of each module in each device in this embodiment of the present application, reference may be made to the corresponding description in the foregoing method, and details are not described herein again.

An embodiment of the present application further provides a vehicle. In an implementation manner, the vehicle may include the above-mentioned driving scene reconstruction device. In one embodiment, the vehicle may include the driving scene reconstruction system described above.

FIG. 17 is a structural block diagram of an electronic device according to an embodiment of the present application. An embodiment of the present application further provides an electronic device. As shown in FIG. 17 , the electronic device includes: at least one processor 920 , and a memory 910 communicatively connected to the at least one processor 920 . The memory 910 has stored therein instructions executable by at least one processor 920 . The instructions are executed by at least one processor 920 . When the processor 920 executes the instruction, the driving scene reconstruction method in the foregoing embodiment is implemented. The number of the memory 910 and the processor 920 may be one or more. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the application described and/or claimed herein.

The electronic device may further include a communication interface 930 for communicating with external devices and performing interactive data transmission. The various devices are interconnected using different buses and can be mounted on a common motherboard or otherwise as desired. The processor 920 may process instructions for execution within the electronic device, including storing in or on memory to display a Graphical User Interface (Graphical User Interface) on an external input/output device, such as a display device coupled to the interface. GUI) commands for graphical information. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and multiple memories, if desired. Likewise, multiple electronic devices may be connected, each providing some of the necessary operations (eg, as a server array, a group of blade servers, or a multiprocessor system). The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one thick line is shown in FIG. 17, but it does not mean that there is only one bus or one type of bus.

Optionally, in specific implementation, if the memory 910, the processor 920 and the communication interface 930 are integrated on one chip, the memory 910, the processor 920 and the communication interface 930 can communicate with each other through an internal interface.

It should be understood that the above-mentioned processor may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processing, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or any conventional processor or the like. It is worth noting that the processor may be a processor supporting an advanced reduced instruction set machine (Advanced RISC Machines, ARM) architecture.

Embodiments of the present application provide a computer-readable storage medium (such as the above-mentioned memory 910 ), which stores computer instructions, and when the program is executed by a processor, implements the methods provided in the embodiments of the present application.

Optionally, the memory 910 may include a stored program area and a stored data area, wherein the stored program area may store an operating system and an application program required by at least one function; Use the created data, etc. Additionally, memory 910 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 910 may optionally include memory located remotely relative to the processor 920, and these remote memories may be connected to the electronics of the driving scene reconstruction method via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "plurality" means two or more, unless otherwise expressly and specifically defined.

Any description of a process or method in a flowchart or otherwise described herein may be understood to represent a representation of executable instructions comprising one or more (two or more) steps for implementing a specified logical function or process. A module, fragment or section of code. Also, the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved.

The logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment.

It should be understood that various parts of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the method in the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or its combination.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the above-mentioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the technical field disclosed in the present application can easily think of various changes or Replacement, these should be covered within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

A driving scene reconstruction method, comprising:

Obtain road traffic information from sensor information, vehicle networking information and map information;

integrating the sensor information, the vehicle networking information and the road traffic situation information in the map information;

Based on the integrated road traffic situation information, the driving scene of the self-vehicle is reconstructed.
The method according to claim 1, wherein obtaining the road traffic situation information in the sensor information comprises at least one of the following:

Receive point cloud data from radar, and analyze the point cloud data to obtain road sign information;

Receive a video stream from an image acquisition device, and parse the video stream to obtain road sign information and lane line information.
The method according to claim 1, wherein obtaining the road traffic situation information in the Internet of Vehicles information comprises:

At least one of road identification information, lane line information, road traffic abnormality information and congestion condition information from the vehicle-mounted unit and/or the roadside unit is acquired.
The method according to claim 1, wherein obtaining the road traffic situation information in the map information comprises:

Obtain at least one of road identification information, lane line information, road traffic scene information and navigation path planning information from the map information.
The method according to claim 1, wherein integrating the sensor information, the Internet of Vehicles information and the road traffic situation information in the map information includes at least one of the following:

integrating the sensor information, the vehicle networking information and the road sign information in the map information;

integrating the sensor information, the vehicle networking information and the lane line information in the map information;

integrating the road traffic scene information in the map information;

Screening and integrating the abnormal road traffic information in the Internet of Vehicles information;

Combining the congestion status information in the vehicle networking information and the navigation path planning information in the map information to obtain optimized navigation path planning information.
The method according to any one of claims 2 to 5, wherein the road sign information includes traffic light information and/or speed limit sign information.
The method according to any one of claims 2 to 5, wherein the lane line information includes information about the lane where the vehicle is located, information about the lane where surrounding vehicles are located, the color information of the lane line, the information of the lane line At least one of type information and position information of lane lines.
The method according to claim 4 or 5, wherein the road traffic scene information includes at least one of intersection information, road merging information, road bifurcation information and ramp information.
The method according to claim 3 or 5, wherein the road traffic abnormal situation information includes at least one of road construction information, abnormal vehicle information and emergency vehicle information.
The method according to any one of claims 1 to 5, wherein, based on the integrated road traffic situation information, reconstructing the driving scene of the self-vehicle includes:

Based on at least one of the integrated road sign information, lane line information, road traffic scene information, road traffic abnormality information and optimized navigation path planning information, the driving scene of the self-vehicle is reconstructed.
The method according to any one of claims 1 to 5, wherein the method further comprises:

acquiring the sensor information and the target information in the vehicle networking information;

integrating the sensor information and the target information in the vehicle networking information;

Based on the integrated road traffic situation information and the integrated target information, the driving scene of the self-vehicle is reconstructed.
The method according to claim 11, wherein the target information includes at least one of size information, type information, position information and direction information of the target.
The method according to any one of claims 1 to 5, wherein the method further comprises:

Extract the autopilot level information and level prompt information of the vehicle;

Based on the integrated road traffic situation information, automatic driving level information and level prompt information, the driving scene of the self-vehicle is reconstructed.
The method according to any one of claims 1 to 5, wherein the method further comprises:

Extract the self-driving function information and function prompt information of the vehicle;

Based on the integrated road traffic condition information, automatic driving function information and function prompt information, the driving scene of the self-vehicle is reconstructed.
A driving scene reconstruction device, characterized in that it includes:

The acquisition module is used to acquire the road traffic situation information in the sensor information, the Internet of Vehicles information and the map information;

an integration module for integrating the sensor information, the vehicle networking information and the road traffic situation information in the map information;

The reconstruction module is used to reconstruct the driving scene of the self-vehicle based on the integrated road traffic situation information.
The apparatus according to claim 15, wherein the acquiring module comprises at least one of the following:

The point cloud data acquisition sub-module is used to receive the point cloud data from the radar, and analyze the point cloud data to obtain the road sign information;

The video stream acquisition sub-module is used to receive the video stream from the image acquisition device, and analyze the video stream to obtain road sign information and lane line information.
The device according to claim 15, wherein the acquiring module comprises:

The vehicle networking information acquisition sub-module is used to acquire at least one of road identification information, lane line information, road traffic abnormal situation information and congestion situation information from the vehicle-mounted unit and/or the roadside unit.
The device according to claim 15, wherein the acquiring module comprises:

The map information acquisition sub-module is used for acquiring at least one of road identification information, lane line information, road traffic scene information and navigation path planning information from the map information.
The device according to claim 15, wherein the integration module comprises at least one of the following:

a road sign information integration sub-module for integrating the sensor information, the vehicle networking information and the road sign information in the map information;

a lane line information integration sub-module for integrating the sensor information, the vehicle networking information and the lane line information in the map information;

a scene information integration sub-module for integrating the road traffic scene information in the map information;

Abnormal situation information integration sub-module, used for screening and integrating the road traffic abnormal situation information in the IoV information;

The navigation path optimization sub-module is used for combining the congestion status information in the vehicle networking information and the navigation path planning information in the map information to obtain optimized navigation path planning information.
The device according to any one of claims 16 to 19, wherein the road sign information includes traffic light information and/or speed limit sign information.
The device according to any one of claims 16 to 19, wherein the lane line information includes information about the lane where the vehicle is located, information about the lane where surrounding vehicles are located, the color information of the lane line, the information of the lane line At least one of type information and position information of lane lines.
The apparatus according to claim 18 or 19, wherein the road traffic scene information includes at least one of intersection information, road merging information, road bifurcation information, and ramp information.
The device according to claim 17 or 19, wherein the road traffic abnormal situation information includes at least one of road construction information, abnormal vehicle information and emergency vehicle information.
The device according to any one of claims 15 to 19, wherein the reconstruction module is configured to optimize the information based on the integrated road sign information, lane line information, road traffic scene information, road traffic abnormality information and At least one of the navigation path planning information, reconstructs the driving scene of the self-vehicle.
The device according to any one of claims 15 to 19, characterized in that,

The acquisition module is further configured to acquire the sensor information and the target information in the vehicle networking information;

The integration module is further configured to integrate the sensor information and the target information in the vehicle networking information;

The reconstruction module is used to reconstruct the driving scene of the self-vehicle based on the integrated road traffic situation information and the integrated target information.
The apparatus according to claim 25, wherein the target information includes at least one of size information, type information, location information and direction information of the target.
The device according to any one of claims 15 to 19, wherein the device further comprises:

The extraction module is used to extract the autopilot level information and level prompt information of the vehicle;

The reconstruction module is used for reconstructing the driving scene of the self-vehicle based on the integrated road traffic situation information, automatic driving level information and level prompt information.
The device according to any one of claims 15 to 19, wherein the device further comprises:

The extraction module is used to extract the automatic driving function information and function prompt information of the self-vehicle;

The reconstruction module is used for reconstructing the driving scene of the self-vehicle based on the integrated road traffic situation information, automatic driving function information and function prompt information.
A driving scene reconstruction system, comprising the driving scene reconstruction device according to any one of claims 15 to 28, the system further comprising:

a sensor, connected to the driving scene reconstruction device, for collecting and outputting sensor information to the driving scene reconstruction device;

an Internet of Vehicles device, connected to the driving scene reconstruction device, and used for outputting the Internet of Vehicles information to the driving scene reconstruction device;

a map device, connected to the driving scene reconstruction device, for outputting map information to the driving scene reconstruction device;

The display device is connected to the driving scene reconstruction device, and is used for receiving data of the reconstructed driving scene from the driving scene reconstruction device, and displaying the reconstructed driving scene.
A vehicle, characterized by comprising the driving scene reconstruction device according to any one of claims 15 to 28 , or comprising the driving scene reconstruction system according to claim 29 .
An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-14 Methods.
A computer-readable storage medium having computer instructions stored in the computer-readable storage medium, the computer instructions implementing the method according to any one of claims 1-14 when executed by a processor.