WO2022001432A1 - 推理车道的方法、训练车道推理模型的方法及装置 - Google Patents

推理车道的方法、训练车道推理模型的方法及装置 Download PDF

Info

Publication number
WO2022001432A1
WO2022001432A1 PCT/CN2021/094256 CN2021094256W WO2022001432A1 WO 2022001432 A1 WO2022001432 A1 WO 2022001432A1 CN 2021094256 W CN2021094256 W CN 2021094256W WO 2022001432 A1 WO2022001432 A1 WO 2022001432A1
Authority
WO
WIPO (PCT)
Prior art keywords
lane
vehicle
inference
traffic element
element data
Prior art date
Application number
PCT/CN2021/094256
Other languages
English (en)
French (fr)
Inventor
丁洁云
高伟
陶永祥
徐冶
熊健晖
魏龙辉
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to EP21834506.4A priority Critical patent/EP4167126A4/en
Publication of WO2022001432A1 publication Critical patent/WO2022001432A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks

Definitions

  • the present application relates to the field of automatic driving, and more particularly, to a method for inferring a lane, a method and an apparatus for training a lane inference model.
  • Artificial intelligence is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that responds in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.
  • Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and basic AI theory.
  • Autopilot is a mainstream application in the field of artificial intelligence.
  • Autopilot technology relies on the cooperation of computer vision, radar, monitoring devices and global positioning systems to allow motor vehicles to achieve autonomous driving without the need for human active operation.
  • Autonomous vehicles use various computing systems to help transport passengers or cargo from one location to another. Some autonomous vehicles may require some initial or continuous input from an operator, such as a pilot, driver, or passenger.
  • An autonomous vehicle allows the operator to switch from a manual mode of operation to an autonomous driving mode or a mode in between. Since automatic driving technology does not require humans to drive motor vehicles, it can theoretically effectively avoid human driving errors, reduce the occurrence of traffic accidents, and improve the efficiency of highway transportation. Therefore, autonomous driving technology is getting more and more attention.
  • Whether the vehicle can obtain the accurate and selectable lanes available during autonomous driving is a key factor affecting the safety of autonomous driving.
  • training data can be collected in an offline environment to train a neural network model, and the trained neural network model can be used for lane inference.
  • currently trained neural network models are often complex.
  • the present application provides a method for inferring a lane, a method and an apparatus for training a lane inference model, which help to reduce the complexity of a neural network model for lane inference.
  • a method for inferring lanes comprising:
  • the design requirements of the model are also very high.
  • it is difficult to design a neural network model that can handle complex tasks at a practical level for example, a neural network model that can handle complex autonomous driving tasks). neural network model).
  • the traffic element data is the data in the same coordinate system, which can avoid the over-complexity of the neural network model due to the need to process a variety of different types of data, thereby reducing the need for a neural network for lane reasoning. Model complexity.
  • the obtaining of the traffic element data detected by various types of sensors may include: directly obtaining the raw data detected by the sensor, that is to say, at this time, the detected raw data is the traffic element data.
  • obtaining the traffic element data detected by a variety of different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, raw data or raw images obtained by the sensors, etc.) After the result is processed, a processing result is obtained. In this case, the processing result obtained after the processing is the traffic element data.
  • the acquiring traffic element data detected by multiple different types of sensors of the vehicle includes: acquiring detection results of the multiple different types of sensors; The detection result is processed to obtain the traffic element data.
  • the detection results of the various types of sensors are acquired, and the detection results are processed to obtain the traffic element data in the same coordinate system, which can avoid the need to directly process the
  • the detection results (the detection results contain many different types of data) cause the neural network model to be too complex, so that the complexity of the neural network model for lane inference can be reduced.
  • the processing of the detection result to obtain the traffic element data includes: performing coordinate system transformation on the detection result to obtain the traffic element data .
  • the coordinate system conversion is performed on the detection result, and the detection result can be converted into the same coordinate system, which can avoid the need to directly process the detection result (the detection result contains a variety of different type of data), which makes the neural network model too complex, so that the complexity of the neural network model for lane inference can be reduced.
  • the plurality of different types of sensors include cameras, lidars, or millimeter-wave radars.
  • the traffic element data is the outline, position and/or motion trajectory of the traffic element, and the traffic element includes lane markings, other vehicles around the vehicle , one or more of pedestrian, road or traffic lights.
  • the method further includes: determining an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the Traffic elements, the environment perception image includes at least one of a look-around image or a front-view image; wherein, using a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle, The method includes: using a preset neural network model to perform lane inference based on the environment perception image to obtain the lane of the vehicle.
  • the environment around the vehicle can be visually represented by the environment perception image, and at this time, lane inference based on the environment perception image can make the lane (of the vehicle) obtained by reasoning more accurate .
  • the determining of the environment perception image according to the traffic element data includes: based on the traffic element data, superimposing the traffic element on an initial image (for example, stitching one or more frames of perception images collected by a sensor).
  • the environment perception image is obtained from the look-around image obtained later, or the generated initial environment image with the vehicle as the center and the environment is blank).
  • using a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: using a preset neural network model The network model performs lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route to be driven by the vehicle.
  • the driving intention of the vehicle is used to indicate the route to be driven by the vehicle, and lane inference is performed based on the traffic element data and the driving intention of the vehicle, which can further improve the inference result (the vehicle ) lane accuracy.
  • the driving intention of the vehicle is a schematic route diagram of a lane where the vehicle is to travel or a vehicle semantic indication, where the vehicle semantic indication is used to indicate the vehicle's At least one of driving direction, driving lane, or driving intersection.
  • the driving intention of the vehicle is a schematic diagram of a route or a vehicle semantic indication, and the driving intention of the vehicle can intuitively and accurately indicate the route to be driven by the vehicle.
  • the lane of the vehicle is represented by a centerline of the lane and the width of the lane, or the lane of the vehicle is represented by a road cost map.
  • the lane of the vehicle can be visually and accurately represented by the center line of the lane and the width of the lane or the road surface cost map representation.
  • a second aspect provides a method for training a lane inference model, the method comprising:
  • the design requirements of the model are also very high.
  • it is difficult to design a neural network model that can handle complex tasks at a practical level for example, a neural network model that can handle complex autonomous driving tasks). neural network model).
  • the traffic element data is data in the same coordinate system, which can avoid the lane inference model being too complicated due to the need to process a variety of different types of data, so that the lane inference model can be reduced. complexity.
  • the obtaining of the traffic element data detected by various types of sensors may include: directly obtaining the raw data detected by the sensor, that is to say, at this time, the detected raw data is the traffic element data.
  • obtaining the traffic element data detected by a variety of different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, raw data or raw images obtained by the sensors, etc.) After the result is processed, a processing result is obtained. In this case, the processing result obtained after the processing is the traffic element data.
  • the acquiring traffic element data detected by multiple different types of sensors of the vehicle includes: acquiring detection results of the multiple different types of sensors; The detection result is processed to obtain the traffic element data.
  • the detection results of the various types of sensors are acquired, and the detection results are processed to obtain the traffic element data in the same coordinate system, which can avoid the need to directly process the
  • the detection result (the detection result contains a variety of different types of data) causes the lane inference model to be too complicated, so that the complexity of the lane inference model can be reduced.
  • the processing of the detection result to obtain the traffic element data includes: performing coordinate system transformation on the detection result to obtain the traffic element data .
  • the coordinate system conversion is performed on the detection result, and the detection result can be converted into the same coordinate system, which can avoid the need to directly process the detection result (the detection result contains a variety of different type of data), which causes the lane inference model to be too complex, so that the complexity of the lane inference model can be reduced.
  • the plurality of different types of sensors include cameras, lidars, or millimeter-wave radars.
  • the traffic element data is the outline, position and/or motion trajectory of the traffic element, and the traffic element includes lane markings, other vehicles around the vehicle , one or more of pedestrian, road or traffic lights.
  • the method further includes: determining an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the Traffic elements, the environment perception image includes at least one of a look-around image or a front-view image; wherein, using a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle, including: using The lane inference model performs lane inference based on the environment perception image to obtain the lane of the vehicle.
  • the environment around the vehicle can be visually represented by the environment perception image, and at this time, lane inference based on the environment perception image can make the lane (of the vehicle) obtained by reasoning more accurate .
  • the determining of the environment perception image according to the traffic element data includes: based on the traffic element data, superimposing the traffic element on an initial image (for example, stitching one or more frames of perception images collected by a sensor).
  • the environment perception image is obtained from the look-around image obtained later, or the generated initial environment image with the vehicle as the center and the environment is blank).
  • the using a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle includes: using the lane inference model, based on Lane inference is performed between the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route to be driven by the vehicle.
  • the driving intention of the vehicle is used to indicate the route to be driven by the vehicle, and lane inference is performed based on the traffic element data and the driving intention of the vehicle, which can further improve the inference result (the vehicle ) lane accuracy.
  • the driving intention of the vehicle is a schematic diagram of a route of a lane in which the vehicle is to travel or a vehicle semantic indication, where the vehicle semantic indication is used to indicate the vehicle's At least one of driving direction, driving lane, or driving intersection.
  • the driving intention of the vehicle is a schematic diagram of a route or a vehicle semantic indication, and the driving intention of the vehicle can intuitively and accurately indicate the route to be driven by the vehicle.
  • the lane of the vehicle is represented by a centerline of the lane and the width of the lane, or the lane of the vehicle is represented by a road cost map.
  • the lane of the vehicle can be visually and accurately represented by the center line of the lane and the width of the lane or the road surface cost map representation.
  • an apparatus for inferring a lane including:
  • the acquisition unit is used to acquire the detection results of various types of sensors of the vehicle; the coordinate system transformation is performed on the detection results to obtain the traffic element data located in the same coordinate system; the lane inference unit is used to use the preset
  • the neural network model performs lane inference based on the traffic element data to obtain the lane of the vehicle.
  • the traffic element data is the data in the same coordinate system, which can avoid the over-complexity of the neural network model due to the need to process a variety of different types of data, thereby reducing the need for a neural network for lane reasoning. Model complexity.
  • the obtaining of the traffic element data detected by a variety of different types of sensors may include: directly obtaining the raw data detected by the sensor, in this case, the detected raw data is the traffic element data.
  • obtaining the traffic element data detected by a variety of different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, raw data or raw images obtained by the sensors, etc.) After the result is processed, a processing result is obtained. In this case, the processing result obtained after the processing is the traffic element data.
  • the acquiring unit is specifically configured to: acquire detection results of the various types of sensors; process the detection results to obtain the traffic element data .
  • the detection results of the various types of sensors are acquired, and the detection results are processed to obtain the traffic element data in the same coordinate system, which can avoid the need to directly process the
  • the detection results (the detection results contain many different types of data) cause the neural network model to be too complex, so that the complexity of the neural network model for lane inference can be reduced.
  • the acquiring unit is specifically configured to: perform coordinate system transformation on the detection result to obtain the traffic element data.
  • the coordinate system conversion is performed on the detection result, and the detection result can be converted into the same coordinate system, which can avoid the need to directly process the detection result (the detection result contains a variety of different type of data), which makes the neural network model too complex, so that the complexity of the neural network model for lane inference can be reduced.
  • the plurality of different types of sensors include cameras, lidars, or millimeter-wave radars.
  • the traffic element data is the outline, position and/or motion trajectory of the traffic element, and the traffic element includes lane markings, other vehicles around the vehicle , one or more of pedestrian, road or traffic lights.
  • the apparatus further includes a superimposing unit, configured to: determine an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the surroundings of the vehicle
  • the environment and the traffic element the environment perception image includes at least one of a look around image or a front view image; wherein, the lane inference unit is specifically configured to: use a preset neural network model, based on the environment perception Lane inference is performed on the image to obtain the lane of the vehicle.
  • the environment around the vehicle can be visually represented by the environment perception image, and at this time, lane inference based on the environment perception image can make the lane (of the vehicle) obtained by reasoning more accurate .
  • the lane inference unit is specifically configured to: use a preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle, to obtain the the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route on which the vehicle is to travel.
  • the driving intention of the vehicle is used to indicate the route to be driven by the vehicle, and lane inference is performed based on the traffic element data and the driving intention of the vehicle, which can further improve the inference result (the vehicle ) lane accuracy.
  • the driving intention of the vehicle is a schematic diagram of a route of a lane in which the vehicle is to travel or a vehicle semantic indication, where the vehicle semantic indication is used to indicate the vehicle's At least one of driving direction, driving lane, or driving intersection.
  • the driving intention of the vehicle is a schematic diagram of a route or a vehicle semantic indication, and the driving intention of the vehicle can intuitively and accurately indicate the route to be driven by the vehicle.
  • the lane of the vehicle is represented by a centerline of the lane and the width of the lane, or the lane of the vehicle is represented by a road cost map.
  • the lane of the vehicle can be visually and accurately represented by the center line of the lane and the width of the lane or the road surface cost map representation.
  • an apparatus for training a lane inference model including:
  • the acquisition unit is used to acquire the detection results of various types of sensors of the vehicle; the coordinate system transformation is performed on the detection results to obtain the traffic element data located in the same coordinate system; the lane inference unit is used to use the lane inference model , performing lane inference based on the traffic element data to obtain the lane of the vehicle; an adjustment unit configured to adjust the parameters of the lane inference model according to the lane of the vehicle.
  • the traffic element data is data in the same coordinate system, which can avoid the lane inference model being too complicated due to the need to process a variety of different types of data, so that the lane inference model can be reduced. complexity.
  • the obtaining of the traffic element data detected by various types of sensors may include: directly obtaining the raw data detected by the sensor, that is to say, at this time, the detected raw data is the traffic element data.
  • obtaining the traffic element data detected by a variety of different types of sensors may also include: after obtaining the detection results detected by the sensors (for example, raw data or raw images obtained by the sensors, etc.) After the result is processed, a processing result is obtained. In this case, the processing result obtained after the processing is the traffic element data.
  • the acquisition unit is specifically configured to: acquire detection results of the various types of sensors; process the detection results to obtain the traffic element data .
  • the detection results of the various types of sensors are acquired, and the detection results are processed to obtain the traffic element data in the same coordinate system, which can avoid the need to directly process the
  • the detection result (the detection result contains a variety of different types of data) causes the lane inference model to be too complicated, so that the complexity of the lane inference model can be reduced.
  • the acquiring unit is specifically configured to: perform coordinate system transformation on the detection result to obtain the traffic element data.
  • the coordinate system conversion is performed on the detection result, and the detection result can be converted into the same coordinate system, which can avoid the need to directly process the detection result (the detection result contains a variety of different type of data), which causes the lane inference model to be too complex, so that the complexity of the lane inference model can be reduced.
  • the plurality of different types of sensors include cameras, lidars, or millimeter-wave radars.
  • the traffic element data is the outline, position and/or motion trajectory of the traffic element, and the traffic element includes lane markings, other vehicles around the vehicle , one or more of pedestrian, road or traffic lights.
  • the apparatus further includes a superimposing unit, configured to: determine an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the surroundings of the vehicle
  • the environment and the traffic element the environment perception image includes at least one of a look around image or a front view image; wherein, the lane inference unit is specifically configured to: use the lane inference model, based on the environment perception image Lane inference is performed to obtain the lane of the vehicle.
  • the environment around the vehicle can be visually represented by the environment perception image, and at this time, lane inference based on the environment perception image can make the lane (of the vehicle) obtained by reasoning more accurate .
  • the lane inference unit is specifically configured to: use the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle, to obtain the The lane of the vehicle whose driving intent is used to indicate the route the vehicle is to travel.
  • the driving intention of the vehicle is used to indicate the route to be driven by the vehicle, and lane inference is performed based on the traffic element data and the driving intention of the vehicle, which can further improve the inference result (the vehicle ) lane accuracy.
  • the driving intention of the vehicle is a schematic diagram of a route of a lane in which the vehicle is to travel or a vehicle semantic indication, where the vehicle semantic indication is used to indicate the vehicle's At least one of driving direction, driving lane, or driving intersection.
  • the driving intention of the vehicle is a schematic diagram of a route or a vehicle semantic indication, and the driving intention of the vehicle can intuitively and accurately indicate the route to be driven by the vehicle.
  • the lane of the vehicle is represented by the centerline of the lane and the width of the lane, or the lane of the vehicle is represented by a road cost map.
  • the lane of the vehicle can be visually and accurately represented by the center line of the lane and the width of the lane or the road surface cost map representation.
  • an apparatus for inferring lanes includes a storage medium and a central processing unit, the storage medium may be a non-volatile storage medium, and a computer-executable program is stored in the storage medium, and the The central processing unit is connected to the non-volatile storage medium and executes the computer-executable program to implement the method in any possible implementation manner of the first aspect.
  • an apparatus for training a lane inference model includes a storage medium and a central processing unit, the storage medium may be a non-volatile storage medium, and a computer-executable program is stored in the storage medium,
  • the central processing unit is connected to the non-volatile storage medium, and executes the computer-executable program to implement the method in any possible implementation manner of the second aspect.
  • a chip in a seventh aspect, includes a processor and a data interface, the processor reads an instruction stored in a memory through the data interface, and executes any one of the first aspect or the second aspect. method in the implementation.
  • the chip may further include a memory, in which instructions are stored, the processor is configured to execute the instructions stored in the memory, and when the instructions are executed, the The processor is configured to perform the method of the first aspect or any of the possible implementations of the second aspect.
  • a computer-readable storage medium stores program code for execution by a device, the program code comprising a program code for executing any of the first aspect or the second aspect.
  • a directive to implement a method in a method is provided, the computer-readable medium stores program code for execution by a device, the program code comprising a program code for executing any of the first aspect or the second aspect.
  • an automobile including any possible device of the third aspect or the fourth aspect.
  • the traffic element data is the data in the same coordinate system, which can avoid the over-complexity of the neural network model due to the need to process a variety of different types of data, thereby reducing the need for a neural network for lane reasoning. Model complexity.
  • FIG. 1 is a schematic structural diagram of an autonomous vehicle provided by an embodiment of the present application.
  • FIG. 2 is a schematic structural diagram of an automatic driving system according to an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a neural network processor according to an embodiment of the present application.
  • FIG. 4 is a schematic diagram of an application of a cloud-side commanded autonomous vehicle provided by an embodiment of the present application
  • FIG. 5 is a schematic structural diagram of an automatic driving system provided by an embodiment of the present application.
  • FIG. 6 is a schematic block diagram of a road structure recognition module provided by an embodiment of the present application.
  • FIG. 7 is a schematic block diagram of a method for inferring a lane provided by an embodiment of the present application.
  • FIG. 8 is a schematic flowchart of a method for inferring a lane provided by another embodiment of the present application.
  • FIG. 9 is a schematic flowchart of processing a detection result of a sensor in an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of determining an environment-aware image in an embodiment of the present application.
  • FIG. 11 is a schematic flowchart of determining an environment-aware image in another embodiment of the present application.
  • FIG. 12 is a schematic flowchart of determining the driving intention of the vehicle in an embodiment of the application.
  • FIG. 13 is a schematic flowchart of determining the driving intention of the vehicle in another embodiment of the present application.
  • FIG. 14 is a schematic block diagram of a schematic diagram of a route generated in an embodiment of the application.
  • FIG. 15 is a schematic block diagram of lane inference using a neural network model according to an embodiment of the present application.
  • FIG. 16 is a schematic block diagram of lane inference using a neural network model in another embodiment of the present application.
  • 17 is a schematic block diagram of a method for training a lane inference model provided by an embodiment of the present application.
  • FIG. 18 is a schematic block diagram of an apparatus for inferring a lane provided by an embodiment of the present application.
  • 19 is a schematic block diagram of an apparatus for training a driving behavior decision-making model provided by an embodiment of the present application.
  • FIG. 20 is a schematic block diagram of an apparatus for an automatic driving vehicle provided by an embodiment of the present application.
  • the vehicle may be a diesel locomotive, an intelligent electric vehicle, or a hybrid vehicle, or the vehicle may also be a vehicle of other power types. Not limited.
  • the vehicle in the embodiment of the present application may be an automatic driving vehicle.
  • the automatic driving vehicle may be configured with an automatic driving mode, and the automatic driving mode may be a fully automatic driving mode, or may also be a partial automatic driving mode.
  • the embodiment is not limited to this.
  • the vehicle in this embodiment of the present application may also be configured with other driving modes, and the other driving modes may include one or more of a variety of driving modes, such as a sport mode, an economy mode, a standard mode, a snow mode, and a hill climbing mode.
  • the automatic driving vehicle can switch between the automatic driving mode and the above-mentioned various driving models (for the driver-driven vehicle), which is not limited in the embodiment of the present application.
  • FIG. 1 is a functional block diagram of a vehicle 100 provided by an embodiment of the present application.
  • the vehicle 100 is configured in a fully or partially autonomous driving mode.
  • the vehicle 100 can control itself while in an autonomous driving mode, and can determine the current state of the vehicle and its surroundings through human manipulation, determine the likely behavior of at least one other vehicle in the surrounding environment, and determine the other vehicle
  • the vehicle 100 is controlled based on the determined information with a confidence level corresponding to the likelihood of performing the possible behavior.
  • the vehicle 100 may be placed to operate without human interaction.
  • Vehicle 100 may include various subsystems, such as travel system 102 , sensor system 104 , control system 106 , one or more peripherals 108 and power supply 110 , computer system 112 , and user interface 116 .
  • vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple elements. Additionally, each of the subsystems and elements of the vehicle 100 may be interconnected by wire or wirelessly.
  • the travel system 102 may include components that provide powered motion for the vehicle 100 .
  • propulsion system 102 may include engine 118 , energy source 119 , transmission 120 , and wheels/tires 121 .
  • the engine 118 may be an internal combustion engine, an electric motor, an air compression engine, or other types of engine combinations, such as a hybrid engine of an air oil engine and an electric motor, a hybrid engine of an internal combustion engine and an air compression engine.
  • Engine 118 converts energy source 119 into mechanical energy.
  • Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity.
  • the energy source 119 may also provide energy to other systems of the vehicle 100 .
  • Transmission 120 may transmit mechanical power from engine 118 to wheels 121 .
  • Transmission 120 may include a gearbox, differential, and driveshafts.
  • transmission 120 may also include other devices, such as clutches.
  • the drive shaft may include one or more axles that may be coupled to one or more wheels 121 .
  • the sensor system 104 may include several sensors that sense information about the environment surrounding the vehicle 100 .
  • the sensor system 104 may include a positioning system 122 (which may be a GPS system, a Beidou system or other positioning system), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and camera 130.
  • the sensor system 104 may also include sensors of the internal systems of the vehicle 100 being monitored (eg, an in-vehicle air quality monitor, a fuel gauge, an oil temperature gauge, etc.). Sensor data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, orientation, velocity, etc.). This detection and identification is a critical function for the safe operation of the autonomous vehicle 100 .
  • the positioning system 122 may be used to estimate the geographic location of the vehicle 100 .
  • the IMU 124 is used to sense position and orientation changes of the vehicle 100 based on inertial acceleration.
  • IMU 124 may be a combination of an accelerometer and a gyroscope.
  • Radar 126 may utilize radio signals to sense objects within the surrounding environment of vehicle 100 . In some embodiments, in addition to sensing objects, radar 126 may be used to sense the speed and/or heading of objects.
  • the laser rangefinder 128 may utilize laser light to sense objects in the environment in which the vehicle 100 is located.
  • the laser rangefinder 128 may include one or more laser sources, laser scanners, and one or more detectors, among other system components.
  • Camera 130 may be used to capture multiple images of the surrounding environment of vehicle 100 .
  • Camera 130 may be a still camera or a video camera.
  • Control system 106 controls the operation of the vehicle 100 and its components.
  • Control system 106 may include various elements including steering system 132 , throttle 134 , braking unit 136 , sensor fusion algorithms 138 , computer vision system 140 , route control system 142 , and obstacle avoidance system 144 .
  • the steering system 132 is operable to adjust the heading of the vehicle 100 .
  • it may be a steering wheel system.
  • the throttle 134 is used to control the operating speed of the engine 118 and thus the speed of the vehicle 100 .
  • the braking unit 136 is used to control the deceleration of the vehicle 100 .
  • the braking unit 136 may use friction to slow the wheels 121 .
  • the braking unit 136 may convert the kinetic energy of the wheels 121 into electrical current.
  • the braking unit 136 may also take other forms to slow the wheels 121 to control the speed of the vehicle 100 .
  • Computer vision system 140 may be operable to process and analyze images captured by camera 130 in order to identify objects and/or features in the environment surrounding vehicle 100 .
  • the objects and/or features may include traffic signals, road boundaries and obstacles.
  • Computer vision system 140 may use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking, and other computer vision techniques.
  • SFM Structure from Motion
  • the computer vision system 140 may be used to map the environment, track objects, estimate the speed of objects, and the like.
  • the route control system 142 is used to determine the travel route of the vehicle 100 .
  • the route control system 142 may combine data from the sensors 138 , the GPS 122 , and one or more predetermined maps to determine a driving route for the vehicle 100 .
  • the obstacle avoidance system 144 is used to identify, evaluate, and avoid or otherwise traverse potential obstacles in the environment of the vehicle 100 .
  • control system 106 may additionally or alternatively include components other than those shown and described. Alternatively, some of the components shown above may be reduced.
  • Peripherals 108 may include a wireless communication system 146 , an onboard computer 148 , a microphone 150 and/or a speaker 152 .
  • peripherals 108 provide a means for a user of vehicle 100 to interact with user interface 116 .
  • the onboard computer 148 may provide information to the user of the vehicle 100 .
  • User interface 116 may also operate on-board computer 148 to receive user input.
  • the onboard computer 148 can be operated via a touch screen.
  • peripheral devices 108 may provide a means for vehicle 100 to communicate with other devices located within the vehicle.
  • microphone 150 may receive audio (eg, voice commands or other audio input) from a user of vehicle 100 .
  • speakers 152 may output audio to a user of vehicle 100 .
  • Wireless communication system 146 may wirelessly communicate with one or more devices, either directly or via a communication network.
  • wireless communication system 146 may use 3G cellular communications, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communications, such as LTE. Or 5G cellular communications.
  • the wireless communication system 146 may communicate with a wireless local area network (WLAN) using WiFi.
  • WLAN wireless local area network
  • the wireless communication system 146 may communicate directly with the device using an infrared link, Bluetooth, or ZigBee.
  • Other wireless protocols, such as various vehicle communication systems, for example, wireless communication system 146 may include one or more dedicated short range communications (DSRC) devices, which may include communication between vehicles and/or roadside stations public and/or private data communications.
  • DSRC dedicated short range communications
  • the power supply 110 may provide power to various components of the vehicle 100 .
  • the power source 110 may be a rechargeable lithium-ion or lead-acid battery.
  • One or more battery packs of such a battery may be configured as a power source to provide power to various components of the vehicle 100 .
  • power source 110 and energy source 119 may be implemented together, such as in some all-electric vehicles.
  • Computer system 112 may include at least one processor 113 that executes instructions 115 stored in a non-transitory computer-readable medium such as data storage device 114 .
  • Computer system 112 may also be multiple computing devices that control individual components or subsystems of vehicle 100 in a distributed fashion.
  • the processor 113 may be any conventional processor, such as a commercially available CPU. Alternatively, the processor may be a dedicated device such as an ASIC or other hardware-based processor.
  • FIG. 1 functionally illustrates the processor, memory, and other elements of the computer 110 in the same block, one of ordinary skill in the art will understand that the processor, computer, or memory may actually be included in the same physical Multiple processors, computers, or memories within a housing.
  • the memory may be a hard drive or other storage medium located within an enclosure other than computer 110 .
  • reference to a processor or computer will be understood to include reference to a collection of processors or computers or memories that may or may not operate in parallel.
  • some components such as the steering and deceleration components may each have their own processor that only performs computations related to component-specific functions .
  • a processor may be located remotely from the vehicle and in wireless communication with the vehicle. In other aspects, some of the processes described herein are performed on a processor disposed within the vehicle while others are performed by a remote processor, including taking steps necessary to perform a single maneuver.
  • the data storage device 114 may include instructions 115 (eg, program logic) executable by the processor 113 to perform various functions of the vehicle 100 , including those described above.
  • Data storage 114 may also contain additional instructions, including sending data to, receiving data from, interacting with, and/or performing data processing on one or more of propulsion system 102 , sensor system 104 , control system 106 , and peripherals 108 . control commands.
  • the data storage device 114 may store data such as road maps, route information, the vehicle's position, direction, speed, and other such vehicle data, among other information. Such information may be used by the vehicle 100 and the computer system 112 during operation of the vehicle 100 in autonomous, semi-autonomous and/or manual modes.
  • a user interface 116 for providing information to or receiving information from a user of the vehicle 100 .
  • the user interface 116 may include one or more input/output devices within the set of peripheral devices 108 , such as a wireless communication system 146 , an onboard computer 148 , a microphone 150 and a speaker 152 .
  • Computer system 112 may control functions of vehicle 100 based on input received from various subsystems (eg, travel system 102 , sensor system 104 , and control system 106 ) and from user interface 116 .
  • computer system 112 may utilize input from control system 106 in order to control steering unit 132 to avoid obstacles detected by sensor system 104 and obstacle avoidance system 144 .
  • computer system 112 is operable to provide control of various aspects of vehicle 100 and its subsystems.
  • one or more of these components described above may be installed or associated with the vehicle 100 separately.
  • data storage device 114 may exist partially or completely separate from vehicle 100 .
  • the above-described components may be communicatively coupled together in a wired and/or wireless manner.
  • FIG. 1 should not be construed as a limitation on the embodiments of the present application.
  • a self-driving car traveling on a road can recognize objects within its surroundings to determine adjustments to the current speed.
  • the objects may be other vehicles, traffic control equipment, or other types of objects.
  • each identified object may be considered independently, and based on the object's respective characteristics, such as its current speed, acceleration, distance from the vehicle, etc., may be used to determine the speed at which the autonomous vehicle is to adjust.
  • the autonomous vehicle vehicle 100 or a computing device associated with the autonomous vehicle 100 may be based on the characteristics of the identified objects and the surrounding environment state (eg, traffic, rain, ice on the road, etc.) to predict the behavior of the identified object.
  • each identified object is dependent on the behavior of the other, so it is also possible to predict the behavior of a single identified object by considering all identified objects together.
  • the vehicle 100 can adjust its speed based on the predicted behavior of the identified object.
  • the self-driving car can determine what steady state the vehicle will need to adjust to (eg, accelerate, decelerate, or stop) based on the predicted behavior of the object.
  • other factors may also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road being traveled, the curvature of the road, the proximity of static and dynamic objects, and the like.
  • the computing device may also provide instructions to modify the steering angle of the vehicle 100 so that the self-driving car follows a given trajectory and/or maintains contact with objects in the vicinity of the self-driving car (eg, , cars in adjacent lanes on the road) safe lateral and longitudinal distances.
  • objects in the vicinity of the self-driving car eg, , cars in adjacent lanes on the road
  • the above-mentioned vehicle 100 can be a car, a truck, a motorcycle, a bus, a boat, an airplane, a helicopter, a lawn mower, a recreational vehicle, a playground vehicle, construction equipment, a tram, a golf cart, a train, a cart, etc.
  • the application examples are not particularly limited.
  • FIG. 2 is a schematic diagram of an automatic driving system provided by an embodiment of the present application.
  • the automatic driving system shown in FIG. 2 includes a computer system 101 , wherein the computer system 101 includes a processor 103 , and the processor 103 is coupled with a system bus 105 .
  • the processor 103 may be one or more processors, each of which may include one or more processor cores.
  • a video adapter 107 which can drive a display 109, is coupled to the system bus 105.
  • System bus 105 is coupled to input-output (I/O) bus 113 through bus bridge 111 .
  • I/O interface 115 is coupled to the I/O bus. I/O interface 115 communicates with various I/O devices, such as input device 117 (eg, keyboard, mouse, touch screen, etc.), media tray 121, (eg, CD-ROM, multimedia interface, etc.).
  • Transceiver 123 (which can send and/or receive radio communication signals), camera 155 (which can capture still and moving digital video images) and external USB interface 125 .
  • the interface connected to the I/O interface 115 may be a USB interface.
  • the processor 103 may be any conventional processor, including a reduced instruction set computing (reduced instruction set computer, RISC) processor, a complex instruction set computing (complex instruction set computer, CISC) processor, or a combination thereof.
  • the processor may be a dedicated device such as an application specific integrated circuit (ASIC).
  • the processor 103 may be a neural network processor or a combination of a neural network processor and the above-mentioned conventional processors.
  • computer system 101 may be located remotely from the autonomous vehicle and may communicate wirelessly with the autonomous vehicle.
  • some of the processes described herein are performed on a processor disposed within the autonomous vehicle, others are performed by a remote processor, including taking actions required to perform a single maneuver.
  • Network interface 129 is a hardware network interface, such as a network card.
  • the network 127 may be an external network, such as the Internet, or an internal network, such as an Ethernet network or a virtual private network (VPN).
  • the network 127 may also be a wireless network, such as a WiFi network, a cellular network, and the like.
  • the hard disk drive interface is coupled to the system bus 105 .
  • the hard drive interface is connected to the hard drive.
  • System memory 135 is coupled to system bus 105 . Data running in system memory 135 may include operating system 137 and application programs 143 of computer 101 .
  • the operating system includes a parser 139 (shell) and a kernel (kernel) 141 .
  • the shell is an interface between the user and the kernel of the operating system.
  • the shell is the outermost layer of the operating system.
  • the shell manages the interaction between the user and the operating system: waiting for user input, interpreting user input to the operating system, and processing various operating system output.
  • Kernel 141 consists of those parts of the operating system that manage memory, files, peripherals, and system resources. Interacting directly with hardware, the operating system kernel usually runs processes and provides inter-process communication, providing CPU time slice management, interrupts, memory management, IO management, and more.
  • Application 143 includes programs that control the autonomous driving of the car, for example, programs that manage the interaction of the autonomous car with obstacles on the road, programs that control the route or speed of the autonomous car, and programs that control the interaction of the autonomous car with other autonomous vehicles on the road. .
  • Application 143 also exists on the system of deploying server 149. In one embodiment, the computer system 101 may download the application 143 from the deploying server 14 when the application 143 needs to be executed.
  • Sensor 153 is associated with computer system 101 .
  • the sensor 153 is used to detect the environment around the computer 101 .
  • the sensor 153 can detect animals, cars, obstacles and pedestrian crossings, etc. Further sensors can also detect the environment around the above-mentioned animals, cars, obstacles and pedestrian crossings, such as: the environment around animals, for example, animals appear around other animals, weather conditions, ambient light levels, etc.
  • the sensors may be cameras, infrared sensors, chemical detectors, microphones, and the like.
  • the application program 143 can perform lane inference according to the surrounding environment information detected by the sensor 153 to obtain the lane to be driven (or the driveable lane) of the vehicle. At this time, based on the lane to be driven, the automatic Drive the vehicle to plan the path, and then determine the control amount of the vehicle to realize the automatic driving of the vehicle.
  • FIG. 3 is a hardware structure diagram of a chip provided by an embodiment of the present application, where the chip includes a neural network processor 20 .
  • the chip may be in the processor 103 as shown in FIG. 2 to perform lane inference according to environmental information.
  • the algorithms of each layer in the pre-trained neural network can be implemented in the chip as shown in FIG. 3 .
  • the method for inferring lanes in this embodiment of the present application can also be implemented in a chip as shown in FIG. 3 , where the chip can be the same chip that implements the above-mentioned pre-trained neural network, or the chip can also be It is a different chip from the chip that implements the above-mentioned pre-trained neural network, which is not limited in this embodiment of the present application.
  • the neural network processor NPU 50NPU is mounted on the main CPU (host CPU) as a co-processor, and tasks are assigned by the host CPU.
  • the core part of the NPU is the operation circuit 50, and the operation circuit 503 is controlled by the controller 504 to extract the matrix data in the memory and perform multiplication operations.
  • the arithmetic circuit 203 includes multiple processing units (process engines, PEs). In some implementations, the arithmetic circuit 203 is a two-dimensional systolic array. The arithmetic circuit 203 may also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, arithmetic circuit 203 is a general-purpose matrix processor.
  • the operation circuit fetches the data corresponding to the matrix B from the weight memory 202 and buffers it on each PE in the operation circuit.
  • the arithmetic circuit fetches the data of matrix A and matrix B from the input memory 201 to perform matrix operation, and stores the partial result or final result of the matrix in the accumulator 208 .
  • the vector calculation unit 207 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison and so on.
  • the vector computing unit 207 can be used for network computation of non-convolutional/non-FC layers in the neural network, such as pooling, batch normalization, local response normalization, etc. .
  • the vector computation unit 207 can store the processed output vectors to the unified buffer 206 .
  • the vector calculation unit 207 may apply a nonlinear function to the output of the arithmetic circuit 203, such as a vector of accumulated values, to generate activation values.
  • vector computation unit 207 generates normalized values, merged values, or both.
  • the vector of processed outputs can be used as activation input to the arithmetic circuit 203, eg, for use in subsequent layers in a neural network.
  • Unified memory 206 is used to store input data and output data.
  • the weight data directly transfers the input data in the external memory to the input memory 201 and/or the unified memory 206 through the storage unit access controller 205 (direct memory access controller, DMAC), and stores the weight data in the external memory into the weight memory 202, And the data in the unified memory 206 is stored in the external memory.
  • DMAC direct memory access controller
  • a bus interface unit (BIU) 210 is used to realize the interaction between the main CPU, the DMAC and the instruction fetch memory 209 through the bus.
  • the instruction fetch memory (instruction fetch buffer) 209 connected with the controller 204 is used to store the instructions used by the controller 204;
  • the controller 204 is used for invoking the instructions cached in the instruction fetch memory 209 to realize and control the working process of the operation accelerator.
  • the unified memory 206 , the input memory 201 , the weight memory 202 and the instruction fetch memory 209 are all on-chip memories, and the external memory is the memory outside the NPU, and the external memory can be double data rate synchronous dynamic random access.
  • Memory double data rate synchronous dynamic random access memory, DDR SDRAM), high bandwidth memory (high bandwidth memory, HBM) or other readable and writable memory.
  • Computer system 112 may also receive information from or transfer information to other computer systems. Alternatively, sensor data collected from the sensor system 104 of the vehicle 100 may be transferred to another computer for processing of the data.
  • Networks and intermediate nodes may include various configurations and protocols, including the Internet, the World Wide Web, Intranets, Virtual Private Networks, Wide Area Networks, Local Area Networks, private networks using one or more of the company's proprietary communication protocols, Ethernet, WiFi and HTTP, and various combinations of the foregoing.
  • Such communications may be by any device capable of transferring data to and from other computers, such as modems and wireless interfaces.
  • server 320 may comprise a server having multiple computers, eg, a load balancing server farm, that exchanges information with different nodes of the network for the purpose of receiving, processing, and transmitting data from computer system 312 .
  • the server may be configured similarly to computer system 312 , with processor 330 , memory 340 , instructions 350 , and data 360 .
  • the data 360 of the server 320 may include map information and/or information about road conditions surrounding the autonomous vehicle.
  • the server 320 may receive, detect, store, update, and transmit information related to road conditions for the autonomous vehicle.
  • the map information may be a high-precision map, and the map information may include pre-stored lane line information (eg, lane marking lines).
  • the relevant information about the road conditions around the autonomous vehicle includes the position information of other vehicles that have a junction with the predicted driving path of the autonomous vehicle and the information of motion parameters, such as the current position information of the other vehicles and the speed information of the other vehicles. , the predicted driving trajectory information of other vehicles, etc.
  • FIG. 5 is an exemplary structural block diagram of an automatic driving system 500 according to an embodiment of the present application.
  • the automatic driving system 500 may include a sensor 510 , a perceptual location map subsystem 520 , a fusion subsystem 530 , a predictive cognition subsystem 540 , a planning subsystem 550 , a control subsystem 560 and an actuator 570 .
  • the automatic driving system 500 shown in FIG. 5 is only an example and not a limitation. Those skilled in the art can understand that the automatic driving system 500 may include more or less modules or subsystems (or units). ), which is not limited in the embodiments of the present application.
  • the sensor 510 may be used to sense objects outside the vehicle (eg, an autonomous vehicle), and transmit the sensed data to the sensed location mapping module 520 .
  • the sensor 510 may include sensors such as cameras, millimeter-wave radars, lidars, and ultrasonic radars.
  • the perceptual location map subsystem 520 may be used to obtain traffic element data in the environment, provide the location of the vehicle (eg, an autonomous vehicle), and the road topology information where the vehicle is located.
  • vehicle e.g, an autonomous vehicle
  • road topology information where the vehicle is located.
  • the perception and positioning map subsystem 520 includes a visual perception module 522, a radar perception module 524, a positioning module 526 and a map module 526, wherein the visual perception module 522 can be used to detect lane markings, moving vehicles, passable Areas and traffic signs, the radar perception module 524 can be used to detect vehicles, road edges and passable areas, the positioning module 526 can be used to achieve global positioning of vehicles, and the map module 526 can be used to obtain surrounding road topology.
  • the visual perception module 522 can be used to detect lane markings, moving vehicles, passable Areas and traffic signs
  • the radar perception module 524 can be used to detect vehicles, road edges and passable areas
  • the positioning module 526 can be used to achieve global positioning of vehicles
  • the map module 526 can be used to obtain surrounding road topology.
  • the fusion subsystem 530 can be used to fuse the output results of multiple perception modules to achieve more accurate moving target detection, passable area detection, and road edge detection.
  • the fusion subsystem 530 includes a target fusion module 532 , a passable road surface fusion module 534 , and a road edge detection and fusion module 536 , wherein the target fusion module 532 can be used to realize the fusion of the visual perception target and the radar detection target. and tracking, the passable road fusion module 534 can be used to realize the fusion and tracking of the passable area detected by visual detection and the passable area detected by radar, and the road edge detection and fusion module 536 can be used for the fusion of visual semantic segmentation results and radar detection road edges and tracking.
  • Predictive cognitive subsystem 540 may be used to enable further understanding and reasoning about the environment.
  • the prediction cognition subsystem 540 includes a target prediction module 542 and a road structure recognition module 544, wherein the target prediction module 542 can be used to realize future motion prediction of detecting moving objects, and the road structure recognition module 544 can It is used to perform lane inference to obtain the lane to be driven (or available lane) of the autonomous vehicle.
  • the planning subsystem 550 may be used to plan the self-driving strategy based on the obtained environmental detection results and environmental cognition results.
  • the planning subsystem 550 includes a highly automated driving (HAD) planner 552, a partially automated driving (PAD) planner 554 and a function manager 556, wherein the HAD planner 552 It can be used for planning in the case of highly automatic driving functions, including many automatic driving functions.
  • the PAD planner 554 can be used for partial automatic driving, such as adaptive cruise and other functions, and the function manager 556 can be used for starting, switching HAD function and PAD function.
  • the control subsystem 560 is configured to calculate the vehicle body control amount based on the planning result, so as to issue a target control amount command to the chassis electronic control system.
  • the control subsystem 560 includes a lateral control module 562, a longitudinal control module 564 and an arbiter 566, wherein the lateral control module 562 can be used to calculate the steering system control amount to achieve the target trajectory, and the longitudinal control module 564 can use
  • the arbiter 566 may be used to implement the arbitration of the multi-function control quantities.
  • the actuator 570 can be used to implement the control commands issued by the control subsystem to achieve steering, acceleration and deceleration in line with expectations.
  • the automatic driving system 500 shown in FIG. 5 above is only an example and not a limitation, and the automatic driving system 500 may include more or less systems, modules, sub-models or units. This is not limited.
  • the road structure recognition module 544 in FIG. 5 can perform lane inference through various solutions, as follows:
  • Lane inference is performed based on the detection of lane markings, that is, through methods such as machine vision or image detection, the lane markings in the image are detected, and the lane to be driven (or drivable) of the autonomous vehicle is determined based on the detected lane markings. Lane).
  • the lane markings are worn or lost
  • the lighting conditions are not good, for example, there is backlight or the shadow of the railings and other light and shadow interference;
  • the road markings are complex, for example, there are texts or there are guide marking lines at the ramp.
  • Lane reasoning based on the high-precision map that is, query the high-precision map, obtain the lane marking information pre-stored in the high-precision map, and determine the lane to be driven (or the driving lane) of the autonomous vehicle based on the lane marking information. .
  • solution 3 a method for lane inference based on environmental vehicles is proposed in solution 3, that is, the interested vehicles are screened according to preset conditions, and after the interesting vehicles are selected, the interesting vehicles are determined.
  • the driving trajectory of the autonomous vehicle is determined based on the driving trajectory of the vehicle of interest.
  • the preset conditions may include the distance range of the vehicle, the curvature of the driving trajectory of the vehicle, the observation period of the vehicle, and the like.
  • this method is too dependent on the ambient vehicle (i.e. the vehicle of interest), for example, when there is no preceding vehicle, the scheme cannot be implemented; the driving behavior of the ambient vehicle does not obey the traffic rules (for example, the ambient vehicle does not follow the lane
  • the lane to be driven determined based on the driving trajectory of the vehicle in this environment may conflict with the driving trajectory of other vehicles, which may lead to driving risks.
  • the solution of end-to-end autonomous driving combined with real-time perception and driving intention that is, firstly, in the offline environment, the neural network model is trained based on the collected training data to obtain a neural network model that can be used for end-to-end autonomous driving; then , In the online link, the local perception (ie, local environment perception) and driving intention are used as inputs, and the neural network model trained in the offline environment is used to calculate the control amount of the car control (ie, the control of the autonomous vehicle).
  • the local perception in this method (scheme 4) only includes the image information collected by the camera, the ability to generalize to light conditions, road types, etc., as well as road cognition, and the matching of road and driving intentions.
  • the processing of dynamic obstacles and static obstacles all need to rely on the same neural network model for processing, which will lead to overly complex neural network models (for end-to-end autonomous driving).
  • the input of this method is the image information (and driving intention) collected by the camera
  • the output is the vehicle control result (for example, the vehicle control result may be the control amount of the autonomous vehicle)
  • the neural network model obtained by training is used to collect the data.
  • the camera of the image and the size of the controlled vehicle ie, the autonomous vehicle
  • the training data needs to be re-collected
  • the neural network model needs to be trained based on the re-collected training data.
  • the output of this method is the vehicle control result, the lane to be driven (or the driveable lane) of the automatic driving vehicle is not output, therefore, the decision control logic when using this method for automatic driving cannot be traced.
  • this application proposes a method for inferring a lane, by acquiring the traffic element data detected by a variety of different types of sensors and located in the same coordinate system, and performing lane inference based on the traffic element data to obtain the lane of the vehicle, Helps reduce the complexity of neural network models for lane inference.
  • FIG. 6 is a schematic structural diagram of a road structure recognition module 600 according to an embodiment of the present application.
  • the road structure recognition module 600 in FIG. 6 may be the road structure recognition module 544 in FIG. 5 described above, and the road structure recognition module 600 may include multiple sub-modules or units.
  • the road structure recognition module 600 may include a perception road structure modeling sub-module 610, a map road structure modeling sub-module 620, a road structure modeling sub-module 630 combining environment and intention, and road model reliability Analysis sub-module 640 .
  • the perceptual road structure modeling sub-module 610 can be used to implement the first solution in the prior art, that is, lane inference based on lane markings.
  • the main application scenario of this module 610 can be that the lane markings are clearly visible and the lines of the lane markings shape, curvature, etc. are within the perceptual capabilities (of a machine vision model or image detection model).
  • the map road structure modeling sub-module 620 can be used to implement the second solution in the prior art, that is, lane inference based on a high-precision map.
  • the main application scenarios of this module 620 can be high-precision positioning and high-precision map availability.
  • the road structure modeling sub-module 630 combined with environment and intent can be used to implement the method for inferring lanes in the embodiments of the present application, and the method does not absolutely rely on lane markings, high-precision positioning or high-precision maps. Therefore, the Applications can be full scenes.
  • the road model reliability analysis sub-module 640 can be used to analyze the road models output by the above-mentioned modules 610, 620 and 630 (for example, the lane to be driven or the driveable lane of the autonomous vehicle). Road model evaluation, determine the reliability level of each road model, and output the road model with the highest reliability level and the reliability level of the road model.
  • the lane to be driven (or the driveable lane) of the autonomous driving vehicle in the embodiments of the present application may be the shape information of the lane, for example, the geometric shape of the lane, or may also be the cost of the lane
  • the information for example, a cost map (or cost grid map) of a lane, or other information that can describe a lane, which is not limited in this embodiment of the present application.
  • FIG. 7 is a schematic flowchart of a method 700 for inferring a lane provided by an embodiment of the present application.
  • the method 700 shown in FIG. 7 can be executed by the processor 113 in the vehicle 100 in FIG. 1 , or can also be executed by the processor 103 in the automatic driving system in FIG.
  • the processor 320 in the server 320 executes.
  • the method 700 shown in FIG. 7 may include steps 710 and 720. It should be understood that the method 700 shown in FIG. 7 is only an example and not a limitation, and the method 700 may include more or less steps. This is not limited, and these steps are described in detail below.
  • the various types of sensors may include cameras, lidars, or millimeter-wave radars, or the various types of sensors may also include other sensors, which are not limited in this embodiment of the present application.
  • S720 Process the detection result to obtain traffic element data located in the same coordinate system.
  • the traffic element data may be data in the same coordinate system.
  • the traffic element data may be the type, outline, position and/or motion track of the traffic element, the traffic element may be a stationary object, or the traffic element may also be a moving object ( moving object).
  • the traffic elements may include one or more of lane markings, other vehicles around the vehicle, pedestrians, roads (eg, roads or curbs around the vehicle, etc.), or traffic lights.
  • obtaining the traffic element data located in the same coordinate system may include: directly acquiring the raw data detected by the sensor, in this case, the detected raw data is the traffic element data.
  • obtaining the traffic element data located in the same coordinate system may also include: after obtaining the detection result detected by the sensor (for example, the original data or original image obtained by the sensor, etc.), performing the detection on the detected detection result. After processing, a processing result is obtained. At this time, the processing result obtained after the processing is the traffic element data.
  • the processing of the detected detection result may refer to S810 in the following method 800 in FIG. 8 .
  • it may include performing at least one of the following multiple processing methods on the detection result:
  • the coordinate system transformation is performed on the detection result (for example, the original data or original image obtained by the sensor).
  • the ranging data detected by lidar can be converted to a specified coordinate system, or the ranging data can also be converted between multiple coordinate systems, and the coordinate system here can include vehicle body coordinates system, relative coordinate system, or image coordinate system, etc., and may also include other common coordinate systems, which are not limited here.
  • the traffic element data can be considered to include the types of traffic elements.
  • traffic elements contained in images captured by the camera can be identified.
  • the detection result (or the traffic element included in the detection result of the sensor) may be processed using an existing algorithm to determine the outline, position and/or movement trajectory of the traffic element.
  • the outline of the traffic element may include information that can describe the outline feature of the traffic element, such as the geometric feature of the traffic element or the centerline of the traffic element, which is not limited in this embodiment of the present application.
  • the above-mentioned method for processing the detected detection results is only an example and not a limitation, and other processing (for example, the movement trajectory of the traffic element) may also be performed on the detection result. filter, etc.), which is not limited in this embodiment of the present application.
  • the method 700 may further include S712.
  • S712 Determine an environment perception image according to the traffic element data.
  • the environment-aware image may be used to indicate the environment around the vehicle, and the environment-aware image may include at least one of a look-around image or a front-view image.
  • the context-aware image may be a world model.
  • the world model may refer to the aggregation of the vehicle's environmental perception information.
  • the traffic elements may be superimposed on the environment-aware image based on the traffic element data.
  • the traffic element data can be converted into the same coordinate system, and then the traffic element data can be superimposed on the environment perception image according to the position and angle of view of each traffic element relative to the vehicle, that is, to obtain A look-around or front-view image of the vehicle.
  • S730 use a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.
  • a preset neural network model may also be used to perform lane inference based on the environment perception image to obtain the lane of the vehicle.
  • the neural network model may be obtained after training based on traffic element data (collected offline) and/or driving intent.
  • traffic element data collected offline
  • driving intent driving intent
  • the driving intention of the vehicle may be a schematic diagram of a route of a lane to be driven by the vehicle or a vehicle semantic indication, where the vehicle semantic indication is used to indicate the driving direction of the vehicle, a driving lane or a driving road in an intersection. at least one.
  • the semantic instructions may include “turn left”, “turn right”, “go straight” and “stop”; “Take the second exit” etc.
  • the method for determining the driving intention is not limited in the embodiments of the present application, and the specific method may refer to the prior art, which will not be repeated here.
  • a preset neural network model may also be used to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the The route the vehicle is to travel.
  • the lane of the vehicle can be represented by the center line of the lane and the width of the lane; alternatively, the lane of the vehicle can also be represented by the lane of the vehicle by the road cost map; or, it can also be Others in the prior art represent the lanes of the vehicle, which are not limited in the embodiments of the present application.
  • FIG. 8 is a schematic flowchart of a method 800 for inferring a lane provided by an embodiment of the present application.
  • the method 700 shown in FIG. 8 can be executed by the processor 113 in the vehicle 100 in FIG. 1 , or can also be executed by the processor 103 in the automatic driving system in FIG.
  • the processor 320 in the server 320 executes.
  • the method 800 shown in FIG. 8 may include steps 810, 820, 830 and 840. It should be understood that the method 800 shown in FIG. 8 is only an example and not a limitation, and the method 800 may include more or less steps , this is not limited in the embodiments of the present application, and the steps are described in detail below.
  • the processing of the detection result of the sensor mentioned here includes: transforming the coordinate system of the detection result of the sensor (for example, the original data or original image obtained by the sensor, etc.), and determining (or identifying) that the detection result of the sensor contains and at least one of the contour, position and/or motion trajectory of the traffic element (corresponding to the detection result) determined based on the detection result of the sensor,
  • the above S810 may be as shown in the method 900 in FIG. 9 , and the method 900 may include step 910 , step 920 and step 930 .
  • the detection result may be regarded as the detection result of the current frame, and the matching and splicing of the current frame (the road in the detection result) and the previous frame (the road in the detection result) can be realized through S910.
  • matching and splicing of the current frame (the road in the detection result) and other frames (the road in the detection result) before the current frame may also be implemented through S910.
  • each step in the above method 900 is not necessarily a step to be executed, and the corresponding step will only be executed when the detection result contains a corresponding traffic element.
  • S910 is executed, otherwise, S910 is not executed.
  • S820 Determine an environment perception image according to the traffic element data.
  • the S820 may be implemented by any of the following solutions.
  • the first solution may include step 1010 and step 1020, as follows.
  • the images collected by the sensors of the multiple viewing angles are spliced to form a complete look-around image.
  • the road is superimposed on the look-around image.
  • the second solution may include step 1110 and step 1120 , the details are as follows.
  • an initial image with the vehicle as the center and a blank environment may be generated, which may be referred to as an initial environmental image.
  • the environment-aware images obtained in the above two solutions may include front-view images and/or surround-view images. Meanwhile, for the splicing method and the superimposing method in the above two solutions, reference may be made to the prior art, which will not be repeated here.
  • the S830 may be implemented by any of the following methods.
  • method one may include the following steps.
  • whether the vehicle deviates from the planned route may be determined according to the current position of the vehicle.
  • the route can be re-planned to obtain the waypoint sequence of the road to be traveled; otherwise, the waypoint sequence of the original road to be traveled can continue to be used, that is, the waypoint sequence of the original road to be traveled is regarded as the roadpoint sequence to be traveled.
  • a sequence of waypoints for the driving road is regarded as the roadpoint sequence to be traveled.
  • the waypoint sequence of the road to be traveled can be represented by ⁇ P(1), P(2), P(3), ..., P(N) ⁇ , where P(k) represents the position of the kth waypoint information, k and N are both positive integers.
  • the current position of the vehicle may be projected into the waypoint sequence of the road to be traveled determined in S1220, and the point closest to the current position of the vehicle in the waypoint sequence of the road to be traveled may be projected.
  • the waypoint P(j) corresponding to the vehicle after time t may be estimated according to the current speed of the vehicle, where t and j may be positive integers.
  • the value of t can be preset, for example, t can be set to 10 seconds; or, the value of t can be determined according to the position of the vehicle and the road on which it travels, for example, whether the vehicle is located on a congested road in the city, Or, if the vehicle is traveling on a highway.
  • S1250 Calculate the change in the heading angle of the vehicle based on P(i) and P(j).
  • the driving intention determined by the first method may be a semantic indication (eg, discrete driving intention), which may include two types:
  • Road-level driving intent can correspond to road-level navigation, including "turn left”, “turn right”, “go straight”, “stop”, etc.;
  • Lane-level driving intent can correspond to lane-level navigation, and can include the lane number to be driven, such as "first lane on the left” and “two lanes on the right”.
  • whether the vehicle deviates from the planned route may be determined according to the current position of the vehicle.
  • the route can be re-planned to obtain the waypoint sequence of the road to be traveled; otherwise, the waypoint sequence of the original road to be traveled can continue to be used, that is, the waypoint sequence of the original road to be traveled is regarded as the roadpoint sequence to be traveled.
  • a sequence of waypoints for the driving road is regarded as the roadpoint sequence to be traveled.
  • the waypoint sequence of the road to be traveled can be represented by ⁇ P(1), P(2), P(3), ..., P(N) ⁇ , where P(k) represents the position of the kth waypoint information, k and N are both positive integers.
  • the current position of the vehicle may be projected into the waypoint sequence of the road to be traveled determined in S1220, and the point closest to the current position of the vehicle in the waypoint sequence of the road to be traveled may be projected.
  • the map road network around the road point P(i) currently corresponding to the vehicle and within the radius D range may be used as a local map.
  • D is a positive integer
  • the value of D can be determined according to the current speed of the vehicle. For example, when the speed of the vehicle is low, the value of D can be 20 meters.
  • a waypoint P(i+m) corresponding to the vehicle after time t may be estimated according to the current speed of the vehicle, where t, i and m are all positive integers.
  • the value of t can be preset, for example, t can be set to 10 seconds; or, the value of t can be determined according to the position of the vehicle and the road on which it travels, for example, whether the vehicle is located on a congested road in the city, Or, if the vehicle is traveling on a highway.
  • a waypoint P(i-n) corresponding to the vehicle before time t may be estimated according to the current speed of the vehicle, where n is a positive integer.
  • ⁇ P(i),...,P(i+m) ⁇ can be drawn into the local map as the waypoints to be traveled, and ⁇ P(in),...,P(i) ⁇ As historical waypoints (ie, traveled waypoints), they are drawn into the local map.
  • the marking of the roadpoint to be traveled in the partial map is different from the marking of the historical roadpoint in the partial map, for example, the marking color and the marking type are different.
  • the thin solid line represents the waypoint to be traveled
  • the dashed line represents the historical waypoint
  • the thick solid line represents the road.
  • the driving intention determined by the second method may be a schematic diagram of a route (for example, a schematic diagram of a partial route), and the schematic diagram of the route may include the historical path and the to-be-driving path of the vehicle, for example, A, B, and Schematic diagram of three routes.
  • a route for example, a schematic diagram of a partial route
  • the schematic diagram of the route may include the historical path and the to-be-driving path of the vehicle, for example, A, B, and Schematic diagram of three routes.
  • part A represents a schematic diagram of going straight on a straight road, and the to-be-driving route is a straight-forward route; part B represents that the vehicle has traveled a right-turn route, and the to-be-traveled route is a straight-forward route; The driving route is a right turn route.
  • S840 use a preset neural network model to perform lane inference based on the environment perception image and the driving intention to obtain the lane of the vehicle.
  • the lane of the vehicle can be expressed in any of the following ways:
  • Lane centerline and width Set the length of the lane according to the vehicle's environment (ie, the vehicle's field of view and system requirements).
  • the length of the lane can be 20m. If sampling is performed at a distance of one point per meter, 20 position points are required to represent the lane, and each position point includes three values. , corresponding to the abscissa, ordinate and width of the resting point, respectively.
  • Cost grid map The area around the vehicle can be divided into P*Q grids, and the lane of the vehicle can also be represented by a plurality of grids in the P*Q grids, where P and Q are both is a positive integer.
  • the resolution of the cost grid map that is, the size of each grid can be designed to be 20cm*20cm, or 10cm*10cm.
  • Each grid has a value in the interval [0,1] to represent its cost value.
  • the neural network model may be obtained after training based on an offline-collected environment perception image and driving intention.
  • the environment perception image determined in S820 and the driving intention determined in S830 may be used as input, and the pre-trained neural network model may be used to perform lane inference to obtain the lane of the vehicle.
  • the environment perception image determined in the scheme 1 in S820 and the driving intention determined in the method 1 in S830 can be used as input, and the output is the center line of the lane and the width of the lane.
  • the structure can be as shown in Figure 15.
  • the neural network model can use the ResNet50 network to extract the features in the environment-aware image to obtain the environment-aware features, and the decoder can be used to decode the driving intention into driving intention features (ie discrete driving intentions).
  • the intent feature can use a four-dimensional vector to correspond to the four discrete driving intents of "turn left”, “turn right”, “go straight”, and "stop”;
  • the neural network model can use a concatenate layer to connect the environmental perception features
  • the driving intent feature the connected feature is obtained, and the connected feature is input into the fully connected layer; the fully connected layer can generate 4 parallel fully connected models (here "4" is the number of discrete driving intentions ), in which each fully connected model has two compression layers, which highly compress the features.
  • the depth of the features output by the fully connected layer (that is, 4 parallel fully connected models) is reduced to 4.
  • each compression layer A dropout can be performed once before, and the purpose of preventing overfitting of features is achieved by shielding some connection elements;
  • the neural network model outputs the features from the fully connected layer according to the input intent (that is, four parallel fully connected layers). model), select the predicted output corresponding to the input intent, that is, one of the fully connected models, and the fully connected model is the inferred lane.
  • the environment perception image determined in the second solution in S820 and the driving intention determined in the second method in S830 can be used as input, and the output is the center line of the lane and the width of the lane.
  • the neural network model The structure can be shown in Figure 16.
  • the neural network model can use the ResNet50 network to extract the features in the environment perception image to obtain the environment perception features.
  • the ResNet50 network can also be used to extract the features in the driving intention to obtain the driving intention features;
  • connection layer concatenate layer
  • the fully connected layer has two compression layers, which can highly compress the features. Finally, the depth of the features output by the fully connected layer is reduced to 1. At the same time, a dropout can be performed before each compression layer.
  • the connection element achieves the purpose of preventing feature overfitting; finally, the fully connected layer outputs the lanes obtained by inference.
  • FIG. 17 is a schematic flowchart of a method 1700 for training a lane inference model provided by an embodiment of the present application.
  • the method 1700 shown in FIG. 17 can be executed by the processor 113 in the vehicle 100 in FIG. 1 , or can also be executed by the processor 103 in the automatic driving system in FIG.
  • the processor 320 in the server 320 executes.
  • the method 1700 shown in FIG. 17 may include steps 1710, 1720 and 1730. It should be understood that the method 1700 shown in FIG. 17 is only an example and not a limitation, and the method 1700 may include more or less steps. This is not limited in the embodiment, and the steps are described in detail below.
  • S1720 Perform coordinate system transformation on the detection result to obtain traffic element data located in the same coordinate system.
  • the traffic element data may be data in the same coordinate system.
  • S1730 use a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.
  • the acquiring traffic element data detected by a variety of different types of sensors of the vehicle may include: acquiring detection results of the various types of sensors; processing the detection results to obtain the traffic elements data.
  • the processing of the detection result to obtain the traffic element data may include: performing coordinate system transformation on the detection result to obtain the traffic element data.
  • the various types of sensors may include cameras, lidars, or millimeter wave radars.
  • the traffic element data may be the outline, position and/or motion trajectory of the traffic element, and the traffic element may include lane markings, other vehicles around the vehicle, pedestrians, roads or traffic lights. one or more of.
  • the method may further include: determining an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the traffic element, and the environment-aware image includes a look-around image or at least one of front-view images; wherein, using a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle may include: using the lane inference model, based on the environment Lane inference is performed on the perceptual image to obtain the lane of the vehicle.
  • the using the lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle may include: using the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle. Lane reasoning to obtain the lane of the vehicle, and the driving intention of the vehicle is used to indicate the route to be driven by the vehicle.
  • the driving intention of the vehicle may be a schematic diagram of a route of a lane to be driven by the vehicle or a vehicle semantic indication, and the vehicle semantic indication may be used to indicate the driving direction of the vehicle, a driving lane or a driving intersection. at least one of.
  • the lane of the vehicle may be represented by the centerline of the lane and the width of the lane, or the lane of the vehicle may be represented by a road cost map.
  • FIG. 18 is a schematic block diagram of an apparatus 1800 for inferring a lane provided by an embodiment of the present application. It should be understood that the apparatus 1800 for inferring a lane shown in FIG. 18 is only an example, and the apparatus 1800 in this embodiment of the present application may further include other modules or units. It should be understood that the behavior planning apparatus 1800 can execute each step in the method of FIG. 7 or FIG. 8 , and in order to avoid repetition, details are not described here.
  • the acquiring unit 1810 is configured to acquire detection results of various types of sensors of the vehicle.
  • the conversion unit 1820 is configured to perform coordinate system conversion on the detection result to obtain traffic element data located in the same coordinate system.
  • the lane inference unit 1830 is configured to use a preset neural network model to perform lane inference based on the traffic element data to obtain the lane of the vehicle.
  • the acquiring unit 1810 is specifically configured to: acquire detection results of the various types of sensors; and process the detection results to obtain the traffic element data.
  • the obtaining unit 1810 is specifically configured to: perform coordinate system transformation on the detection result to obtain the traffic element data.
  • the plurality of different types of sensors include cameras, lidars or millimeter wave radars.
  • the traffic element data is the outline, position and/or motion trajectory of the traffic element, and the traffic element includes one of lane markings, other vehicles around the vehicle, pedestrians, roads or traffic lights, or variety.
  • the apparatus 1800 further includes a superimposing unit 1840, configured to: determine an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the traffic element, the The environment-aware image includes at least one of a look-around image or a front-view image; wherein the lane inference unit 1830 is specifically configured to: use a preset neural network model to perform lane inference based on the environment-aware image to obtain the vehicle lane.
  • a superimposing unit 1840 configured to: determine an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the traffic element, the The environment-aware image includes at least one of a look-around image or a front-view image; wherein the lane inference unit 1830 is specifically configured to: use a preset neural network model to perform lane inference based on the environment-aware image to obtain the vehicle lane.
  • the lane inference unit 1830 is specifically configured to: use a preset neural network model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, the driving intention of the vehicle Used to indicate the route the vehicle is to travel.
  • the driving intention of the vehicle is a schematic route diagram of a lane where the vehicle is to be driven or a vehicle semantic indication, where the vehicle semantic indication is used to indicate at least one of a driving direction, a driving lane or a driving intersection of the vehicle. kind.
  • the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road cost map.
  • module here can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a “module” may be a software program, a hardware circuit, or a combination of the two that implement the above-mentioned functions.
  • the hardware circuits may include application specific integrated circuits (ASICs), electronic circuits, processors for executing one or more software or firmware programs (eg, shared processors, proprietary processors, or group processors) etc.) and memory, merge logic and/or other suitable components to support the described functions.
  • ASICs application specific integrated circuits
  • processors for executing one or more software or firmware programs (eg, shared processors, proprietary processors, or group processors) etc.) and memory, merge logic and/or other suitable components to support the described functions.
  • the apparatus 1800 for inferring a lane provided in this embodiment of the present application may be an in-vehicle device of an autonomous driving vehicle, or may be a chip configured in the in-vehicle device, so as to execute the method described in the embodiment of this application.
  • FIG. 19 is a schematic block diagram of an apparatus 1900 for training a lane inference model provided by an embodiment of the present application. It should be understood that the apparatus 1900 for training a lane inference model shown in FIG. 19 is only an example, and the apparatus 1900 in this embodiment of the present application may further include other modules or units. It should be understood that the behavior planning apparatus 1900 is capable of executing various steps in the method of FIG. 17 , which will not be described in detail here in order to avoid repetition.
  • the acquiring unit 1910 is configured to acquire detection results of various types of sensors of the vehicle.
  • the conversion unit 1920 is configured to perform coordinate system conversion on the detection result to obtain traffic element data located in the same coordinate system.
  • a lane inference unit 1930 configured to use a lane inference model to perform lane inference based on the traffic element data to obtain the lane of the vehicle;
  • the adjusting unit 1940 is configured to adjust the parameters of the lane inference model according to the lane of the vehicle.
  • the acquiring unit 1910 is specifically configured to: acquire detection results of the various types of sensors; and process the detection results to obtain the traffic element data.
  • the acquiring unit 1910 is specifically configured to: perform coordinate system transformation on the detection result to obtain the traffic element data.
  • the plurality of different types of sensors include cameras, lidars or millimeter wave radars.
  • the traffic element data is the outline, position and/or motion trajectory of the traffic element, and the traffic element includes one of lane markings, other vehicles around the vehicle, pedestrians, roads or traffic lights, or variety.
  • the apparatus 1900 further includes a superimposing unit 1950, configured to: determine an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the traffic element, the The environment-aware image includes at least one of a look-around image or a front-view image; wherein, the lane inference unit 1930 is specifically configured to: use the lane inference model to perform lane inference based on the environment-aware image to obtain the vehicle's Lane.
  • a superimposing unit 1950 configured to: determine an environment-aware image according to the traffic element data, where the environment-aware image is used to indicate the environment around the vehicle and the traffic element, the The environment-aware image includes at least one of a look-around image or a front-view image; wherein, the lane inference unit 1930 is specifically configured to: use the lane inference model to perform lane inference based on the environment-aware image to obtain the vehicle's Lane.
  • the lane inference unit 1930 is specifically configured to: use the lane inference model to perform lane inference based on the traffic element data and the driving intention of the vehicle to obtain the lane of the vehicle, and the driving intention of the vehicle is determined by using the lane inference model. to indicate the route on which the vehicle is to travel.
  • the driving intention of the vehicle is a schematic route diagram of a lane where the vehicle is to be driven or a vehicle semantic indication, where the vehicle semantic indication is used to indicate at least one of a driving direction, a driving lane or a driving intersection of the vehicle. kind.
  • the lane of the vehicle is represented by the center line of the lane and the width of the lane, or the lane of the vehicle is represented by a road cost map.
  • module here can be implemented in the form of software and/or hardware, which is not specifically limited.
  • a “module” may be a software program, a hardware circuit, or a combination of the two that implement the above-mentioned functions.
  • the hardware circuits may include application specific integrated circuits (ASICs), electronic circuits, processors for executing one or more software or firmware programs (eg, shared processors, proprietary processors, or group processors) etc.) and memory, merge logic and/or other suitable components to support the described functions.
  • ASICs application specific integrated circuits
  • processors for executing one or more software or firmware programs (eg, shared processors, proprietary processors, or group processors) etc.) and memory, merge logic and/or other suitable components to support the described functions.
  • the apparatus 1900 for training a lane inference model provided in this embodiment of the present application may be an in-vehicle device of an autonomous vehicle, or may be a chip configured in the in-vehicle device, so as to execute the method described in the embodiment of this application. .
  • FIG. 20 is a schematic block diagram of an apparatus 800 according to an embodiment of the present application.
  • the apparatus 800 shown in FIG. 20 includes a memory 801 , a processor 802 , a communication interface 803 and a bus 804 .
  • the memory 801 , the processor 802 , and the communication interface 803 are connected to each other through the bus 804 for communication.
  • the memory 801 may be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM).
  • the memory 801 may store a program.
  • the processor 802 is configured to execute the various steps of the method for planning an inference lane or the method for training a lane inference model according to the embodiment of the present application. Each step of the embodiment shown in FIG. 7 , FIG. 8 or FIG. 17 is performed.
  • the processor 802 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for executing related programs to The method for inferring a lane or the method for training a lane inference model according to the method embodiment of the present application is implemented.
  • CPU central processing unit
  • ASIC application-specific integrated circuit
  • the processor 802 may also be an integrated circuit chip with signal processing capability.
  • each step of the method for planning an autonomous driving vehicle in the embodiment of the present application may be completed by an integrated logic circuit of hardware in the processor 802 or instructions in the form of software.
  • the above-mentioned processor 802 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, Discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processor
  • ASIC application-specific integrated circuit
  • FPGA field programmable gate array
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor.
  • the software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art.
  • the storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801 and, in combination with its hardware, completes the functions required to be performed by the units included in the behavior planning apparatus of the autonomous driving vehicle in the embodiment of the present application, or executes the method of the present application.
  • the method for planning an autonomous vehicle may perform various steps/functions of the embodiments shown in FIG. 7 , FIG. 8 , or FIG. 17 .
  • the communication interface 803 may use, but is not limited to, a transceiver such as a transceiver to implement communication between the apparatus 800 and other devices or a communication network.
  • the bus 804 may include a pathway for communicating information between the various components of the apparatus 800 (eg, the memory 801, the processor 802, the communication interface 803).
  • the apparatus 800 shown in this embodiment of the present application may be an in-vehicle device in an autonomous vehicle, or may also be a chip configured in the in-vehicle device.
  • the processor in the embodiment of the present application may be a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), application-specific integrated circuits (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the memory in the embodiments of the present application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically programmable Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory.
  • Volatile memory may be random access memory (RAM), which acts as an external cache.
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • DRAM synchronous dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • DDR SDRAM double data rate synchronous dynamic random access memory
  • enhanced SDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous connection dynamic random access memory Fetch memory
  • direct memory bus random access memory direct rambus RAM, DR RAM
  • the above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination.
  • the above-described embodiments may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.
  • the computer may be a general purpose computer, special purpose computer, computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center by wire (eg, infrared, wireless, microwave, etc.).
  • the computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server, a data center, or the like containing one or more sets of available media.
  • the usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media.
  • the semiconductor medium may be a solid state drive.
  • At least one means one or more, and “plurality” means two or more.
  • At least one item(s) below” or similar expressions thereof refer to any combination of these items, including any combination of single item(s) or plural items(s).
  • at least one item (a) of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple .
  • the size of the sequence numbers of the above-mentioned processes does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not be dealt with in the embodiments of the present application. implementation constitutes any limitation.
  • the disclosed system, apparatus and method may be implemented in other manners.
  • the apparatus embodiments described above are only illustrative.
  • the division of the units is only a logical function division. In actual implementation, there may be other division methods.
  • multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium.
  • the technical solution of the present application can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution.
  • the computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program codes .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Multimedia (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Traffic Control Systems (AREA)

Abstract

一种推理车道的方法、训练车道推理模型的方法及装置，涉及人工智能领域中自动驾驶领域的环境认知技术。该方法包括：获取车辆的多种不同类型的传感器的检测结果(710)；对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据(720)；使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道(730)。上述方法，有助于降低用于车道推理的神经网络模型的复杂度。

Description

推理车道的方法、训练车道推理模型的方法及装置
本申请要求于2020年07月02日提交中国专利局、申请号为202010626061.4、申请名称为“推理车道的方法、训练车道推理模型的方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及自动驾驶领域，并且更具体地，涉及一种推理车道的方法、训练车道推理模型的方法及装置。
背景技术
人工智能(artificial intelligence，AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。人工智能领域的研究包括机器人，自然语言处理，计算机视觉，决策与推理，人机交互，推荐与搜索，AI基础理论等。
自动驾驶是人工智能领域的一种主流应用，自动驾驶技术依靠计算机视觉、雷达、监控装置和全球定位系统等协同合作，让机动车辆可以在不需要人类主动操作下，实现自动驾驶。自动驾驶的车辆使用各种计算系统来帮助将乘客或货物从一个位置运输到另一位置。一些自动驾驶车辆可能要求来自操作者(诸如，领航员、驾驶员、或者乘客)的一些初始输入或者连续输入。自动驾驶车辆允许操作者从手动模操作式切换到自动驾驶模式或者介于两者之间的模式。由于自动驾驶技术无需人类来驾驶机动车辆，所以理论上能够有效避免人类的驾驶失误，减少交通事故的发生，且能够提高公路的运输效率。因此，自动驾驶技术越来越受到重视。
车辆能否获得在自动驾驶时可用的准确可选的车道，是影响自动驾驶安全性的关键因素。例如，可以在离线环境采集训练数据训练神经网络模型，并使用训练好的神经网络模型进行车道推理。但是，目前训练好的神经网络模型往往较为复杂。
发明内容
本申请提供一种推理车道的方法、训练车道推理模型的方法及装置，有助于降低用于车道推理的神经网络模型的复杂度。
第一方面，提供了一种推理车道的方法，该方法包括：
获取车辆的多种不同类型的传感器的检测结果；对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；使用预设的神经网络模型，基于所述交通元素数 据进行车道推理，得到所述车辆的车道。
在现有技术中，当神经网络模型需要实现的功能较多时，例如，对光线条件、路面类型等泛化能力，以及路面认知、路面与驾驶意图的匹配、对动态障碍物及静态障碍物的处理都依靠同一个神经网络模型进行处理，可能会导致神经网络模型过于复杂。
对于这种复杂度较高的神经网络模型，其模型的设计要求也非常高，目前很难设计出一个达到实用性水平的能够处理复杂任务的神经网络模型(例如，能够处理复杂自动驾驶任务的神经网络模型)。
在本申请实施例中，所述交通元素数据为同一个坐标系下的数据，可以避免因需要处理多种不同类型的数据而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
需要说明的是，上述获取多种不同类型的传感器检测到的交通元素数据可以包括：直接获取传感器检测到的原始数据，也就是说，此时，所述检测到的原始数据就是所述交通元素数据。
或者，上述获取多种不同类型的传感器检测到的交通元素数据也可以包括：获取传感器检测到的检测结果(例如，传感器获取到的原始数据或原始图像等)后，对所述检测到的检测结果进行处理后得到处理结果，此时，所述处理后得到的处理结果就是所述交通元素数据。
结合第一方面，在第一方面的某些实现方式中，所述获取车辆的多种不同类型的传感器检测到的交通元素数据，包括：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
在本申请实施例中，获取所述多种不同类型的传感器的检测结果，并对所述检测结果进行处理，得到同一个坐标系下的所述交通元素数据，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
结合第一方面，在第一方面的某些实现方式中，所述对所述检测结果进行处理得到所述交通元素数据，包括：对所述检测结果进行坐标系转换，得到所述交通元素数据。
在本申请实施例中，对所述检测结果进行坐标系转换，可以将所述检测结果转换到同一个坐标系下，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
结合第一方面，在第一方面的某些实现方式中，所述多种不同类型的传感器包括摄像头、激光雷达或毫米波雷达。
结合第一方面，在第一方面的某些实现方式中，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
结合第一方面，在第一方面的某些实现方式中，所述方法还包括：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：使用预 设的神经网络模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
在本申请实施例中，通过所述环境感知图像可以直观的表示所述车辆周围的环境，此时，基于所述环境感知图像进行车道推理，可以使得推理得到(所述车辆的)车道更加准确。
可选地，所述根据所述交通元素数据确定环境感知图像包括：基于所述交通元素数据，将所述交通元素叠加到初始图像(例如，对传感器采集的一帧或多帧感知图像进行拼接后得到的环视图像，或生成的以所述车辆为中心、环境空白的初始环境图像)中，得到所述环境感知图像。
结合第一方面，在第一方面的某些实现方式中，所述使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：使用预设的神经网络模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
在本申请实施例中，所述车辆的驾驶意图用于指示所述车辆待行驶的路线，基于所述交通元素数据和所述车辆的驾驶意图进行车道推理，可以进一步提高推理得到(所述车辆的)车道的准确性。
结合第一方面，在第一方面的某些实现方式中，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
在本申请实施例中，所述车辆的驾驶意图为路线示意图或车辆语义指示，通过所述车辆的驾驶意图可以直观地和准确地指示所述车辆待行驶的路线。
结合第一方面，在第一方面的某些实现方式中，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
在本申请实施例中，通过所述车道的中心线和所述车道的宽度或者路面代价图表示，可以直观地和准确地表示所述车辆的车道。
第二方面，提供了一种训练车道推理模型的方法，该方法包括：
获取车辆的多种不同类型的传感器的检测结果；对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道；根据所述车辆的车道，调整所述车道推理模型的参数。
在现有技术中，当神经网络模型需要实现的功能较多时，例如，对光线条件、路面类型等泛化能力，以及路面认知、路面与驾驶意图的匹配、对动态障碍物及静态障碍物的处理都依靠同一个神经网络模型进行处理，可能会导致神经网络模型过于复杂。
对于这种复杂度较高的神经网络模型，其模型的设计要求也非常高，目前很难设计出一个达到实用性水平的能够处理复杂任务的神经网络模型(例如，能够处理复杂自动驾驶任务的神经网络模型)。
在本申请实施例中，所述交通元素数据为同一个坐标系下的数据，可以避免因需要处理多种不同类型的数据而导致所述车道推理模型过于复杂，从而能够降低所述车道推理模型的复杂度。
需要说明的是，上述获取多种不同类型的传感器检测到的交通元素数据可以包括：直接获取传感器检测到的原始数据，也就是说，此时，所述检测到的原始数据就是所述交通 元素数据。
或者，上述获取多种不同类型的传感器检测到的交通元素数据也可以包括：获取传感器检测到的检测结果(例如，传感器获取到的原始数据或原始图像等)后，对所述检测到的检测结果进行处理后得到处理结果，此时，所述处理后得到的处理结果就是所述交通元素数据。
结合第二方面，在第二方面的某些实现方式中，所述获取车辆的多种不同类型的传感器检测到的交通元素数据，包括：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
在本申请实施例中，获取所述多种不同类型的传感器的检测结果，并对所述检测结果进行处理，得到同一个坐标系下的所述交通元素数据，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致所述车道推理模型过于复杂，从而能够降低所述车道推理模型的复杂度。
结合第二方面，在第二方面的某些实现方式中，所述对所述检测结果进行处理得到所述交通元素数据，包括：对所述检测结果进行坐标系转换，得到所述交通元素数据。
在本申请实施例中，对所述检测结果进行坐标系转换，可以将所述检测结果转换到同一个坐标系下，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致所述车道推理模型过于复杂，从而能够降低所述车道推理模型的复杂度。
结合第二方面，在第二方面的某些实现方式中，所述多种不同类型的传感器包括摄像头、激光雷达或毫米波雷达。
结合第二方面，在第二方面的某些实现方式中，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：使用所述车道推理模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
在本申请实施例中，通过所述环境感知图像可以直观的表示所述车辆周围的环境，此时，基于所述环境感知图像进行车道推理，可以使得推理得到(所述车辆的)车道更加准确。
可选地，所述根据所述交通元素数据确定环境感知图像包括：基于所述交通元素数据，将所述交通元素叠加到初始图像(例如，对传感器采集的一帧或多帧感知图像进行拼接后得到的环视图像，或生成的以所述车辆为中心、环境空白的初始环境图像)中，得到所述环境感知图像。
结合第二方面，在第二方面的某些实现方式中，所述使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：使用所述车道推理模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
在本申请实施例中，所述车辆的驾驶意图用于指示所述车辆待行驶的路线，基于所述 交通元素数据和所述车辆的驾驶意图进行车道推理，可以进一步提高推理得到(所述车辆的)车道的准确性。
结合第二方面，在第二方面的某些实现方式中，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
在本申请实施例中，所述车辆的驾驶意图为路线示意图或车辆语义指示，通过所述车辆的驾驶意图可以直观地和准确地指示所述车辆待行驶的路线。
结合第二方面，在第二方面的某些实现方式中，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
在本申请实施例中，通过所述车道的中心线和所述车道的宽度或者路面代价图表示，可以直观地和准确地表示所述车辆的车道。
第三方面，提供了一种推理车道的装置，包括：
获取单元，用于获取车辆的多种不同类型的传感器的检测结果；对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；车道推理单元，用于使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道。
在本申请实施例中，所述交通元素数据为同一个坐标系下的数据，可以避免因需要处理多种不同类型的数据而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
需要说明的是，上述获取多种不同类型的传感器检测到的交通元素数据可以包括：直接获取传感器检测到的原始数据，此时，所述检测到的原始数据就是所述交通元素数据。
或者，上述获取多种不同类型的传感器检测到的交通元素数据也可以包括：获取传感器检测到的检测结果(例如，传感器获取到的原始数据或原始图像等)后，对所述检测到的检测结果进行处理后得到处理结果，此时，所述处理后得到的处理结果就是所述交通元素数据。
结合第三方面，在第三方面的某些实现方式中，所述获取单元具体用于：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
在本申请实施例中，获取所述多种不同类型的传感器的检测结果，并对所述检测结果进行处理，得到同一个坐标系下的所述交通元素数据，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
结合第三方面，在第三方面的某些实现方式中，所述获取单元具体用于：对所述检测结果进行坐标系转换，得到所述交通元素数据。
在本申请实施例中，对所述检测结果进行坐标系转换，可以将所述检测结果转换到同一个坐标系下，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
结合第三方面，在第三方面的某些实现方式中，所述多种不同类型的传感器包括摄像头、激光雷达或毫米波雷达。
结合第三方面，在第三方面的某些实现方式中，所述交通元素数据为交通元素的轮廓、 位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
结合第三方面，在第三方面的某些实现方式中，所述装置还包括叠加单元，用于：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述车道推理单元具体用于：使用预设的神经网络模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
在本申请实施例中，通过所述环境感知图像可以直观的表示所述车辆周围的环境，此时，基于所述环境感知图像进行车道推理，可以使得推理得到(所述车辆的)车道更加准确。
结合第三方面，在第三方面的某些实现方式中，所述车道推理单元具体用于：使用预设的神经网络模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
在本申请实施例中，所述车辆的驾驶意图用于指示所述车辆待行驶的路线，基于所述交通元素数据和所述车辆的驾驶意图进行车道推理，可以进一步提高推理得到(所述车辆的)车道的准确性。
结合第三方面，在第三方面的某些实现方式中，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
在本申请实施例中，所述车辆的驾驶意图为路线示意图或车辆语义指示，通过所述车辆的驾驶意图可以直观地和准确地指示所述车辆待行驶的路线。
结合第三方面，在第三方面的某些实现方式中，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
在本申请实施例中，通过所述车道的中心线和所述车道的宽度或者路面代价图表示，可以直观地和准确地表示所述车辆的车道。
第四方面，提供了一种训练车道推理模型的装置，包括：
获取单元，用于获取车辆的多种不同类型的传感器的检测结果；对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；车道推理单元，用于使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道；调整单元，用于根据所述车辆的车道，调整所述车道推理模型的参数。
在本申请实施例中，所述交通元素数据为同一个坐标系下的数据，可以避免因需要处理多种不同类型的数据而导致所述车道推理模型过于复杂，从而能够降低所述车道推理模型的复杂度。
需要说明的是，上述获取多种不同类型的传感器检测到的交通元素数据可以包括：直接获取传感器检测到的原始数据，也就是说，此时，所述检测到的原始数据就是所述交通元素数据。
或者，上述获取多种不同类型的传感器检测到的交通元素数据也可以包括：获取传感器检测到的检测结果(例如，传感器获取到的原始数据或原始图像等)后，对所述检测到的检测结果进行处理后得到处理结果，此时，所述处理后得到的处理结果就是所述交通元 素数据。
结合第四方面，在第四方面的某些实现方式中，所述获取单元具体用于：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
在本申请实施例中，获取所述多种不同类型的传感器的检测结果，并对所述检测结果进行处理，得到同一个坐标系下的所述交通元素数据，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致所述车道推理模型过于复杂，从而能够降低所述车道推理模型的复杂度。
结合第四方面，在第四方面的某些实现方式中，所述获取单元具体用于：对所述检测结果进行坐标系转换，得到所述交通元素数据。
在本申请实施例中，对所述检测结果进行坐标系转换，可以将所述检测结果转换到同一个坐标系下，可以避免因需要直接处理所述检测结果(所述检测结果包含多种不同类型的数据)而导致所述车道推理模型过于复杂，从而能够降低所述车道推理模型的复杂度。
结合第四方面，在第四方面的某些实现方式中，所述多种不同类型的传感器包括摄像头、激光雷达或毫米波雷达。
结合第四方面，在第四方面的某些实现方式中，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
结合第四方面，在第四方面的某些实现方式中，所述装置还包括叠加单元，用于：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述车道推理单元具体用于：使用所述车道推理模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
在本申请实施例中，通过所述环境感知图像可以直观的表示所述车辆周围的环境，此时，基于所述环境感知图像进行车道推理，可以使得推理得到(所述车辆的)车道更加准确。
结合第四方面，在第四方面的某些实现方式中，所述车道推理单元具体用于：使用所述车道推理模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
在本申请实施例中，所述车辆的驾驶意图用于指示所述车辆待行驶的路线，基于所述交通元素数据和所述车辆的驾驶意图进行车道推理，可以进一步提高推理得到(所述车辆的)车道的准确性。
结合第四方面，在第四方面的某些实现方式中，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
在本申请实施例中，所述车辆的驾驶意图为路线示意图或车辆语义指示，通过所述车辆的驾驶意图可以直观地和准确地指示所述车辆待行驶的路线。
结合第四方面，在第四方面的某些实现方式中，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
在本申请实施例中，通过所述车道的中心线和所述车道的宽度或者路面代价图表示， 可以直观地和准确地表示所述车辆的车道。
第五方面，提供了一种推理车道的装置，所述装置包括存储介质和中央处理器，所述存储介质可以是非易失性存储介质，所述存储介质中存储有计算机可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述计算机可执行程序以实现所述第一方面中的任一可能的实现方式中的方法。
第六方面，提供了一种训练车道推理模型的装置，所述装置包括存储介质和中央处理器，所述存储介质可以是非易失性存储介质，所述存储介质中存储有计算机可执行程序，所述中央处理器与所述非易失性存储介质连接，并执行所述计算机可执行程序以实现所述第二方面中的任一可能的实现方式中的方法。
第七方面，提供一种芯片，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，执行第一方面中或第二方面中的任一可能的实现方式中的方法。
可选地，作为一种实现方式，所述芯片还可以包括存储器，所述存储器中存储有指令，所述处理器用于执行所述存储器上存储的指令，当所述指令被执行时，所述处理器用于执行第一方面中或第二方面中的任一可能的实现方式中的方法。
第八方面，提供一种计算机可读存储介质，所述计算机可读介质存储用于设备执行的程序代码，所述程序代码包括用于执行第一方面中或第二方面中的任一可能的实现方式中的方法的指令。
第九方面，提供一种汽车，所述汽车包括上述第三方面中或第四方面中的任一可能的装置。
在本申请实施例中，所述交通元素数据为同一个坐标系下的数据，可以避免因需要处理多种不同类型的数据而导致神经网络模型过于复杂，从而能够降低用于车道推理的神经网络模型的复杂度。
附图说明
图1为本申请实施例提供的一种自动驾驶汽车的结构示意图；
图2为本申请实施例提供的一种自动驾驶系统的结构示意图；
图3为本申请实施例提供的一种神经网络处理器的结构示意图；
图4为本申请实施例提供的一种云侧指令自动驾驶汽车的应用示意图；
图5为本申请实施例提供的一种自动驾驶系统的结构示意图；
图6为本申请实施例提供的一种道路结构认知模块的示意性框图；
图7为本申请一个实施例提供的推理车道的方法的示意性框图；
图8为本申请另一个实施例提供的推理车道的方法的示意性流程图；
图9为本申请一个实施例中对传感器的检测结果进行处理的示意性流程图；
图10为本申请一个实施例中的确定环境感知图像的示意性流程图；
图11为本申请另一个实施例中的确定环境感知图像的示意性流程图；
图12为本申请一个实施例中的确定车辆的驾驶意图的示意性流程图；
图13为本申请另一个实施例中的确定车辆的驾驶意图的示意性流程图；
图14为本申请一个实施例中生成的路线示意图的示意性框图；
图15为本申请一个实施例中使用神经网络模型进行车道推理的示意性框图；
图16为本申请另一个实施例中使用神经网络模型进行车道推理的示意性框图；
图17为本申请一个实施例提供的训练车道推理模型的方法的示意性框图；
图18是本申请实施例提供的推理车道的装置的示意性框图；
图19是本申请实施例提供的训练驾驶行为决策模型的装置的示意性框图；
图20是本申请实施例提供的一种用于自动驾驶车辆的装置的示意性框图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
本申请实施例的技术方案可以应用于各种车辆，该车辆具体可以为内燃机车、智能电动车或者混合动力车，或者，该车辆也可以为其他动力类型的车辆等，本申请实施例对此并不限定。
本申请实施例中的车辆可以为自动驾驶车辆，例如，所述自动驾驶车辆可以配置有自动驾驶模式，该自动驾驶模式可以为完全自动驾驶模式，或者，也可以为部分自动驾驶模式，本申请实施例对此并不限定。
本申请实施例中的车辆还可以配置有其他驾驶模式，所述其他驾驶模式可以包括运动模式、经济模式、标准模式、雪地模式及爬坡模式等多种驾驶模式中的一种或多种。所述自动驾驶车辆可以在自动驾驶模式和上述多种(驾驶员驾驶车辆的)驾驶模型之间进行切换，本申请实施例对此并不限定。
图1是本申请实施例提供的车辆100的功能框图。
在一个实施例中，将车辆100配置为完全或部分地自动驾驶模式。
例如，车辆100可以在处于自动驾驶模式中的同时控制自身，并且可通过人为操作来确定车辆及其周边环境的当前状态，确定周边环境中的至少一个其他车辆的可能行为，并确定该其他车辆执行可能行为的可能性相对应的置信水平，基于所确定的信息来控制车辆100。在车辆100处于自动驾驶模式中时，可以将车辆100置为在没有和人交互的情况下操作。
车辆100可包括各种子系统，例如行进系统102、传感器系统104、控制系统106、一个或多个外围设备108以及电源110、计算机系统112和用户接口116。
可选地，车辆100可包括更多或更少的子系统，并且每个子系统可包括多个元件。另外，车辆100的每个子系统和元件可以通过有线或者无线互连。
行进系统102可包括为车辆100提供动力运动的组件。在一个实施例中，推进系统102可包括引擎118、能量源119、传动装置120和车轮/轮胎121。引擎118可以是内燃引擎、电动机、空气压缩引擎或其他类型的引擎组合，例如，气油发动机和电动机组成的混动引擎，内燃引擎和空气压缩引擎组成的混动引擎。引擎118将能量源119转换成机械能量。
能量源119的示例包括汽油、柴油、其他基于石油的燃料、丙烷、其他基于压缩气体的燃料、乙醇、太阳能电池板、电池和其他电力来源。能量源119也可以为车辆100的其他系统提供能量。
传动装置120可以将来自引擎118的机械动力传送到车轮121。传动装置120可包括 变速箱、差速器和驱动轴。
在一个实施例中，传动装置120还可以包括其他器件，比如离合器。其中，驱动轴可包括可耦合到一个或多个车轮121的一个或多个轴。
传感器系统104可包括感测关于车辆100周边的环境的信息的若干个传感器。
例如，传感器系统104可包括定位系统122(定位系统可以是GPS系统，也可以是北斗系统或者其他定位系统)、惯性测量单元(inertial measurement unit，IMU)124、雷达126、激光测距仪128以及相机130。传感器系统104还可包括被监视车辆100的内部系统的传感器(例如，车内空气质量监测器、燃油量表、机油温度表等)。来自这些传感器中的一个或多个的传感器数据可用于检测对象及其相应特性(位置、形状、方向、速度等)。这种检测和识别是自主车辆100的安全操作的关键功能。
定位系统122可用于估计车辆100的地理位置。IMU 124用于基于惯性加速度来感测车辆100的位置和朝向变化。在一个实施例中，IMU 124可以是加速度计和陀螺仪的组合。
雷达126可利用无线电信号来感测车辆100的周边环境内的物体。在一些实施例中，除了感测物体以外，雷达126还可用于感测物体的速度和/或前进方向。
激光测距仪128可利用激光来感测车辆100所位于的环境中的物体。在一些实施例中，激光测距仪128可包括一个或多个激光源、激光扫描器以及一个或多个检测器，以及其他系统组件。
相机130可用于捕捉车辆100的周边环境的多个图像。相机130可以是静态相机或视频相机。
控制系统106为控制车辆100及其组件的操作。控制系统106可包括各种元件，其中包括转向系统132、油门134、制动单元136、传感器融合算法138、计算机视觉系统140、路线控制系统142以及障碍物避免系统144。
转向系统132可操作来调整车辆100的前进方向。例如在一个实施例中可以为方向盘系统。
油门134用于控制引擎118的操作速度并进而控制车辆100的速度。
制动单元136用于控制车辆100减速。制动单元136可使用摩擦力来减慢车轮121。在其他实施例中，制动单元136可将车轮121的动能转换为电流。制动单元136也可采取其他形式来减慢车轮121转速从而控制车辆100的速度。
计算机视觉系统140可以操作来处理和分析由相机130捕捉的图像以便识别车辆100周边环境中的物体和/或特征。所述物体和/或特征可包括交通信号、道路边界和障碍物。计算机视觉系统140可使用物体识别算法、运动中恢复结构(Structure from Motion，SFM)算法、视频跟踪和其他计算机视觉技术。在一些实施例中，计算机视觉系统140可以用于为环境绘制地图、跟踪物体、估计物体的速度等等。
路线控制系统142用于确定车辆100的行驶路线。在一些实施例中，路线控制系统142可结合来自传感器138、GPS 122和一个或多个预定地图的数据以为车辆100确定行驶路线。
障碍物避免系统144用于识别、评估和避免或者以其他方式越过车辆100的环境中的潜在障碍物。
当然，在一个实例中，控制系统106可以增加或替换地包括除了所示出和描述的那些 以外的组件。或者也可以减少一部分上述示出的组件。
车辆100通过外围设备108与外部传感器、其他车辆、其他计算机系统或用户之间进行交互。外围设备108可包括无线通信系统146、车载电脑148、麦克风150和/或扬声器152。
在一些实施例中，外围设备108提供车辆100的用户与用户接口116交互的手段。例如，车载电脑148可向车辆100的用户提供信息。用户接口116还可操作车载电脑148来接收用户的输入。车载电脑148可以通过触摸屏进行操作。在其他情况中，外围设备108可提供用于车辆100与位于车内的其它设备通信的手段。例如，麦克风150可从车辆100的用户接收音频(例如，语音命令或其他音频输入)。类似地，扬声器152可向车辆100的用户输出音频。
无线通信系统146可以直接地或者经由通信网络来与一个或多个设备无线通信。例如，无线通信系统146可使用3G蜂窝通信，例如CDMA、EVD0、GSM/GPRS，或者4G蜂窝通信，例如LTE。或者5G蜂窝通信。无线通信系统146可利用WiFi与无线局域网(wireless local area network，WLAN)通信。在一些实施例中，无线通信系统146可利用红外链路、蓝牙或ZigBee与设备直接通信。其他无线协议，例如各种车辆通信系统，例如，无线通信系统146可包括一个或多个专用短程通信(dedicated short range communications，DSRC)设备，这些设备可包括车辆和/或路边台站之间的公共和/或私有数据通信。
电源110可向车辆100的各种组件提供电力。在一个实施例中，电源110可以为可再充电锂离子或铅酸电池。这种电池的一个或多个电池组可被配置为电源为车辆100的各种组件提供电力。在一些实施例中，电源110和能量源119可一起实现，例如一些全电动车中那样。
车辆100的部分或所有功能受计算机系统112控制。计算机系统112可包括至少一个处理器113，处理器113执行存储在例如数据存储装置114这样的非暂态计算机可读介质中的指令115。计算机系统112还可以是采用分布式方式控制车辆100的个体组件或子系统的多个计算设备。
处理器113可以是任何常规的处理器，诸如商业可获得的CPU。替选地，该处理器可以是诸如ASIC或其它基于硬件的处理器的专用设备。尽管图1功能性地图示了处理器、存储器、和在相同块中的计算机110的其它元件，但是本领域的普通技术人员应该理解该处理器、计算机、或存储器实际上可以包括在相同的物理外壳内的多个处理器、计算机、或存储器。例如，存储器可以是硬盘驱动器或位于不同于计算机110的外壳内的其它存储介质。因此，对处理器或计算机的引用将被理解为包括对可以或者可以不并行操作的处理器或计算机或存储器的集合的引用。不同于使用单一的处理器来执行此处所描述的步骤，诸如转向组件和减速组件的一些组件每个都可以具有其自己的处理器，所述处理器只执行与特定于组件的功能相关的计算。
在此处所描述的各个方面中，处理器可以位于远离该车辆并且与该车辆进行无线通信。在其它方面中，此处所描述的过程中的一些在布置于车辆内的处理器上执行而其它则由远程处理器执行，包括采取执行单一操纵的必要步骤。
在一些实施例中，数据存储装置114可包含指令115(例如，程序逻辑)，指令115 可被处理器113执行来执行车辆100的各种功能，包括以上描述的那些功能。数据存储装置114也可包含额外的指令，包括向推进系统102、传感器系统104、控制系统106和外围设备108中的一个或多个发送数据、从其接收数据、与其交互和/或对其进行控制的指令。
除了指令115以外，数据存储装置114还可存储数据，例如道路地图、路线信息，车辆的位置、方向、速度以及其它这样的车辆数据，以及其他信息。这种信息可在车辆100在自主、半自主和/或手动模式中操作期间被车辆100和计算机系统112使用。
用户接口116，用于向车辆100的用户提供信息或从其接收信息。可选地，用户接口116可包括在外围设备108的集合内的一个或多个输入/输出设备，例如无线通信系统146、车车在电脑148、麦克风150和扬声器152。
计算机系统112可基于从各种子系统(例如，行进系统102、传感器系统104和控制系统106)以及从用户接口116接收的输入来控制车辆100的功能。例如，计算机系统112可利用来自控制系统106的输入以便控制转向单元132来避免由传感器系统104和障碍物避免系统144检测到的障碍物。在一些实施例中，计算机系统112可操作来对车辆100及其子系统的许多方面提供控制。
可选地，上述这些组件中的一个或多个可与车辆100分开安装或关联。例如，数据存储装置114可以部分或完全地与车辆100分开存在。上述组件可以按有线和/或无线方式来通信地耦合在一起。
可选地，上述组件只是一个示例，实际应用中，上述各个模块中的组件有可能根据实际需要增添或者删除，图1不应理解为对本申请实施例的限制。
在道路行进的自动驾驶汽车，如上面的车辆100，可以识别其周围环境内的物体以确定对当前速度的调整。所述物体可以是其它车辆、交通控制设备、或者其它类型的物体。在一些示例中，可以独立地考虑每个识别的物体，并且基于物体的各自的特性，诸如它的当前速度、加速度、与车辆的间距等，可以用来确定自动驾驶汽车所要调整的速度。
可选地，自动驾驶汽车车辆100或者与自动驾驶车辆100相关联的计算设备(如图1的计算机系统112、计算机视觉系统140、数据存储装置114)可以基于所识别的物体的特性和周围环境的状态(例如，交通、雨、道路上的冰、等等)来预测所述识别的物体的行为。可选地，每一个所识别的物体都依赖于彼此的行为，因此还可以将所识别的所有物体全部一起考虑来预测单个识别的物体的行为。车辆100能够基于预测的所述识别的物体的行为来调整它的速度。换句话说，自动驾驶汽车能够基于所预测的物体的行为来确定车辆将需要调整到(例如，加速、减速、或者停止)什么稳定状态。在这个过程中，也可以考虑其它因素来确定车辆100的速度，诸如，车辆100在行驶的道路中的横向位置、道路的曲率、静态和动态物体的接近度等等。
除了提供调整自动驾驶汽车的速度的指令之外，计算设备还可以提供修改车辆100的转向角的指令，以使得自动驾驶汽车遵循给定的轨迹和/或维持与自动驾驶汽车附近的物体(例如，道路上的相邻车道中的轿车)的安全横向和纵向距离。
上述车辆100可以为轿车、卡车、摩托车、公共汽车、船、飞机、直升飞机、割草机、娱乐车、游乐场车辆、施工设备、电车、高尔夫球车、火车、和手推车等，本申请实施例不做特别的限定。
图2是本申请实施例提供的自动驾驶系统的示意图。
如图2所示的自动驾驶系统包括计算机系统101，其中，计算机系统101包括处理器103，处理器103和系统总线105耦合。处理器103可以是一个或者多个处理器，其中每个处理器都可以包括一个或多个处理器核。显示适配器(video adapter)107，显示适配器可以驱动显示器109，显示器109和系统总线105耦合。系统总线105通过总线桥111和输入输出(I/O)总线113耦合。I/O接口115和I/O总线耦合。I/O接口115和多种I/O设备进行通信，比如输入设备117(如：键盘，鼠标，触摸屏等)，多媒体盘(media tray)121,(例如，CD-ROM，多媒体接口等)。收发器123(可以发送和/或接受无线电通信信号)，摄像头155(可以捕捉静态和动态数字视频图像)和外部USB接口125。其中，可选地，和I/O接口115相连接的接口可以是USB接口。
其中，处理器103可以是任何传统处理器，包括精简指令集计算(reduced instruction set computer，RISC)处理器、复杂指令集计算(complex instruction set computer，CISC)处理器或上述的组合。可选地，处理器可以是诸如专用集成电路(application specific integrated circuit，ASIC)的专用装置。可选地，处理器103可以是神经网络处理器或者是神经网络处理器和上述传统处理器的组合。
可选地，在本文所述的各种实施例中，计算机系统101可位于远离自动驾驶车辆的地方，并且可与自动驾驶车辆无线通信。在其它方面，本文所述的一些过程在设置在自动驾驶车辆内的处理器上执行，其它由远程处理器执行，包括采取执行单个操纵所需的动作。
计算机101可以通过网络接口129和软件部署服务器149通信。网络接口129是硬件网络接口，比如，网卡。网络127可以是外部网络，比如因特网，也可以是内部网络，比如以太网或者虚拟私人网络(virtual private network，VPN)。可选地，网络127还可以是无线网络，比如WiFi网络，蜂窝网络等。
硬盘驱动接口和系统总线105耦合。硬件驱动接口和硬盘驱动器相连接。系统内存135和系统总线105耦合。运行在系统内存135的数据可以包括计算机101的操作系统137和应用程序143。
操作系统包括解析器139(shell)和内核(kernel)141。shell是介于使用者和操作系统之内核(kernel)间的一个接口。shell是操作系统最外面的一层。shell管理使用者与操作系统之间的交互：等待使用者的输入，向操作系统解释使用者的输入，并且处理各种各样的操作系统的输出结果。
内核141由操作系统中用于管理存储器、文件、外设和系统资源的那些部分组成。直接与硬件交互，操作系统内核通常运行进程，并提供进程间的通信，提供CPU时间片管理、中断、内存管理、IO管理等等。
应用程序143包括控制汽车自动驾驶相关的程序，比如，管理自动驾驶的汽车和路上障碍物交互的程序，控制自动驾驶汽车路线或者速度的程序，控制自动驾驶汽车和路上其他自动驾驶汽车交互的程序。应用程序143也存在于deploying server 149的系统上。在一个实施例中，在需要执行应用程序143时，计算机系统101可以从deploying server14下载应用程序143。
传感器153和计算机系统101关联。传感器153用于探测计算机101周围的环境。举例来说，传感器153可以探测动物，汽车，障碍物和人行横道等，进一步传感器还可以探 测上述动物，汽车，障碍物和人行横道等物体周围的环境，比如：动物周围的环境，例如，动物周围出现的其他动物，天气条件，周围环境的光亮度等。可选地，如果计算机101位于自动驾驶车辆上，传感器可以是摄像头，红外线感应器，化学检测器，麦克风等。
例如，应用程序143可以根据传感器153探测到的周围的环境信息进行车道推理，得到车辆的待行驶的车道(或可行驶的车道)，此时，基于所述待行驶的车道，就可以对自动驾驶车辆进行路径规划，进而确定车辆的控制量，实现车辆的自动驾驶。
图3是本申请实施例提供的一种芯片硬件结构图，该芯片包括神经网络处理器20。该芯片可以为在如图2所示的处理器103中，用以根据环境信息进行车道推理。在本申请实施例中，预训练的神经网络中各层的算法均可在如图3所示的芯片中得以实现。
本申请实施例中的推理车道的方法也可以在如图3所示的芯片中得以实现，其中，该芯片可以与实现上述预训练的神经网络的芯片是同一个芯片，或者，该芯片也可以与实现上述预训练的神经网络的芯片是不同的芯片，本申请实施例对此并不限定。
神经网络处理器NPU 50NPU作为协处理器挂载到主CPU(host CPU)上，由Host CPU分配任务。NPU的核心部分为运算电路50，通过控制器504控制运算电路503提取存储器中的矩阵数据并进行乘法运算。
在一些实现中，运算电路203内部包括多个处理单元(process engine,PE)。在一些实现中，运算电路203是二维脉动阵列。运算电路203还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路203是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器202中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器201中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)208中。
向量计算单元207可以对运算电路的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。例如，向量计算单元207可以用于神经网络中非卷积/非FC层的网络计算，如池化(pooling)，批归一化(batch normalization)，局部响应归一化(local response normalization)等。
在一些实现种，向量计算单元能207将经处理的输出的向量存储到统一缓存器206。例如，向量计算单元207可以将非线性函数应用到运算电路203的输出，例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元207生成归一化的值、合并值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路203的激活输入，例如，用于在神经网络中的后续层中的使用。
统一存储器206用于存放输入数据以及输出数据。
权重数据直接通过存储单元访问控制器205(direct memory access controller，DMAC)将外部存储器中的输入数据搬运到输入存储器201和/或统一存储器206、将外部存储器中的权重数据存入权重存储器202，以及将统一存储器206中的数据存入外部存储器。
总线接口单元(bus interface unit，BIU)210，用于通过总线实现主CPU、DMAC和取指存储器209之间进行交互。
与控制器204连接的取指存储器(instruction fetch buffer)209，用于存储控制器204 使用的指令；
控制器204，用于调用取指存储器209中缓存的指令，实现控制该运算加速器的工作过程。
一般地，统一存储器206，输入存储器201，权重存储器202以及取指存储器209均为片上(On-Chip)存储器，外部存储器为该NPU外部的存储器，该外部存储器可以为双倍数据率同步动态随机存储器(double data rate synchronous dynamic random access memory,DDR SDRAM)、高带宽存储器(high bandwidth memory，HBM)或其他可读可写的存储器。
计算机系统112还可以从其它计算机系统接收信息或转移信息到其它计算机系统。或者，从车辆100的传感器系统104收集的传感器数据可以被转移到另一个计算机对此数据进行处理。
例如，如图4所示，来自计算机系统312的数据可以经由网络被传送到云侧的服务器320用于进一步的处理。网络以及中间节点可以包括各种配置和协议，包括因特网、万维网、内联网、虚拟专用网络、广域网、局域网、使用一个或多个公司的专有通信协议的专用网络、以太网、WiFi和HTTP、以及前述的各种组合。这种通信可以由能够传送数据到其它计算机和从其它计算机传送数据的任何设备，诸如调制解调器和无线接口。
在一个示例中，服务器320可以包括具有多个计算机的服务器，例如，负载均衡服务器群，为了从计算机系统312接收、处理并传送数据的目的，其与网络的不同节点交换信息。该服务器可以被类似于计算机系统312配置，具有处理器330、存储器340、指令350、和数据360。
示例性地，服务器320的数据360可以包括地图信息和/或自动驾驶车辆周围道路情况的相关信息。例如，服务器320可以接收、检测、存储、更新、以及传送与自动驾驶车辆道路情况的相关信息。
例如，地图信息可以为高精度地图，地图信息中可以包括预存的车道线信息(例如，车道标识线)。
再例如，自动驾驶车辆周围道路情况的相关信息包括与自动驾驶车辆的预测行驶路径具有交汇点的其他车辆的位置信息以及运动参数的信息，比如，其他车辆当前的位置信息、其他车辆的速度信息、其他车辆的预测行驶轨迹信息等。
图5为本申请实施例的一个自动驾驶系统500的示例性结构框图。如图5所示，自动驾驶系统500可以包括传感器510、感知定位地图子系统520、融合子系统530、预测认知子系统540、规划子系统550、控制子系统560及执行器570。
需要说明的是，图5所示的自动驾驶系统500仅为示例而非限定，本领域的普通技术人员可以理解，自动驾驶系统500中可以包括更多或更少的模块或子系统(或单元)，本申请实施例中对此并不限定。
传感器510可以用于感知车辆(例如，自动驾驶车辆)外部的物体，并将感知到的数据传送给感知定位发图模块520。其中，传感器510可以包括摄像头、毫米波雷达、激光雷达以及超声波雷达等传感器
感知定位地图子系统520可以用于获取环境中的交通要素数据，提供车辆(例如，自动驾驶车辆)的定位，及车辆所在的道路拓扑信息。
如图5所示，感知定位地图子系统520包括视觉感知模块522、雷达感知模块524、定位模块526及地图模块526，其中，视觉感知模块522可以用于检测车道标识线、行驶车辆、可通行区域、交通标识，雷达感知模块524可以用于检测车辆、道路边沿及可通行区域，定位模块526可以用于实现车辆的全局定位，地图模块526可以用于获取周边道路拓扑。
融合子系统530可以用于多个感知模块输出结果的融合，实现更精准的运动目标检测、可通行区域检测和路沿检测。
如图5所示，融合子系统530包括目标融合模块532、可通行路面融合模块534、路沿检测及融合模块536，其中，目标融合模块532可以用于实现视觉感知目标与雷达检测目标的融合和跟踪，可通行路面融合模块534可以用于实现视觉检测可通行区域与雷达检测可通行区域的融合和跟踪，路沿检测及融合模块536可以用于视觉语义分割结果与雷达检测路沿的融合和跟踪。
预测认知子系统540可以用于实现对环境的进一步理解和推理。
如图5所示，预测认知子系统540包括目标预测模块542及道路结构认知模块544，其中，目标预测模块542可以用于实现检测运动目标的未来运动预测，道路结构认知模块544可以用于进行车道推理，获得自动驾驶车辆的待行驶的车道(或可行驶的车道)。
规划子系统550可以用于基于获得的环境检测结果及环境认知结果规划自车驾驶策略。
如图5所示，规划子系统550包括高度自动驾驶(highly automated driving，HAD)规划器552、部分自动驾驶(partial automated driving，PAD)规划器554及功能管理器556，其中，HAD规划器552可以用于进行高度自动驾驶功能情况下的规划，所包含的自动驾驶功能较多，PAD规划器554可以用于部分自动驾驶，如自适应巡航等功能，功能管理器556可以用于启动、切换HAD功能和PAD功能。
控制子系统560用于基于规划结果计算车身控制量，从而向底盘电控系统发出目标控制量指令。
如图5所示，控制子系统560包括横向控制模块562、纵向控制模块564及仲裁器566，其中，横向控制模块562可以用于计算实现目标轨迹的转向系统控制量，纵向控制模块564可以用于计算实现目标速度曲线的动力和制动系统控制量，仲裁器566可以用于实现多功能控制量的仲裁。
执行器570可以用于实现控制子系统发出的控制指令，实现符合预期的转向和加减速。
需要说明的，上述图5所示的自动驾驶系统500仅为示例而非限定，所述自动驾驶系统500中可以包括更多或更少的系统、模块、子模型或单元，本申请实施例中对此并不限定。
在现有技术中，上述图5中的道路结构认知模块544可以通过多种解决方案进行车道推理，具体如下：
方案一：
基于车道标识线的检测进行车道推理，即通过机器视觉或图像检测等方法，检测图像中的车道标识线，并基于检测到的车道标识线确定自动驾驶车辆的待行驶的车道(或可行 驶的车道)。
但是，该方法过度依赖于实际的车道标识线及实时的环境条件。
例如，在当前的技术条件下，车道标识线检测在以下几种情况下容易出现失效：
1、车道线磨损或丢失；
2、光照条件不佳，比如，有逆光或者有栏杆阴影等光影干扰；
3、路面标识复杂，比如，有文字或者在匝道处有引导标识线。
方案二：
基于高精度地图进行车道推理，即查询高精度地图，获得高精度地图中预先存储的车道标识线信息，并基于该车道标识线信息确定自动驾驶车辆的待行驶的车道(或可行驶的车道)。
但是，该方法过度依赖于高精度地图的制作、更新及维护，且还需要较高精度的定位，才能获得准确可靠的车道信息。
方案三：
为了弥补上述方案一及方案二中的不足，方案三中提出了基于环境车辆进行车道推理的方法，即根据预设的条件筛选感兴趣车辆，在筛选出感兴趣车辆之后，确定该感兴趣车辆的行驶轨迹，并基于该感兴趣车辆的行驶轨迹确定自动驾驶车辆的待行驶的车道(或可行驶的车道)。其中，所述预设的条件可以包括车辆的距离范围、车辆的行驶轨迹曲率、车辆的观测周期等。
但是，该方法对环境车辆(即感兴趣车辆)的依赖性太强，例如，在无前车时，该方案无法实施；在环境车辆的驾驶行为不遵守交通规则(例如，环境车辆不按车道标识线的指示行驶)时，基于该环境车辆的行驶轨迹确定出的待行驶的车道可能会与其他车辆的行驶轨迹发生冲突，进而可能会带来驾驶风险。
方案四：
结合实时感知和驾驶意图进行端到端自动驾驶的方案，即先在离线环境中，基于采集的训练数据对神经网络模型进行训练，得到可以用于端到端自动驾驶的神经网络模型；接下来，在在线环节中，将局部感知(即局部环境感知)与驾驶意图作为输入，采用离线环境中训练得到的神经网络模型，计算得到控车(即控制自动驾驶车辆)的控制量。
但是，本申请通过研究及分析发现，该方法(方案四)中的局部感知仅包括摄像头采集的图像信息，对光线条件、路面类型等泛化能力，以及路面认知、路面与驾驶意图的匹配、对动态障碍物及静态障碍物的处理都需要依靠同一个神经网络模型进行处理，会导致(用于端到端自动驾驶的)神经网络模型过于复杂。
同时，该方法的输入为摄像头采集的图像信息(及驾驶意图)，输出为控车结果(例如，所述控车结果可以为对自动驾驶车辆的控制量)，训练得到的神经网络模型对采集图像的摄像头、及所控车辆(即自动驾驶车辆)的尺寸等均有较强的依赖。例如，若更换摄像头的安装位置或车辆，则需要重新采集训练数据，并基于重新采集的训练数据对神经网络模型进行训练。而且，由于该方法的输出为控车结果，并没有输出自动驾驶车辆的待行驶的车道(或可行驶的车道)，因此，会导致使用该方法进行自动驾驶时的决策控制逻辑不可追溯。
因此，本申请提出了一种推理车道的方法，通过获取多种不同类型的传感器检测到的 位于同一个坐标系下的交通元素数据，并基本该交通元素数据进行车道推理，得到车辆的车道，有助于降低用于车道推理的神经网络模型的复杂度。
下面结合图6至图16对本申请实施例中的推理车道的方法进行详细说明。
图6为本申请实施例提供的一种道路结构认知模块600的结构示意图。
图6中的道路结构认知模块600可以为上述图5中的道路结构认知模块544，该道路结构认知模块600可以包括多个子模块或单元。
如图6所示，该道路结构认知模块600可以包括感知道路结构建模子模块610、地图道路结构建模子模块620、结合环境和意图的道路结构建模子模块630及道路模型可靠性分析子模块640。
感知道路结构建模子模块610可以用于实现前述现有技术中的方案一，即基于车道标识线进行车道推理，该模块610的主要应用场景可以为车道标识线清晰可见且车道标识线的线型、曲率等在(机器视觉模型或图像检测模型的)感知范围能力内。
地图道路结构建模子模块620可以用于实现前述现有技术中的方案二，即基于高精度地图进行车道推理，该模块620的主要应用场景可以为高精度定位及高精度地图可用。
结合环境和意图的道路结构建模子模块630可以用于实现本申请实施例中的推理车道的方法，该方法不绝对依赖车道标识线、高精度定位或高精度地图，因此，该模块630的应用可以为全场景。
道路模型可靠性分析子模块640可以用于分析上述模块610、模块620及模块630输出的道路模型(例如，自动驾驶车辆的待行驶的车道或可行驶的车道)，对这几个模块输出的道路模型评估，确定出各道路模型的可靠性等级，输出可靠性等级最高的道路模型及该道路模型的可靠性等级。
需要说明的是，本申请实施例中所说的自动驾驶车辆的待行驶的车道(或可行驶的车道)可以为车道的形状信息，例如，车道的几何形状，或者，也可以为车道的代价信息，例如，车道的代价图(或代价栅格图)，或者，还可以为其他能够描述车道的信息，本申请实施例中对此并不限定。
图7是本申请实施例提供的推理车道的方法700的示意性流程图。图7所示的方法700可以由图1中的车辆100中的处理器113执行、或者，也可以由图2中的自动驾驶系统中的处理器103执行，或者，还可以由图4中的服务器320中的处理器320执行。
图7所示的方法700可以包括步骤710及步骤720，应理解，图7所示的方法700仅为示例而非限定，方法700中可以包括更多或更少的步骤，本申请实施例中对此并不限定，下面分别对这几个步骤进行详细的介绍。
S710，获取车辆的多种不同类型的传感器的检测结果。
其中，所述多种不同类型的传感器可以包括摄像头、激光雷达或毫米波雷达，或者，所述多种不同类型的传感器也可以包括其他传感器，本申请实施例中对此并不限定。
S720，对所述检测结果进行处理，得到位于同一个坐标系下的交通元素数据。
其中，所述交通元素数据可以为同一个坐标系下的数据。
可选地，所述交通元素数据可以为交通元素的类型、轮廓、位置和/或运动轨迹，所述交通元素可以为静止物体(stationary object)，或者，所述交通元素也可以为运动物体(moving object)。
例如，所述交通元素可以包括车道标识线、所述车辆周围的其他车辆、行人、道路(例如，所述车辆周围的道路或路沿等)或交通灯中的一种或多种。
应理解，上述得到位于同一个坐标系下的交通元素数据可以包括：直接获取传感器检测到的原始数据，此时，所述检测到的原始数据就是所述交通元素数据。
或者，上述得到位于同一个坐标系下的交通元素数据也可以包括：获取传感器检测到的检测结果(例如，传感器获取到的原始数据或原始图像等)后，对所述检测到的检测结果进行处理后得到处理结果，此时，所述处理后得到的处理结果就是所述交通元素数据。
需要说明的，这里所说的对所述检测到的检测结果(例如，传感器的检测结果)进行处理，可以是指下述图8中方法800中的S810。
具体地，可以包括对所述检测结果进行下述多种处理方式中的至少一种：
处理方式一：
对所述检测结果(例如，传感器获取到的原始数据或原始图像等)进行坐标系转换。
例如，可以将激光雷达检测到的测距数据转换到指定的坐标系，或者，也可以将所述测距数据在多种坐标系之间进行转换，这里所说的坐标系可以包括车体坐标系、相对坐标系或图像坐标系等，也可以包括其他常用坐标系，这里并不作限定。
处理方式二：
确定(或识别)所述检测结果中包含的交通元素的类型(例如，车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯等)。也就是说，可以认为所述交通元素数据包括交通元素的类型。
例如，可以识别出摄像头到的图像中包含的交通元素。
处理方式三：
基于所述检测结果确定(该检测结果对应的)交通元素的轮廓、位置和/或运动轨迹。
例如，可以使用现有算法对所述检测结果(或所述传感器的检测结果中包括的交通元素)进行处理，确定所述交通元素的轮廓、位置和/或运动轨迹。
其中，所述交通元素的轮廓可以包括所述交通元素的几何特征或所述交通元素的中心线等能够描述所述交通元素的轮廓特征的信息，本申请实施例中对此并不限定。
上述多种处理方式的具体实现方法可以参考现有技术，本申请实施例中不再赘述。
应理解，上述对所述检测到的检测结果(例如，传感器的检测结果)进行处理的方法仅为示例而非限定，还可以对所述检测结果进行其他处理(例如，对交通元素的运动轨迹进行滤波等)，本申请实施例中对此并不限定。
在本申请实施例中，所述方法700还可以包括S712。
S712，根据所述交通元素数据确定环境感知图像。
其中，所述环境感知图像可以用于指示所述车辆周围的环境，所述环境感知图像可以包括环视图像或前视图像中的至少一种。
例如，所述环境感知图像可以为世界模型(world model)。在自动驾驶领域中，所述世界模型可以指所述车辆的环境感知信息的汇总。
可选地，可以基于所述交通元素数据，将所述交通元素叠加到所述环境感知图像中。
例如，可以将所述交通元素数据转换到同一个坐标系下，然后根据每一个交通元素相对于所述车辆的位置和视角，将所述交通元素数据叠加到所述环境感知图像中，即得到所 述车辆的环视图像或前视图像。
具体的拼接算法可以参考现有技术，本申请实施例中对此不再赘述。
S730，使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道。
可选地，在上述S720中根据所述交通元素数据确定环境感知图像的情况下，还可以使用预设的神经网络模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
其中，所述神经网络模型可以是基于(离线采集的)交通元素数据和/或驾驶意图(intention)训练后得到的。具体的训练方法可以参考现有技术，这里不再赘述。
可选地，所述车辆的驾驶意图可以为所述车辆的待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
例如，所述语义指示可以包括“左转”、“右转”、“直行”及“停止”；或者，所述语义指示也可以包括“走右侧两车道”、“走左一车道”及“从第二个出口驶出”等。
应理解，上述举例仅为示例而非限定，本申请实施例中对所述语义指示的方式(或形式)并不限定。
同时，本申请实施例中对确定所述驾驶意图的方法也不限定，具体方法可参考现有技术，这里不再赘述。
在本申请实施例中，还可以使用预设的神经网络模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
可选地，所述车辆的车道可以通过所述车道的中心线和所述车道的宽度表示；或者，所述车辆的车道也可以通过所述车辆的车道通过路面代价图表示；或者，也可以现有技术中的其他表示所述车辆的车道，本申请实施例中对此并不限定。
图8是本申请实施例提供的推理车道的方法800的示意性流程图。图8所示的方法700可以由图1中的车辆100中的处理器113执行、或者，也可以由图2中的自动驾驶系统中的处理器103执行，或者，还可以由图4中的服务器320中的处理器320执行。
图8所示的方法800可以包括步骤810、步骤820、步骤830及步骤840，应理解，图8所示的方法800仅为示例而非限定，方法800中可以包括更多或更少的步骤，本申请实施例中对此并不限定，下面分别对这几个步骤进行详细的介绍。
S810，对传感器的检测结果进行处理，得到交通元素数据。
这里所说的对传感器的检测结果进行处理包括：对传感器的检测结果(例如，传感器获取到的原始数据或原始图像等)进行坐标系转换、确定(或识别)所述传感器的检测结果中包含的交通元素，以及基于传感器的检测结果确定(该检测结果对应的)交通元素的轮廓、位置和/或运动轨迹中的至少一种，
具体可以参见图7中的(对所述传感器的检测结果进行的)多种处理方式的描述，这里不再赘述。
可选地，上S810可以如图9中的方法900所示，该方法900可以包括步骤910、步骤920及步骤930。
S910，将所述检测结果中的道路与上一时刻(或上一帧)的道路进行匹配和拼接。
可选地，所述检测结果可以认为是当前帧的检测结果，通过S910可以实现当前帧(的检测结果中的道路)与上一帧(的检测结果中的道路)的匹配和拼接。
可选地，通过S910也可以实现当前帧(的检测结果中的道路)与当前帧以前的其他帧(的检测结果中的道路)的匹配和拼接。
S920，将所述检测结果中的车道标识线与上一时刻的车道标识线进行匹配和拼接。
S930，更新所述检测结果中的运动物体的轨迹，并对该轨迹进行滤波处理。
需要说明的是，上述方法900中的各个步骤都并非必须执行的步骤，只有在所述检测结果中包含相应的交通元素的情况下，才会执行相应的步骤。
例如，在所述检测结果中包含道路的情况下，执行S910，否则，不执行S910。
S820，根据所述交通元素数据，确定环境感知图像。
可选地，所述S820可以通过下述任一方案实现。
方案一：
如图10所示，方案一可以包括步骤1010及步骤1020，具体如下。
S1010，拼接环视图像。
根据多个视角的传感器采集的图像相对于所述车辆的位置和视角，将所述多个视角的传感器采集的图像进行拼接，形成一个完整的环视图像。
S1020，将所述交通元素叠加到所述环视图像中，得到环境感知图像。
需要说明的是，在上述S1020中，只有在包含某一种交通元素的情况下，才会执行该交通元素对应的步骤。
例如，在所述检测结果中包含道路的情况下，将道路叠加到所述环视图像中。
方案二：
如图11所示，方案二可以包括步骤1110及步骤1120，具体如下。
S1110，生成初始环境图像。
例如，可以生成以所述车辆为中心、环境空白的初始图像，该图像就可以称为初始环境图像。
S1120，将所述交通元素叠加到所述初始环境图像中，得到环境感知图像。
与上述S1020类似，在S1120中，只有在包含某一种交通元素的情况下，才会执行该交通元素对应的步骤。
上述两种方案得到的所述环境感知图像都可以包括前视图像和/或环视图像。同时，上述两种方案中的拼接方法及叠加方法具体可以参考现有技术，这里不再赘述。
S830，确定车辆的驾驶意图。
可选地，所述S830可以通过下述任一方法实现。
方法一：
如图12中的方法1200所示，方法一可以包括下述多个步骤。
S1210，是否需要重新规划路线。
可选地，可以根据车辆的当前位置确定所述车辆是否偏离已规划路线。
S1220，获取待行驶道路的路点序列。
若所述车辆已偏离已规划路线，则可以重新规划路线，获得待行驶道路的路点序列；否则，可以继续使用原待行驶道路的路点序列，即将原待行驶道路的路点序列作为待行驶 道路的路点序列。
所述待行驶道路的路点序列可以使用{P(1)，P(2)，P(3)，…，P(N)}表示，其中，P(k)表示第k个路点的位置信息，k和N均为正整数。
S1230，得到车辆当前对应的路点P(i)。
可选地，可以将所述车辆的当前位置投影到S1220中确定的所述待行驶道路的路点序列中，将所述待行驶道路的路点序列中与所述车辆的当前位置最近的点作为所述当前对应的路点P(i)，其中，i为正整数。
S1240，估计t时刻后的车辆对应的路点P(j)。
可选地，可以根据所述车辆的当前速度，估计t时刻后的车辆对应的路点P(j)，其中，t和j可以为正整数。
其中，t的取值可以预先设定，例如，可以设定t为10秒；或者，t的取值可以根据车辆的位置及行驶的道路等确定，例如，车辆是否位于城市中的拥挤路段，或者，车辆是否行驶在高速公路上。
S1250，根据P(i)及P(j)，计算车辆的朝向角变化。
S1260，根据车辆的朝向角变化确定驾驶意图。
可选地，方法一确定的驾驶意图可以为语义指示(例如，离散的驾驶意图)，可以包括两种：
(1)道路级驾驶意图
道路级驾驶意图可以对应道路级导航，可以包括“左转”、“右转”、“直行”、“停止”等；
(2)车道级驾驶意图
车道级驾驶意图可以对应车道级导航，可以包括待行驶车道号，如“左侧第一车道”、“右侧两个车道”。
方法二：
S1310，是否需要重新规划路线。
可选地，可以根据车辆的当前位置确定所述车辆是否偏离已规划路线。
S1320，获取待行驶道路的路点序列。
若所述车辆已偏离已规划路线，则可以重新规划路线，获得待行驶道路的路点序列；否则，可以继续使用原待行驶道路的路点序列，即将原待行驶道路的路点序列作为待行驶道路的路点序列。
所述待行驶道路的路点序列可以使用{P(1)，P(2)，P(3)，…，P(N)}表示，其中，P(k)表示第k个路点的位置信息，k和N均为正整数。
S1330，得到车辆当前对应的路点P(i)。
可选地，可以将所述车辆的当前位置投影到S1220中确定的所述待行驶道路的路点序列中，将所述待行驶道路的路点序列中与所述车辆的当前位置最近的点作为所述当前对应的路点P(i)，其中，i为正整数。
S1340，确定路点P(i)周围的局部地图。
可选地，可以将所述车辆当前对应的路点P(i)周围、半径D范围内的地图路网作为局部地图。
其中，D为正整数，D的取值可以根据车辆的当前速度确定。例如，当车辆的车速较 低时，D的取值可以为20米。
S1350，估计t时刻后的车辆对应的路点P(i+m)。
可选地，可以根据所述车辆的当前速度，估计t时刻后的车辆对应的路点P(i+m)，其中，t、i和m均为正整数。
其中，t的取值可以预先设定，例如，可以设定t为10秒；或者，t的取值可以根据车辆的位置及行驶的道路等确定，例如，车辆是否位于城市中的拥挤路段，或者，车辆是否行驶在高速公路上。
S1360，估计t时刻前的车辆对应的路点P(i-n)。
可选地，可以根据所述车辆的当前速度，估计t时刻前的车辆对应的路点P(i-n)，其中，n为正整数。
S1370，根据路点P(i-n)、路点P(i)及路点P(i+m)，绘制局部地图。
可选地，可以将{P(i)，…，P(i+m)}作为待行驶的路点，绘制到所述局部地图中，将{P(i-n)，…，P(i)}作为历史路点(即已行驶的路点)，绘制到所述局部地图中。其中，所述待行驶的路点在所述局部地图中的标记与所述历史路点在所述局部地图中的标记不同，例如，标记的颜色、标记的类型等不同。
如图14所示，细实线表示待行驶的路点，虚线表示历史路点，粗实线表示道路。
S1380，根据局部地图确定车辆的驾驶意图。
可选地，方法二确定的驾驶意图可以为路线示意图(例如，局部路线示意图)，路线示意图可以包括所述车辆的历史道径和待行驶路径，例如，图14中所示的A、B及C三种路线示意图。
在图14中，A部分表示直道上的直行示意图，待行驶路径为直行路线；B部分表示车辆已经行驶过右转路线，待行驶路径为直行路线；C部分表示车辆已经行驶过直行路线，待行驶路线为右转路线。
S840，使用预设的神经网络模型，基于所述环境感知图像和所述驾驶意图进行车道推理，得到所述车辆的车道。
其中，所述车辆的车道，可以通过下述任一方式表达：
方式一：
车道中心线及宽度：根据车辆所处的环境(即车辆的视野范围和系统需求)，设定车道的长度需求。
例如，当车辆处于车速较慢的环境时，车道的长度需求可以为20m，以每米一个点的距离进行采样的话，需要20个位置点来表示车道，其中，每个位置点包括三个数值，分别对应该搁置点的横坐标、纵坐标和宽度。
方式二：
代价栅格图：可以将车辆周围的区域划分为P*Q个栅格，所述车辆的车道也可以通过该P*Q个栅格中的多个栅格进行表示，其中，P和Q均为正整数。
根据车载传感器的认知能力，代价栅格图的分辨率，即每个栅格的大小可以设计为20cm*20cm，或者10cm*10cm。每个栅格有[0,1]区间的数值表示其代价值。
可选地，所述神经网络模型可以是基于离线采集的环境感知图像和驾驶意图训练后得到的。
可选地，在S840中，可以将S820中确定的环境感知图像和S830中确定的驾驶意图作为输入，使用所述预先训练得到的神经网络模型进行车道推理，得到所述车辆的车道。
例如，如图15所示，可以将S820中方案一确定的环境感知图像和S830中方法一确定的驾驶意图作为输入，输出为车道的中心线和所述车道的宽度，所述神经网络模型的结构可以如图15所示。
如图15所示，所述神经网络模型可以使用ResNet50网络提取环境感知图像中的特征，得到环境感知特征，可以使用解码器将驾驶意图解码为驾驶意图特征(即离散的驾驶意图)，该驾驶意图特征可以使用四维向量分别对应“左转”、“右转”、“直行”、“停止”这四种离散的驾驶意图；所述神经网络模型可以使用连接层(concatenate layer)连接环境感知特征与驾驶意图特征，得到连接后的特征，并将该连接后的特征输入全连接层；所述全连接层可以生成4个平行的全连接模型(这里的“4”为离散的驾驶意图的数目)，其中，每一个全连接模型有两层压缩层，将特征高度压缩，最终，全连接层输出的特征(即4个平行的全连接模型)的深度下降为4，同时，每个压缩层前都可以进行一次丢弃(dropout)，通过屏蔽部分连接元达到防止特征过拟合的目的；最后，所述神经网络模型根据输入意图，从全连接层输出的特征(即4个平行的全连接模型)中挑选出该输入意图所对应的预测输出，即其中一个全连接模型，该全连接模型就是推理得到的车道。
再例如，如图16所示，可以将S820中方案二确定的环境感知图像和S830中方法二确定的驾驶意图作为输入，输出为车道的中心线和所述车道的宽度，所述神经网络模型的结构可以如图16所示。
如图16所示，所述神经网络模型可以使用ResNet50网络提取环境感知图像中的特征，得到环境感知特征，同样地，也可以使用ResNet50网络提取驾驶意图中的特征，得到驾驶意图特征；
所述神经网络模型使用连接层(concatenate layer)连接环境感知特征与驾驶意图特征，得到连接后的特征，并将该连接后的特征输入全连接层；
所述全连接层有两层压缩层，可以将特征高度压缩，最终，全连接层输出的特征的深度下降为1，同时，每个压缩层前都可以进行一次丢弃(dropout)，通过屏蔽部分连接元达到防止特征过拟合的目的；最后，所述全连接层输出推理得到的车道。
需要说明的是，上述图15和图16中的所述神经网络模型的处理过程具体可以参考现有技术，本申请实施例中不再赘述。
图17是本申请实施例提供的训练车道推理模型的方法1700的示意性流程图。图17所示的方法1700可以由图1中的车辆100中的处理器113执行、或者，也可以由图2中的自动驾驶系统中的处理器103执行，或者，还可以由图4中的服务器320中的处理器320执行。
图17所示的方法1700可以包括步骤1710、步骤1720及步骤1730，应理解，图17所示的方法1700仅为示例而非限定，方法1700中可以包括更多或更少的步骤，本申请实施例中对此并不限定，下面分别对这几个步骤进行详细的介绍。
S1710，获取车辆的多种不同类型的传感器的检测结果。
S1720，对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据。
其中，所述交通元素数据可以为同一个坐标系下的数据。
S1730，使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道。
S1740，根据所述车辆的车道，调整所述车道推理模型的参数。
可选地，所述获取车辆的多种不同类型的传感器检测到的交通元素数据，可以包括：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
可选地，所述对所述检测结果进行处理得到所述交通元素数据，可以包括：对所述检测结果进行坐标系转换，得到所述交通元素数据。
可选地，所述多种不同类型的传感器可以包括摄像头、激光雷达或毫米波雷达。
可选地，所述交通元素数据可以为所述交通元素的轮廓、位置和/或运动轨迹，所述交通元素可以包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
可选地，所述方法还可以包括：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，可以包括：使用所述车道推理模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
可选地，所述使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，可以包括：使用所述车道推理模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
可选地，所述车辆的驾驶意图可以为所述车辆的待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示可以用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
可选地，所述车辆的车道可以通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道可以通过路面代价图表示。
图18是本申请一个实施例提供的推理车道的装置1800的示意性框图。应理解，图18示出的推理车道的装置1800仅是示例，本申请实施例的装置1800还可包括其他模块或单元。应理解，行为规划装置1800能够执行图7或图8的方法中的各个步骤，为了避免重复，此处不再详述。
获取单元1810，用于获取车辆的多种不同类型的传感器的检测结果。
转换单元1820，用于对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据。
车道推理单元1830，用于使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道。
可选地，所述获取单元1810具体用于：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
可选地，所述获取单元1810具体用于：对所述检测结果进行坐标系转换，得到所述交通元素数据。
可选地，所述多种不同类型的传感器包括摄像头、激光雷达或毫米波雷达。
可选地，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
可选地，所述装置1800还包括叠加单元1840，用于：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述车道推理单元1830具体用于：使用预设的神经网络模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
可选地，所述车道推理单元1830具体用于：使用预设的神经网络模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
可选地，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
可选地，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
应理解，这里的推理车道的装置1800以功能模块的形式体现。这里的术语“模块”可以通过软件和/或硬件形式实现，对此不作具体限定。例如，“模块”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
作为一个示例，本申请实施例提供的推理车道的装置1800可以是自动驾驶车辆的车载设备，或者，可以是配置于车载设备中的芯片，以用于执行本申请实施例所述的方法。
图19是本申请一个实施例提供的训练车道推理模型的装置1900的示意性框图。应理解，图19示出的训练车道推理模型的装置1900仅是示例，本申请实施例的装置1900还可包括其他模块或单元。应理解，行为规划装置1900能够执行图17的方法中的各个步骤，为了避免重复，此处不再详述。
获取单元1910，用于获取车辆的多种不同类型的传感器的检测结果。
转换单元1920，用于对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据。
车道推理单元1930，用于使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道；
调整单元1940，用于根据所述车辆的车道，调整所述车道推理模型的参数。
可选地，所述获取单元1910具体用于：获取所述多种不同类型的传感器的检测结果；对所述检测结果进行处理得到所述交通元素数据。
可选地，所述获取单元1910具体用于：对所述检测结果进行坐标系转换，得到所述交通元素数据。
可选地，所述多种不同类型的传感器包括摄像头、激光雷达或毫米波雷达。
可选地，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素 包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
可选地，所述装置1900还包括叠加单元1950，用于：根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；其中，所述车道推理单元1930具体用于：使用所述车道推理模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
可选地，所述车道推理单元1930具体用于：使用所述车道推理模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
可选地，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
可选地，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
应理解，这里的训练车道推理模型的装置1900以功能模块的形式体现。这里的术语“模块”可以通过软件和/或硬件形式实现，对此不作具体限定。例如，“模块”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
作为一个示例，本申请实施例提供的训练车道推理模型的装置1900可以是自动驾驶车辆的车载设备，或者，可以是配置于车载设备中的芯片，以用于执行本申请实施例所述的方法。
图20是本申请一个实施例的装置800的示意性框图。图20所示的装置800包括存储器801、处理器802、通信接口803以及总线804。其中，存储器801、处理器802、通信接口803通过总线804实现彼此之间的通信连接。
存储器801可以是只读存储器(read only memory，ROM)，静态存储设备，动态存储设备或者随机存取存储器(random access memory，RAM)。存储器801可以存储程序，当存储器801中存储的程序被处理器802执行时，处理器802用于执行本申请实施例的规划推理车道的方法或训练车道推理模型的方法的各个步骤，例如，可以执行图7、图8或图17所示实施例的各个步骤。
处理器802可以采用通用的中央处理器(central processing unit，CPU)，微处理器，应用专用集成电路(application specific integrated circuit，ASIC)，或者一个或多个集成电路，用于执行相关程序，以实现本申请方法实施例的推理车道的方法或训练车道推理模型的方法。
处理器802还可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，本申请实施例的规划自动驾驶车辆的方法的各个步骤可以通过处理器802中的硬件的集成逻辑电路或者软件形式的指令完成。
上述处理器802还可以是通用处理器、数字信号处理器(digital signal processing，DSP)、专用集成电路(ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执 行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器801，处理器802读取存储器801中的信息，结合其硬件完成本申请实施例中自动驾驶车辆的行为规划装置包括的单元所需执行的功能，或者，执行本申请方法实施例的规划自动驾驶车辆的方法，例如，可以执行图7、图8或图17所示实施例的各个步骤/功能。
通信接口803可以使用但不限于收发器一类的收发装置，来实现装置800与其他设备或通信网络之间的通信。
总线804可以包括在装置800各个部件(例如，存储器801、处理器802、通信接口803)之间传送信息的通路。
应理解，本申请实施例所示的装置800可以是自动驾驶车辆中的车载设备，或者，也可以是配置于车载设备中的芯片。
应理解，本申请实施例中的处理器可以为中央处理单元(central processing unit，CPU)，该处理器还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
还应理解，本申请实施例中的存储器可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的随机存取存储器(random access memory，RAM)可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlink DRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。
上述实施例，可以全部或部分地通过软件、硬件、固件或其他任意组合来实现。当使用软件实现时，上述实施例可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令或计算机程序。在计算机上加载或执行所述计算机指令或计算机程序时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以为通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存 储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集合的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质。半导体介质可以是固态硬盘。
应理解，本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况，其中A,B可以是单数或者复数。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系，但也可能表示的是一种“和/或”的关系，具体可参考前后文进行理解。
本申请中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“以下至少一项(个)”或其类似表达，是指的这些项中的任意组合，包括单项(个)或复数项(个)的任意组合。例如，a,b,或c中的至少一项(个)，可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机 软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (29)

  1. 一种推理车道的方法，其特征在于，包括：
    获取车辆的多种不同类型的传感器的检测结果；
    对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；
    使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道。
  2. 如权利要求1所述的方法，其特征在于，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
  3. 如权利要求1或2所述的方法，其特征在于，所述方法还包括：
    根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；
    其中，所述使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：
    使用预设的神经网络模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
  4. 如权利要求1至3中任一项所述的方法，其特征在于，所述使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：
    使用预设的神经网络模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
  5. 如权利要求4所述的方法，其特征在于，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
  6. 如权利要求1至5中任一项所述的方法，其特征在于，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
  7. 一种训练车道推理模型的方法，其特征在于，包括：
    获取车辆的多种不同类型的传感器的检测结果；
    对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；
    使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道；
    根据所述车辆的车道，调整所述车道推理模型的参数。
  8. 如权利要求7所述的方法，其特征在于，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
  9. 如权利要求7或8所述的方法，其特征在于，所述方法还包括：
    根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；
    其中，所述使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆 的车道，包括：
    使用所述车道推理模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
  10. 如权利要求7至9中任一项所述的方法，其特征在于，所述使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道，包括：
    使用所述车道推理模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
  11. 如权利要求10所述的方法，其特征在于，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
  12. 如权利要求7至11中任一项所述的方法，其特征在于，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
  13. 一种推理车道的装置，其特征在于，包括：
    获取单元，用于获取车辆的多种不同类型的传感器的检测结果；
    转换单元，用于对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；
    车道推理单元，用于使用预设的神经网络模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道。
  14. 如权利要求13所述的装置，其特征在于，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
  15. 如权利要求13或14所述的装置，其特征在于，所述装置还包括叠加单元，用于：
    根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；
    其中，所述车道推理单元具体用于：
    使用预设的神经网络模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
  16. 如权利要求13至15中任一项所述的装置，其特征在于，所述车道推理单元具体用于：
    使用预设的神经网络模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
  17. 如权利要求16所述的装置，其特征在于，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
  18. 如权利要求13至17中任一项所述的装置，其特征在于，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
  19. 一种训练车道推理模型的装置，其特征在于，包括：
    获取单元，用于获取车辆的多种不同类型的传感器的检测结果；
    转换单元，用于对所述检测结果进行坐标系转换，得到位于同一个坐标系下的交通元素数据；
    车道推理单元，用于使用车道推理模型，基于所述交通元素数据进行车道推理，得到所述车辆的车道；
    调整单元，用于根据所述车辆的车道，调整所述车道推理模型的参数。
  20. 如权利要求19所述的装置，其特征在于，所述交通元素数据为交通元素的轮廓、位置和/或运动轨迹，所述交通元素包括车道标识线、所述车辆周围的其他车辆、行人、道路或交通灯中的一种或多种。
  21. 如权利要求19或20所述的装置，其特征在于，所述装置还包括叠加单元，用于：
    根据所述交通元素数据确定环境感知图像，所述环境感知图像用于指示所述车辆周围的环境及所述交通元素，所述环境感知图像包括环视图像或前视图像中的至少一种；
    其中，所述车道推理单元具体用于：
    使用所述车道推理模型，基于所述环境感知图像进行车道推理，得到所述车辆的车道。
  22. 如权利要求19至21中任一项所述的装置，其特征在于，所述车道推理单元具体用于：
    使用所述车道推理模型，基于所述交通元素数据和车辆的驾驶意图进行车道推理，得到所述车辆的车道，所述车辆的驾驶意图用于指示所述车辆待行驶的路线。
  23. 如权利要求22所述的装置，其特征在于，所述车辆的驾驶意图为所述车辆待行驶的车道的路线示意图或车辆语义指示，所述车辆语义指示用于指示所述车辆的行驶方向、行驶车道或行驶路口中的至少一种。
  24. 如权利要求19至23中任一项所述的装置，其特征在于，所述车辆的车道通过所述车道的中心线和所述车道的宽度表示，或者，所述车辆的车道通过路面代价图表示。
  25. 一种推理车道的装置，其特征在于，包括处理器和存储器，所述存储器用于存储程序指令，所述处理器用于调用所述程序指令来执行权利要求1至6中任一项所述的方法。
  26. 一种训练车道推理模型的装置，其特征在于，包括处理器和存储器，所述存储器用于存储程序指令，所述处理器用于调用所述程序指令来执行权利要求7至12中任一项所述的方法。
  27. 一种汽车，其特征在于，包括权利要求13至26任一项所述的装置。
  28. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质中存储有程序指令，当所述程序指令由处理器运行时，实现权利要求1至12中任一项所述的方法。
  29. 一种芯片，其特征在于，所述芯片包括处理器与数据接口，所述处理器通过所述数据接口读取存储器上存储的指令，以执行如权利要求1至12中任一项所述的方法。
PCT/CN2021/094256 2020-07-02 2021-05-18 推理车道的方法、训练车道推理模型的方法及装置 WO2022001432A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP21834506.4A EP4167126A4 (en) 2020-07-02 2021-05-18 TRACK INFERENCE METHOD AND TRACK INFERENCE MODEL TRAINING METHOD AND APPARATUS

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010626061.4A CN113963325A (zh) 2020-07-02 2020-07-02 推理车道的方法、训练车道推理模型的方法及装置
CN202010626061.4 2020-07-02

Publications (1)

Publication Number Publication Date
WO2022001432A1 true WO2022001432A1 (zh) 2022-01-06

Family

ID=79317381

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/094256 WO2022001432A1 (zh) 2020-07-02 2021-05-18 推理车道的方法、训练车道推理模型的方法及装置

Country Status (3)

Country Link
EP (1) EP4167126A4 (zh)
CN (1) CN113963325A (zh)
WO (1) WO2022001432A1 (zh)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104766058A (zh) * 2015-03-31 2015-07-08 百度在线网络技术（北京）有限公司 一种获取车道线的方法和装置
CN107092862A (zh) * 2017-03-16 2017-08-25 浙江零跑科技有限公司 一种基于卷积神经网络的车道边缘检测方法
US20190147254A1 (en) * 2017-11-15 2019-05-16 Uber Technologies, Inc. Autonomous Vehicle Lane Boundary Detection Systems and Methods
US20190266418A1 (en) * 2018-02-27 2019-08-29 Nvidia Corporation Real-time detection of lanes and boundaries by autonomous vehicles
CN110610153A (zh) * 2019-09-10 2019-12-24 重庆工程职业技术学院 一种用于自动驾驶的车道识别方法及系统
CN111316288A (zh) * 2019-02-28 2020-06-19 深圳市大疆创新科技有限公司 道路结构信息的提取方法、无人机及自动驾驶系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104766058A (zh) * 2015-03-31 2015-07-08 百度在线网络技术（北京）有限公司 一种获取车道线的方法和装置
CN107092862A (zh) * 2017-03-16 2017-08-25 浙江零跑科技有限公司 一种基于卷积神经网络的车道边缘检测方法
US20190147254A1 (en) * 2017-11-15 2019-05-16 Uber Technologies, Inc. Autonomous Vehicle Lane Boundary Detection Systems and Methods
US20190266418A1 (en) * 2018-02-27 2019-08-29 Nvidia Corporation Real-time detection of lanes and boundaries by autonomous vehicles
CN111316288A (zh) * 2019-02-28 2020-06-19 深圳市大疆创新科技有限公司 道路结构信息的提取方法、无人机及自动驾驶系统
CN110610153A (zh) * 2019-09-10 2019-12-24 重庆工程职业技术学院 一种用于自动驾驶的车道识别方法及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4167126A4

Also Published As

Publication number Publication date
EP4167126A4 (en) 2023-11-29
EP4167126A1 (en) 2023-04-19
CN113963325A (zh) 2022-01-21

Similar Documents

Publication Publication Date Title
CN109901574B (zh) 自动驾驶方法及装置
CN110379193B (zh) 自动驾驶车辆的行为规划方法及行为规划装置
WO2022027304A1 (zh) 一种自动驾驶车辆的测试方法及装置
WO2021102955A1 (zh) 车辆的路径规划方法以及车辆的路径规划装置
US20220332348A1 (en) Autonomous driving method, related device, and computer-readable storage medium
WO2021000800A1 (zh) 道路可行驶区域推理方法及装置
WO2022001773A1 (zh) 轨迹预测方法及装置
WO2021212379A1 (zh) 车道线检测方法及装置
WO2022148172A1 (zh) 车道线规划方法及相关装置
CN113160547A (zh) 一种自动驾驶方法及相关设备
WO2022142839A1 (zh) 一种图像处理方法、装置以及智能汽车
WO2022016901A1 (zh) 一种规划车辆行驶路线的方法以及智能汽车
WO2022062825A1 (zh) 车辆的控制方法、装置及车辆
CN112672942A (zh) 一种车辆换道方法及相关设备
WO2022051951A1 (zh) 车道线检测方法、相关设备及计算机可读存储介质
US20230048680A1 (en) Method and apparatus for passing through barrier gate crossbar by vehicle
CN114440908A (zh) 一种规划车辆驾驶路径的方法、装置、智能车以及存储介质
WO2022017307A1 (zh) 自动驾驶场景生成方法、装置及系统
WO2022061702A1 (zh) 驾驶提醒的方法、装置及系统
CN113859265A (zh) 一种驾驶过程中的提醒方法及设备
US20220309806A1 (en) Road structure detection method and apparatus
EP4159564A1 (en) Method and device for planning vehicle longitudinal motion parameters
WO2022001432A1 (zh) 推理车道的方法、训练车道推理模型的方法及装置
WO2022041820A1 (zh) 换道轨迹的规划方法及装置
WO2022061725A1 (zh) 交通元素的观测方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21834506

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021834506

Country of ref document: EP

Effective date: 20230110

NENP Non-entry into the national phase

Ref country code: DE