WO2019134110A1 - Procédés et systèmes de conduite autonome - Google Patents

Procédés et systèmes de conduite autonome Download PDF

Info

Publication number
WO2019134110A1
WO2019134110A1 PCT/CN2018/071516 CN2018071516W WO2019134110A1 WO 2019134110 A1 WO2019134110 A1 WO 2019134110A1 CN 2018071516 W CN2018071516 W CN 2018071516W WO 2019134110 A1 WO2019134110 A1 WO 2019134110A1
Authority
WO
WIPO (PCT)
Prior art keywords
driving
vehicle
arrow
driving situation
cognitive
Prior art date
Application number
PCT/CN2018/071516
Other languages
English (en)
Inventor
Deyi Li
Original Assignee
Driving Brain International Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Driving Brain International Ltd. filed Critical Driving Brain International Ltd.
Priority to CA3087361A priority Critical patent/CA3087361A1/fr
Priority to PCT/CN2018/071516 priority patent/WO2019134110A1/fr
Publication of WO2019134110A1 publication Critical patent/WO2019134110A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
    • G05D1/02—Control of position or course in two dimensions
    • G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
    • G05D1/0212—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory
    • G05D1/0221—Control of position or course in two dimensions specially adapted to land vehicles with means for defining a desired trajectory involving a learning process

Definitions

  • the application is directed to self driving methods and systems. More specifically, the application is directed to autonomous driving methods and systems incorporating deep learning techniques.
  • autonomous driving refers to level 3, 4 and 5 automation. Passengers in a level 3, 4 or 5 autonomously driven vehicle are physically disengaged from controlling the vehicle (i.e. their hands and feet are disengaged from the vehicle’s controls) , and may only be responsible for taking control of the vehicle when required by the system.
  • Many existing autonomous driving systems capture images and parameters of objects surrounding a vehicle to create a view of the area around the vehicle.
  • the view may be in two or three dimensions.
  • the system then takes actions based on the view.
  • most autonomous driving systems are based on a library of if-then rules covering various circumstances. For example, if an object is within a certain distance of the vehicle, moving towards the vehicle, and within the direction of travel of the vehicle, then the system should apply the brakes.
  • Such existing systems are limited to a finite number of circumstances they are capable of handling, since one must be in the library for the system to correctly respond to it. This is particularly limiting in the instance of exceptional and unexpected driving circumstances which may be encountered while driving.
  • the application is directed to autonomous driving methods and systems, which define and describe how a device composed of hardware and software can be used to drive vehicles and other ground conveyances to achieve level 3 (L3) and above autonomous driving as defined by SAE.
  • L3 level 3
  • SAE level 3
  • One aspect of the systems and methods described herein is the deep learning techniques implemented to deal with uncertainties on the road similar to how human drivers would. With proper sensors equipped on a vehicle, the systems can be used to learn from human drivers’cognitive ability, accumulate driving skills, and eventually replace human drivers by replicating a human driver’s driving skills.
  • One aspect of the invention provides a method of autonomously driving a vehicle, the method comprising: receiving, refining and integrating data from one or more sensors; creating a graphical representation of the driving situation based on the sensor data; producing a graphical driving command based on the graphical representation; and controlling the vehicle based as least in part on the graphical driving command.
  • the graphical driving command is based on the graphical representation and previously learnt driving skills.
  • the graphical representation comprises a graphical representation of the driving situation.
  • the graphical representation is expressed in particular embodiments using a polar coordinate system comprising an angle dimension and a radius dimension.
  • the graphical representation comprises a current driving situation map
  • the graphical driving command comprises a cognitive arrow
  • generating the cognitive arrow comprises: searching a set of historical driving situation maps; determining a closest matching driving situation map from the set of historical driving situation maps most similar to the current driving situation map; and retrieving a cognitive arrow associated with the closest matching driving situation map.
  • determining a closest matching driving situation map comprises executing a machine learning algorithm.
  • the machine learning algorithm uses one or more of a convolutional neural network and a recurrent neural network.
  • calculating the cognitive arrow comprises: determining a right of way of the vehicle from the current driving situation map; and calculating the cognitive arrow based at least in part on the right of way.
  • Another aspect-of the invention provides a method of determining the final driving command for a driverless vehicle, which is a comprehensive result of two individual commands derived from two training mechanisms: the first driving command is produced based on a sequence of driving situation maps and to achieve the purpose of the trip based on a pre-defined route before the journey begins; the other driving command is generated based on various parameters when the vehicle is operating, which are not related to the purpose of the trip but are vital for the safety, comfort, smoothness, and energy efficiency of the passengers’experience and operation of the vehicle.
  • Another aspect of the invention provides a method for determining a driving command for a vehicle, the method comprising: receiving a graphical representation of a driving situation; searching a first database of previous driving situations for a closest matching driving situation; retrieving a first driving command from the first database associated with the closest matching driving situation; receiving sensor data; determining a driving scenario from the sensor data; searching a second database of driving scenarios for a closest matching driving scenario; retrieving a second driving command from the database of driving scenarios associated with the closest matching driving scenario; and combining the first driving command and the second driving command to generate a composite driving command.
  • the first database comprises a graphical database.
  • the training of an autonomous system for driving the vehicle in accordance with driving situations and/or a determination of a closest matching driving situation map comprises executing a machine learning algorithm.
  • This machine learning algorithm may be based on a convolutional neural network.
  • the training of an autonomous system for driving a vehicle in accordance with driving scenarios and/or a determination of a closest matching driving scenario comprises executing a machine learning algorithm.
  • This machine learning algorithm may be based on a recurrent neural network.
  • Another aspect of the invention provides a method of providing autonomous driving for a vehicle, wherein the vehicle comprises an autonomous driving system, a manual driving system and a sensor system, the method comprising: operating the vehicle in either one of a training and a driving mode, wherein operating the vehicle in the training mode comprises: (i) representing a driving situation at a point in time based on data output from the sensor system; (ii) recording an output of the manual driving system at the point in time, and representing the output as a first image denoting an actual steering direction and an actual acceleration or deceleration; (iii) generating, by the autonomous driving system, an autonomous driving output based on the driving situation at the point in time and a training route plan, wherein the autonomous driving output is represented as a second image denoting a planned steering direction and a planned acceleration or deceleration; (iv) determining a difference between the manual driving system output and the autonomous driving output; and (v) configuring a real time planning module of the autonomous driving system based at least in part on the difference
  • operating the vehicle in the driving mode comprises: (i) representing a driving situation at a second point in time based on data output from the sensor system; (ii) generating, by the autonomous driving system, an autonomous driving output based on the driving situation at the second point in time and a predefined route plan, and representing the output as a third image denoting a calculated steering direction and a calculated acceleration or deceleration; and (iii) applying the autonomous driving output to one or more control units of a vehicle to drive the vehicle along the predefined route plan.
  • FIG. 1 illustrates an autonomous driving system according to one embodiment.
  • FIG. 2 illustrates modules of an autonomous driving system which may be implemented as software.
  • FIG. 3A illustrates a flowchart of a method of generating one or more Driving Situation Maps (DSMs) which may be executed by an autonomous driving system according to one embodiment.
  • DSMs Driving Situation Maps
  • FIG. 3B illustrates an example driving situation.
  • FIG. 3C illustrates an example DSM.
  • FIG. 4 is a flowchart of a general route planning method which may be executed by an autonomous driving system according to one embodiment.
  • FIG. 5A is a flowchart of a method which may be executed by the cognition module of an autonomous driving system according to one embodiment to generate driving commands.
  • FIGS. 5B to 5H illustrate example cognitive arrows that may be generated by an autonomous driving system according to one embodiment.
  • FIGS. 5I-1 to 5I-8 illustrate an example sequence of cognitive arrows that may be generated for the scenario of overtaking another vehicle.
  • FIG. 5J illustrates a flowchart of a method for generating cognitive arrows during an autonomous driving mode.
  • FIG. 6A illustrates a flowchart of a method for generating cognitive arrows including strategic and tactical considerations.
  • FIG. 6B illustrates a right of way during an example driving situation.
  • FIG. 7 illustrates a flowchart of a method for training an autonomous driving system according to one embodiment.
  • FIG. 8 shows perception, cognition and control modules of an autonomous driving system according to one embodiment.
  • FIG. 9 is a data flowchart illustrating the factors that affect the decision-making process of the cognition module.
  • FIG. 10 is a data flowchart illustrating the relationships between instantaneous memory, working memory and long- term memory.
  • FIG. 11 illustrates positive learning performed by a cognition module of an autonomous driving system according to one embodiment.
  • FIG. 12 illustrates negative learning performed by a cognition module of an autonomous driving system according to one embodiment.
  • FIG. 13 schematically illustrates a control chip for tactical driving and the input that may be provided to such control chip for purposes of determining a tactical cognitive arrow.
  • FIG. 14 illustrates a comparison between a model driver’s trajectory and an autonomous driving system or robot’s trajectory during training mode.
  • Vehicle includes any ground conveyance, including ground conveyances that are not necessarily primarily for purposes of transportation or cargo, such as, for example, lawnmowers, vacuums, ice cleaners, tractors, forklifts, camera vehicles, tracking vehicles, and/or the like.
  • Modules implemented by systems described herein emulate or replace a series of driver activities such as visual perception, auditory hearing, thinking, memory, learning, reasoning, decision making, interaction, control, and so on.
  • the system may apply deep learning techniques to deal with uncertainties on the road.
  • the system can learn from a human driver’s cognitive ability, accumulate driving skills, and replace many or all of driving functions normally performed by a human driver.
  • An objective of such an autonomous driving system is to remove human drivers from driving tasks so that they are free to enjoy the ride as passengers.
  • an autonomous driving system does not become distracted and is not affected by emotions, or physical or mental fatigue.
  • Embodiments of autonomous driving systems as described herein may be developed based on how human drivers perceive and process images, receive feedback and information during driving, and apply accumulated experiences to deal with uncertainties on the road. For example, which vehicle has the right of way when two vehicles meet head to head in a narrow road? To answer this question, a study of drivers’behaviour is important. Drivers’behaviour informs communications and interactions with other vehicles.
  • a human driver’s decision-making process may be decomposed into three stages: perception, cognition and control.
  • Perception is the state of seeing, feeling or detecting surrounding objects such as other vehicles, the lane width or curvature, traffic signs, traffic signals and the like. Perception may also include sensing conditions relating to the vehicle itself like vehicle status, speed, acceleration, deceleration, vibrations, or shaking, etc.
  • the information collected at the perception stage may be used by the driver during the cognition stage to make a driving decision like steering around an obstacle or applying the brake.
  • the decisions are converted into particular ways in which to control the vehicle at the control stage, for example, by turning the steering wheel by a certain angle, or pressing the brake by a certain amount.
  • an autonomous driving system may include a plurality of sensors in or on the vehicle which receive sensor data or otherwise provide data relevant to perception. These devices may include visible light cameras, infrared cameras, radar, LIDAR (Light Detection and Ranging) , satellite navigation sensors such as GPS, inertial navigation sensors such as accelerometers and gyroscopes, thermometers, compasses, clocks, and/or other sensors.
  • LIDAR Light Detection and Ranging
  • satellite navigation sensors such as GPS, inertial navigation sensors such as accelerometers and gyroscopes, thermometers, compasses, clocks, and/or other sensors.
  • the perception module collects data from these sensors, and provides them to the cognition module for further processing.
  • Embodiments of autonomous driving systems described herein comprise a cognition module for generating real-time driving decisions, such as changing lanes, steering, accelerating and/or braking, based in part on the end result of the perception module.
  • the graphical representation comprises a driving situation map (DSM) , as described in further detail below, for use by a cognition module to determine a driving decision.
  • DSM driving situation map
  • a DSM represents in a particular format the information provided by the perception module, so as to place greater emphasis or more selective attention on objects which are closer to the vehicle. This is similar to human perception of a situation.
  • the DSM also contains information that is relevant to the driving situation and to making assessments based on the Right of Way (RoW) , defined below.
  • the DSM reflects real time situations, like whether or not there is a pedestrian crossing in front of the vehicle.
  • the DSM does not show information that is not relevant to the driving situation, for example, what particular clothing a pedestrian is wearing.
  • the autonomous driving system uses the DSMs to calculate a RoW for the vehicle.
  • the autonomous driving system also performs validity checks for safety, comfort, smoothness, speed and/or energy efficiency. It is in the cognition domain that an experienced driver would also apply the knowledge from previous journeys to the current situation to make a driving decision, for example, knowledge based on experience driving on icy roads, wet roads, or other edge driving conditions.
  • particular embodiments may gather driving data from model drivers during edge driving conditions, and use the data as part of the cognitive decision-making process.
  • the vehicle In the control stage, the vehicle is operated to adjust its direction and/or speed (e.g. turning the steering wheel, engaging or releasing the accelerator or gas pedal, engaging or releasing the brake) to implement the decision made by the cognition module.
  • direction and/or speed e.g. turning the steering wheel, engaging or releasing the accelerator or gas pedal, engaging or releasing the brake
  • an autonomous driving system according to the invention as described herein are not based on “if-then” rules.
  • the system is based instead on employing certain deep learning techniques during a learning or a training phase to enable it to make real-time decisions to deal with uncertainties on the road.
  • Deep learning techniques using Convolutional Neural Networks and Recurrent Neural Networks may be employed here, as described elsewhere herein (see Imagenet Classification with Deep Convolutional Neural Networks published at the Neural Information Processing Systems Conference by: Alex Krizhevsky, Ilya Sutaskever and Geoffrey E. Hinton Year 2012, and Speech Recognition with Deep Recurrent Neural Networks published at IEEEDOI: 10.1109/ICASSP. 2013.6638947 by: Alex Graves; Abdel-rahman Mohamed and Geoffrey Hinton Year 2013) .
  • input-output pairings comprise a sequence of driving situation maps (DSMs) as an input and a cognitive arrow (CA) as an output.
  • DSM driving situation maps
  • CA cognitive arrow
  • the DSM is a graphical representation of the driving situation.
  • the cognitive arrow is a graphical representation of the command to be applied to drive the vehicle.
  • CAs and DSMs graphically provides many advantages, specifically in the context of deep learning algorithms. Some deep learning algorithms are particularly tuned to operate with images, and can be efficiently applied to the tasks described herein. Furthermore, graphical representations may provide advantages in classifying, pairing, storing, searching, and retrieving CAs and DSMs.
  • the methods described herein enable an autonomous driving system to learn how to control a vehicle from the actions of humans, including humans serving as model drivers.
  • the output of the control, learning and/or training methods is a cognitive arrow, which is an image that can be translated into parameters that directly control a vehicle’s electronic control units.
  • An autonomous driving mode is one wherein the autonomous driving system evaluates a driving situation and makes a decision to control the vehicle.
  • a training mode is one wherein a model human driver drives the vehicle and the system uses the output provided by the model human driver’s actions to train the control algorithms for the autonomous driving system.
  • the autonomous driving system Before a vehicle can be self-driving, the autonomous driving system is operated in training mode, wherein the autonomous driving control algorithm is trained by a human operating the vehicle.
  • the algorithm is trained with pairs of DSM sets and cognitive arrows.
  • sensor outputs are captured at discrete time increments (e.g. every 100 ms) over a certain period.
  • the sensor outputs may be processed to generate a sequence of DSMs.
  • a set of sequential DSMs may be used to represent a driving situation.
  • Cognitive arrows may indicate a direction and an acceleration or deceleration of the vehicle at a point in time.
  • a human model driver In a training mode, when a set of DSMs are generated, the system records a human model driver’s operation in a graphic representation as Human Operation Cognitive Arrow (HOCA) . Concurrently, the system generates a Program Generated Cognitive Arrow (PGCA) wherein a deep learning algorithm is embedded in the generation of the PGCA.
  • PGCA Program Generated Cognitive Arrow
  • the PGCAs generated by the system will more and more closely approximate the HOCAs generated for the same driving situation.
  • An objective is to train the autonomous driving system so that the PGCA approaches the HOCA for a similar driving situation. Over time, such a system may be tuned to perform more and more like a human driver.
  • FIG. 1 shows one embodiment of an autonomous driving system 100.
  • the autonomous driving system 100 comprises a Self-Driving Cognitive System (SDCS) 110.
  • SDCS 110 may be in communication with a plurality of vehicle sensors 140, Controller Area Network (CAN) Bus 150, and user interface 180.
  • SDCS 110 comprises hardware 120, and software 130 stored thereon.
  • Hardware 120 may comprise processors, data storage devices, application specific integrated circuits (ASIC) , field programmable gate arrays (FPGA) , and communications devices.
  • ASIC application specific integrated circuits
  • FPGA field programmable gate arrays
  • Sensors 140 may comprise visible light cameras, infrared cameras, radar, LIDAR (Light Detection and Ranging) , satellite navigation receivers for receiving satellite signals such as GPS or BeiDou signals, inertial navigation sensors such as accelerometers and gyroscopes, thermometers, compasses, clocks, and/or other sensors.
  • LIDAR Light Detection and Ranging
  • satellite navigation receivers for receiving satellite signals such as GPS or BeiDou signals
  • inertial navigation sensors such as accelerometers and gyroscopes, thermometers, compasses, clocks, and/or other sensors.
  • SDCS 110 receives data from sensors 140 using one or more of a number of communication methods. These communication methods may include TCP/IP, Wi-Fi, Bluetooth wireless, or other communications protocols.
  • CAN Bus 150 is in communication with various Electronic Control Unit (ECU) subsystems, for example, steering ECU 160 and throttle/brake ECU 170 as shown in FIG. 1.
  • SDCS 110 may issue commands to steering ECU 160 and throttle/brake ECU 170 via CAN Bus 150.
  • Steering ECU 160 controls the direction of the vehicle.
  • Throttle/brake ECU 170 controls the acceleration and deceleration of the vehicle.
  • User interface 180 may comprise any interface capable of receiving user commands.
  • user interface 180 may be a hardware user-interface such as a touch screen or vocal user interface to enable a driver/passenger to relay their commands by touch or voice.
  • User interface 180 may comprise a wired or wireless interface for receiving commands from another user device, for example a wireless handheld device such as a smart phone or tablet, or from another system external to autonomous driving system 100.
  • software 130 of SDCS 110 implements three modules: perception module 210, cognition module 220, and control module 230.
  • Software instructions for implementing each of modules 210, 220, and 230 may be provided in different memory stores.
  • Perception module 210 implements perception method 300 shown in FIG. 3A.
  • Perception method 300 comprises data receiving step 310, and data processing step 320.
  • Data receiving step 310 comprises receiving sensor data 330 from one or more of sensors 140.
  • the set of data from sensors 140 prior to processing may be referred to as a Rich Point Cloud (RPC) 330.
  • RPC 330 may be derived from various sources, such as a satellite navigation receiver, a sensor equipped with inertial navigation, a radar/LIDAR channel, and/or a camera channel.
  • Data receiving step 310 may capture sensor data comprising RPC 330 at regular time intervals (e.g. every 100 ms) .
  • DSM 350 may be generated as frequently as RPC 330 is captured (e.g. every 100 ms, in the example above) .
  • the generation of DSMs 350 marks the completion of perception and provides a starting point for cognition.
  • data processing step 320 processes RPC 330 to produce DSMs 350.
  • Each DSM may show static objects that are in the vicinity of the vehicle like road markings, green belts, road shoulders, traffic lights and roadblocks, and dynamic objects in the vicinity of the vehicle, for example, other moving vehicles, pedestrians and animals.
  • FIG. 3C is an example of a single DSM 350 representing both static and dynamic objects in example driving situation 301 represented in FIG. 3B.
  • DSM 350 shows objects within a certain radius of vehicle 311 to enable assessment for possible collision from all directions including, front, back, left and right of the vehicle.
  • data receiving step 310 comprises receiving sensor data 330 from one or more of sensors 140 at two or more different times (e.g. t 1 , t 2 , t 3 , separated by 100 ms intervals) .
  • Data processing step 320 may comprise comparing sensor data 330 from one or more of sensors 140 at different times (e.g. t 1 , t 2 , t 3 , separated by 100 ms intervals) to identify objects in a vicinity of the vehicle which are static, and objects in a vicinity of the vehicle which are dynamic.
  • Data processing step 320 may comprise generating a DSM based in part on the identification of dynamic and static objects from the sensor data.
  • Perception method 300 comprises performing data receiving step 310 and data processing step 320 with a frequency F.
  • frequency F may be fixed for a particular autonomous driving system 100 and may depend on the speed at which sensor data or RPC 330 can be captured and processed into DSMs.
  • frequency F varies with the speed of the vehicle, for example, a higher frequency F for a higher vehicle speed.
  • the frequency corresponds to a new DSM being generated for every 1 m travelled by the vehicle.
  • the frequency at which DSMs are generated is fixed, as the vehicle speed increases the distance that a vehicle travels in between successive DSMs will increase. Conversely, as the vehicle speed decreases, the distance that a vehicle travels in between successive DSMs will decrease.
  • a sequence of DSMs (at various times t 1 , t 2 , t 3 , etc. ) is generated while the vehicle is moving. Each DSM in the sequence has an associated time stamp and location identification to identify where the DSM was captured.
  • the sequence of DSMs may be used to make driving decisions by the cognition module 220 which is also referred to herein as a decision-making module.
  • the cognition module of the SDCS 110 may determine the position, velocity and acceleration of all driving situation-related objects in the vicinity of the vehicle at any given time.
  • the coordinates of a DSM may be expressed in polar coordinates, such as logarithmic polar coordinates (or log-polar coordinates) .
  • the forward direction of the vehicle may be used as the baseline of the logarithmic polar coordinate system (i.e. directly in front of the vehicle would be an angle of 0°) .
  • the baseline of the logarithmic polar coordinate system may be dynamic, for example a forward direction when the vehicle is moving forward, and a reverse direction when the vehicle is moving in reverse (i.e. backing up) .
  • Each object or point on an object may be represented in the DSM by two numbers: a first number ⁇ for the logarithm of distance between the object and the vehicle, and a second number ⁇ denoting the angle between the object and the baseline.
  • the position of any object in the vicinity of the vehicle may therefore be represented by ⁇ and ⁇ .
  • ⁇ the baseline is the direction of travel of the vehicle
  • an object directly behind the vehicle 180°.
  • a DSM is divided into a grid of regions.
  • the 360° surrounding the vehicle may be divided into a number of regions each of an equal angle.
  • the distance from the vehicle ⁇ may be divided into regions of equal size.
  • the DSM implements distance ⁇ as a logarithm of the distance of an object from the vehicle
  • the distance from the vehicle represented by each region in the grid will increase logarithmically with distance from the vehicle, if each region in the DSM is equal in size.
  • the logarithmic nature of distance ⁇ facilitates perception and analysis at a higher resolution closer to a vehicle, and perception and analysis at a lower resolution further from the vehicle.
  • Logarithmic coordinates rather than, for example, GPS-84 direct coordinates, enable selective attention: the closer the object it is, the more important it is to calculations of the vehicle’s right of way by the SDCS 110.
  • a DSM is not simply a reflection of nearby objects, but enables the selective attention of the SDCS 110, similar to a human driver.
  • a DSM may omit irrelevant information. For example, there is no need to identify the year, make or model of surrounding vehicles, and no need to capture a pedestrian’s gender, age, clothing, etc. in order for SDCS 110 to understand the driving situation.
  • a DSM comprising a two-dimensional image showing the surrounding objects in logarithmic polar coordinates may be sufficient for cognition module 220 to understand the situation and make driving decisions.
  • data processing step 320 comprises prioritizing and selecting a subset of data from RPC 330 before generating one or more DSMs 350.
  • each object appearing in a DSM may be associated with a set of attributes A.
  • Attributes A may include an object type such as: vehicle, pedestrian, green belt, road shoulder, etc., and a size of the object.
  • the size of an object may be represented by the regions in the grid occupied by the object, and stored in an array. Due to the logarithmic nature of the regions in the grid, an object of a given size will occupy more regions in the grid if it is closer to the vehicle than if it is further from the vehicle.
  • the polar regions are asymptotic as they approach the center of the vehicle. For this reason, polar regions less than a certain distance from a vehicle in a DSM may not be represented in the DSM. Such polar regions may be safely ignored, because they represent regions inside the vehicle.
  • objects represented in a DSM are stored in an array associated with the DSM.
  • the array may be indexed by the position ( ⁇ , ⁇ ) of each object on the grid in the DSM.
  • a large number of objects and their attributes may be stored and quickly searched by the SDCS.
  • “Surface Coefficient of Friction” and “Reflection of Light on the Road” may both be stored for the same coordinate ( ⁇ 1 , ⁇ 1 ) in the grid, and simultaneously retrieved when the SDCS searches for that coordinate ( ⁇ 1 , ⁇ 1 ) . Storing and indexing objects in this manner allows a large number of attributes to be efficiently stored and quickly retrieved.
  • Each DSM may have a timestamp T, representing the time at which the sensor data that was processed to produce the DMS was captured or received.
  • the SDCS may calculate a speed and/or an acceleration of an object represented in a DSM by:
  • ⁇ calculating the change in position ( ⁇ ’, ⁇ ’) of an object A with position ( ⁇ 1 , ⁇ 1 ) in DSM 1 and position ( ⁇ 2 , ⁇ 2 ) in DSM 2 ;
  • FIG. 3C An example DSM 350 is depicted in more detail in FIG. 3C.
  • DSM 350 is generated from an example driving situation 301 which is shown schematically in FIG. 3B as a birds-eye view.
  • FIG. 3B shows an autonomous vehicle 311 driven along a road by an autonomous driving system 100.
  • the road comprises three lanes which are marked by line markings 341, 351, 361 and 371.
  • Line markings 351 and 361 are broken lines, denoting boundaries between lanes of traffic travelling in the same direction. The boundaries marked by line markings 351 and 361 may be safely crossed by vehicle 311 in certain situations.
  • Line markings 341 and 371 are solid lines, denoting boundaries of lanes that vehicle 311 may not cross. For example, line markings 341 and 371 may denote boundaries between lanes of traffic travelling in an opposite direction, or a shoulder of the road.
  • driving situation 301 other vehicles, including vehicles 321 and 331, are seen travelling along the same road and in the same direction as vehicle 311, but in a separate lane from vehicle 311.
  • DSM 350 depicted in FIG. 3C comprises a structured set of data representing objects and the environment in the vicinity of vehicle 311 in driving situation 301.
  • the objects depicted in DSM 350 may be represented in a logarithmic polar coordinate system, with an angle ⁇ , a logarithmic distance ⁇ , and a set of attributes A, as described above.
  • DSM 350 comprises boundaries 312, 322 and 332 of equally sized polar regions concentric with vehicle 311. Due to the nature of the polar coordinates, each polar region represents an increasingly large area from vehicle 311 the further the polar region is from vehicle 311. For example, the region bounded by 332 and 322 is larger than the region bounded by 322 and 312.
  • lanes 341, 351, 361 and 371 are represented by a series of dots. Lanes 351 and 361 are represented by hollow dots to denote the broken line, and lanes 341 and 371 are represented by solid dots to denote the solid lines.
  • a general route plan (a “GRP” , described below) , is represented by lined dots 381.
  • GRP general route plan
  • the parallel lines defining the lanes in FIG. 3B are depicted in FIG. 3C as lines curving toward each other with increasing distance away from the center of the logarithmic polar coordinate system, the center representing the vehicle.
  • Vehicles 321 and 331 are represented in DSM 350 by irregular polygons due to the transformation of a rectangular object in FIG. 3B by the logarithmic polar coordinates used in FIG. 3C.
  • the polar coordinates of DSM 350 result in the display of objects closer to vehicle 311 at a higher resolution, and the display of objects further from vehicle 311 at a relatively lower resolution.
  • lane dots 351 appear further apart closer to vehicle 311, and closer together further from vehicle 311.
  • vehicle 321 is represented by a larger shape than vehicle 331, because vehicle 321 is closer to vehicle 311 than vehicle 331.
  • the representation of closer objects as larger shapes in DSM 350 prioritizes the closer objects in the DSM and tunes the cognition module as described below to emphasize closer objects more than further objects.
  • the DSMs implement selective attention in accordance with Weber-Fechner’s Law, which states that a change ⁇ x in a large data set is more difficult to detect than the same change ⁇ x in a smaller data set (see Scheler G. (2017) . "Logarithmic distributions prove that intrinsic learning is Hebbian" . F1000research. 6: 1222. doi: 10.12688/f1000research. 12130.2. PMC 5639933. PMID 29081973) .
  • the DSM focuses the cognition module on making decisions on objects closer to the vehicle.
  • Each DSM has a time stamp. Differentiation between static and dynamic objects, and the motion (speed and direction) of dynamic objects may be determined by analysing at least two DSMs.
  • the present DSM as expressed as DSM t along with one or more of the DSMs generated prior to the present DSM expressed as DSM t-3 , DSM t-2 , DSM t-1 , etc., may be stored in short-term memory, and they together represent a driving situation, providing information about objects and road markings in the vicinity of the vehicle which may be used for decision making by the cognition module 220 of SDCS 110.
  • SDCS 110 calculates a general route plan (GRP) of the trip from the starting point to the destination.
  • the GRP provides an ideal plan of travel on the roads on which the vehicle should travel to reach the desired destination, assuming no vehicles or other obstacles on the roads.
  • General route planning may be performed with the help of high-definition (HD) maps.
  • the GRP may detail the specific steps required to reach the destination, including which lane to take, when to change lanes, when to turn, etc.
  • the GRP accounts for driving rules such as speed limits, traffic signals, and driving lanes.
  • the GRP does not account for obstacles which may be encountered along the way, for example, other vehicles, pedestrians, and construction or traffic obstacles or detours.
  • the portion of the GRP that is within the vicinity of the vehicle i.e. within the area covered by the instant DSM for the vehicle
  • the vehicle While the vehicle is on the road, other vehicles may be encountered and other disturbances or threats may appear (e.g. road obstructions, detours, emergency vehicles requiring right of way, pedestrians, animals, etc. ) .
  • the vehicle has a right of way (RoW) defined as the physical space in the vicinity of the vehicle which the vehicle may safely move into.
  • RoW right of way
  • SDCS 110 responds by evaluating the current driving situation, and making necessary changes such as changing lanes, stopping, or temporarily deviating from the route set by the GRP. In this instance, the vehicle may follow a different route from the GRP.
  • the new route becomes the actual driving trajectory line, also referred to herein as the real-time route plan (RTRP) .
  • RTRP real-time route plan
  • the steps for calculating the actual driving trajectory line in response to the current situation may be referred to herein as real-time route planning.
  • the required RoW is related to various factors including: the vehicle size, speed, surrounding traffic, etc. The higher the speed is, the greater the RoW needed. Therefore, the RoW may be considered as a non-linear function of various parameters like vehicle length, speed, acceleration, and any other parameter which may impact the space the vehicle requires to safely travel.
  • a required or expected RoW which is the RoW that is required for the vehicle to safely move forward without touching or colliding with other objects.
  • an actual RoW which refers to the actual space that is available for the vehicle to move into.
  • SDCS 110 realizes the necessity for adjustment. Since the SDCS 110 makes its judgment by comparing the required RoW and actual RoW, it constantly performs the steps of detecting, calculating, comparing, requesting, competing, abandoning (a previous decision) or occupying the actual RoW.
  • operating the vehicle in the driving mode comprises: (i) determining the available moving spaces called Right of Way needed for the vehicle based on the change of the driving situation maps; (ii) making real time decisions for the vehicle if the Right of Way available does not meet the requirement to pursue the predefined route plan before the journey begins; (iii) conducting route changes based on the actual Right of Way and knowing when to get back to the predefined route plan after the temporary route deviation.
  • FIG. 4 shows a general route planning method 400 that may be implemented by the cognition module 220 of FIG. 2.
  • FIG. 5A shows a real-time route planning method 500 that may be implemented by the cognition module 220 of FIG. 2.
  • General route planning method 400 and real-time route planning method 500 may be executed while SDCS 110 is operating in driving mode.
  • Various steps of real-time route planning method 500 may also be executed while SDCS 110 is operating in training mode for comparison to a Human Operation Cognitive Arrow generated during a training mode.
  • Driving and training modes of SDCS 110 are described in further detail below.
  • General route planning method 400 comprises steps for receiving a destination 440 and vehicle information 445 at block 410, receiving one or more high-definition maps 421 at block 420, and generating a General Route Plan (GRP) 450 at block 430.
  • GRP 450 may comprise a plurality calibration points used to calibrate the actual route of the vehicle along a journey, and to aid in navigation.
  • Generating GRP 450 at block 430 comprises calculating the ideal path to reach the destination 440, assuming no other vehicles or other obstacles are on the road.
  • the ideal path will depend in part on the vehicle attributes, for example the length, height, and/or weight of the vehicle. As an example, a taller vehicle may be unable to pass under an overpass, necessitating a different path than a shorter vehicle.
  • Generating GRP 450 at block 430 may also depend on roadmaps and traffic rules.
  • GRP 450 may be based on a RoW along the ideal path to destination 440.
  • the required or expected RoW along the ideal path may be referred to as a restraint band.
  • the RoW may be represented graphically on each generated DSM.
  • Destination 440 may be provided by a variety of methods, including from a user through user interface 180. In other embodiments, destination 440 may be predefined for multiple journeys like those taken by cargo trucks, commuting buses and lawn-mowers, in which case destination 440 is stored within cognition module 220, and receiving destination step 410 comprise retrieving destination 440.
  • Real-time route planning method 500 of FIG. 5A comprises receiving input at block 510, and generating output at block 520 to control the actual trajectory of the vehicle in response to the current driving situation, while maintaining the vehicle as close to the general route plan as possible.
  • Input received at 510 may comprise one or more DSMs 530, which are generated by the perception module 210 executing perception method 300 (see FIG. 3A) .
  • the output generated at block 520 comprises a cognitive arrow 550.
  • a cognitive arrow is the expression of a cognitive decision made to control the vehicle.
  • cognitive arrow 550 may represent a driving command comprising a steering direction and an acceleration or deceleration for the vehicle.
  • each cognitive arrow 550 is stored in database 580 along with the DSM set 530 from which it was generated.
  • Database 580 comprises pairs of cognitive arrows and DSM sets.
  • Database 580 may contain DSM sets for common sequences such as crossing an intersection, merging onto a highway, making a right turn in city traffic, making a left turn in city traffic, giving way to a pedestrian, etc.
  • SDCS 110 When evaluating a DSM set to determine a driving action, SDCS 110 first searches in the database for a matching sequence; if one is found, a cognitive arrow may be generated fairly quickly because these cognitive arrows are already stored in memory.
  • database 580 is a graphical database.
  • the SDCS When there is no existing sequence of DSMs that is comparable to the current one, the SDCS will initiate an embedded program based on a predictive algorithm.
  • the algorithm calculates an available RoW for the front, left, and right of the vehicle. When the RoW is expanding, the vehicle may accelerate in the direction of the increasing RoW. When the RoW is shrinking, the vehicle may decelerate in the direction of the decreasing RoW.
  • the algorithm may also implement other pre-defined rules.
  • the pre-defined rules may implement existing driving skills expressed in semantic expressions like “decelerate rather than change lane when the situation is uncertain. ” In the case of such a rule, the SDCS prioritizes decelerating over changing a direction of travel.
  • cognitive arrow 550 is represented graphically.
  • cognitive arrow 550 is represented as a single image.
  • the graphical representation of cognitive arrow 550 may comprise an arrow.
  • a steering direction for controlling the vehicle steering wheel may be indicated by the direction of the arrow.
  • the angle of the arrow with respect to a reference center axis may indicate how much the steering wheel should be rotated and in which direction.
  • Different colors may be used to indicate a particular type of action such as acceleration and deceleration. For example, a red arrow may be used to indicate acceleration, and a blue arrow may be used to indicate deceleration.
  • a magnitude of an acceleration or deceleration may be indicated by a width and/or a length of the arrow.
  • the arrow may have a minimum width and/or length, wherein an arrow of the minimum width and/or length represents a steering direction and an acceleration of zero.
  • Figures 5B – 5H depict seven example cognitive arrows corresponding to the following steering directions and accelerations (wherein a magnitude of the acceleration corresponds to a length of the arrow depicted) :
  • Figures 5I-1 to 5I-8 depict example cognitive arrows for an example overtaking scenario:
  • Figure 5J shows an example method 501 which may be implemented at generate output step 520 (see FIG. 5A) to generate cognitive arrow 550.
  • Method 501 receives a current DSM set 511 comprising 2 or more DSMs.
  • Method 501 comprises at step 521 searching a database of DSMs and associated cognitive arrows to determine, at step 531, if there is a DSM set in the database matching DSM set 511. If there is a matching DSM set, then method 501 comprises at step 541 retrieving the cognitive arrow paired with the matching DSM set. If the database does not contain a matching DSM set, then method 501 comprises at steps 551 and 561 respectively, calculating a RoW and generating a cognitive arrow based on the predictive changes of RoW.
  • the determination of RoW at step 551 takes into account the physical space required for the vehicle to safely advance (without colliding into any objects) .
  • the determination of the cognitive arrow, including representation of the steering direction and magnitude of acceleration, is based on what is required to advance the vehicle to the point along the GRP, or as close to the next point along the GRP if the vehicle is required to deviate from the GRP, as described elsewhere herein.
  • FIG. 6B depicts an example RoW 641 for a driving situation 601.
  • driving situation 601 an autonomous vehicle 611 is traveling along a three-lane road.
  • Vehicles 621 and 631 are travelling in the same direction as vehicle 611, in a lane to the left-hand side of vehicle 611.
  • the RoW 641 is the room available for vehicle 611 to travel into. Because vehicles 621 and 631 occupy the lane to the left-hand side of vehicle 611, RoW 641 does not include the lane to the left-hand side of vehicle 611. Because the lane to the right-hand side of vehicle 611 is unoccupied, RoW 641 includes the lane to the right-hand side of vehicle 611.
  • the DSM has a radius r in the range of 50 and 150 meters (therefore showing objects that are within a distance r of the vehicle) . In certain embodiments, the DSM has a radius r in the range of 50 and 80 meters.
  • the DSM takes the driver as the center of the log polar coordinate system (log ⁇ , ⁇ ) . As the center of the log polar coordinate system is fixed with respect to the center of the vehicle, the DSM moves along with the vehicle.
  • the coordinate baseline may be set as the direction extending straight forward from the center of the vehicle. Whenever there is an object that falls into a RoW section, the closer it is, the more attention it receives from the SDCS 110.
  • the DSM reflects the driver's selective attention.
  • the DSM is a map showing the vehicle’s surrounding situation in a particular instance.
  • the DSM may show the portion of the General Route Plan and/or the actual trajectory of the real-time route plan that is located within the radius r of the DSM which is a direct mapping from the WPS-84 coordination system.
  • the partial General Route Plan appears most of the time as a straight path in front of the vehicle in the DSM.
  • An online predictive control algorithm may be developed for application in the cognition domain.
  • the online predictive algorithm is designed based on the following ideas:
  • the algorithm is designed to calculate and select from available options to move from S 0 to S g , by turning the steering wheel along angles and changing the acceleration along a 0 , a 1 , ..., a n .
  • Such an online predictive control algorithm may be subject to fine-tuning control changes when the vehicle is on the road.
  • Real-time Route Planning (RTRP) combined with predictive control may significantly reduce the computational complexity for determining the output for controlling the vehicle’s ECUs.
  • Convolution neural network (CNN) deep learning methodologies may be applied to develop data-driven algorithms based on pairing of a DSM set with a cognitive arrow for end-to-end decision making.
  • FIG. 6A shows a method 600 according to one embodiment that may be used to generate the output at method 500’s block 520 in one embodiment.
  • Method 600 comprises a real-time route planning step 610 comprising determining a strategic cognitive arrow 650 based on DSM sets 630 and general route plan 640. Determination of strategic cognitive arrow 650 may involve, for example, using the steps as discussed in relation to FIG. 5J’s method 501 to determine a cognitive arrow by calculating a cognitive arrow or determining a best match cognitive arrow, in order to navigate the vehicle along or close to the General Route Plan.
  • the cognitive arrow resulting from the performance of method 501 (depicted in FIG. 5J) is shown as strategic cognitive arrow 650 in FIG. 6A’s method 600.
  • the determination of strategic cognitive arrow 650 at step 610 may be implemented using a deep learning algorithm such as one using a cognitive neural network, as described herein.
  • a specific Cognitive Arrow (referred to as the tactical cognitive arrow) is produced to accommodate the edge driving conditions and keep the vehicle steady and safe.
  • Method 600 also incorporates tactical cognitive processing.
  • An objective of tactical cognitive processing is to ensure steady and safe movement of vehicle 611 as it moves along its real-time route planning trajectory. Therefore, method 600 comprises retrieving a tactical cognitive arrow 670 at step 620.
  • the tactical cognitive processing is realized through capturing a “driver’s fingerprint” .
  • Each experienced driver has his or her unique behavioural patterns and habits for driving.
  • a driver’s personalized driving style, behaviour, and skills may be referred to as the driver’s fingerprint.
  • driving comfort, ride characteristics and energy consumption may be influenced by the driver’s operational style for application of the steering wheel, accelerator or gas pedal, and brake pedal.
  • a driver’s fingerprint may also be used to uniquely identify the driver.
  • the driver’s fingerprint represents the driver’s special skills in maintaining the balance of the vehicle, saving fuel when driving, and keeping passengers comfortable in different road conditions or scenarios (also referred to elsewhere herein as “tactical” cognition; different drivers may produce different tactical cognitive arrows in certain driving conditions in accordance with their unique driver’s fingerprint) .
  • These techniques may not be related to the route that the vehicle is taking or the destination of the trip. For example, in theory if the driver is driving forward at a steady speed, the cognitive arrow should indicate an acceleration of 0. However, even on a straight path the driver controls the vehicle to keep the balance of the vehicle, and maintain comfort for the passengers. This results in a series of cognitive arrows (e.g. tactical cognitive arrows) for control of the vehicle specific to that driver, although such cognitive arrows are not used for reaching the destination (i.e. they are not controls for navigating the vehicle along the route plan) .
  • cognitive arrows e.g. tactical cognitive arrows
  • Different drivers may have different fingerprints, and the tactical cognition process described herein may result in different tactical cognitive arrows for different drivers related to a particular person’s driving style under an edge condition. For example, one driver may rush through a flooded area while another one may drive slowly to pass a flooded area. These two different driving styles would result in two different tactical cognitive arrows for the same driving situation corresponding to two different drivers’fingerprints.
  • parameters that may be monitored and stored include speed, mileage, four-wheel or two-wheel rotating speed, slightly unbalance degree, pitch and roll, engine torque, sideslip angle, body vibration frequency and amplitude, tire pressure, etc. These parameters, along with the commands to control the vehicle as performed by an experienced driver, may form the driver’s fingerprint.
  • the driver’s fingerprint may be stored on a dedicated chip.
  • the parameters and the driver’s commands may be provided as inputs to a machine learning algorithm.
  • Parking the vehicle is an example application of the driver’s fingerprint.
  • the SDCS 110 may be trained by collecting a model driver’s fingerprint for different kinds of parking maneuvers (e.g. parallel parking, parking in a slope, rear-in parking, head-in parking etc. ) , and in different road conditions (e.g. slopes, mud roads, cobblestones, etc. ) .
  • input 660 may be provided which may comprise a library of tactical cognitive arrows paired with real-time data collected from vehicle sensors which are relevant to certain driving conditions (also referred to herein as scenarios) . As seen in FIG. 13, such parameters may include vehicle speed, mileage, tire rotation speed, distortion angle, body course, angle of pitch, parallel rolling, engine torque, and the like.
  • the tactical cognitive arrow 670 may be retrieved based on input 660.
  • the database of tactical cognitive arrows from which tactical cognitive arrow 670 is retrieved may be developed from prior training data.
  • Such training data may be collected by recording the actions of a model driver driving in similar conditions on a straight of way, and by recording the corresponding driving condition parameters.
  • the model driver maintains the vehicle moving forward and in a steady speed.
  • all the driving actions of the driver are primarily for the purpose of keeping the vehicle steady and smooth and have no relationship to where the vehicle is driven to.
  • the driver’s actions may include: angle/torque of wheel, drive pedal displacement and brake pedal displacement (see FIG. 13) .
  • Each of the driver’s actions may be represented, at various points in time, as a tactical cognitive arrow paired with a set of corresponding driving condition parameters.
  • the model driver may be recorded handling the steering wheel and/or pressing or releasing the gas pedal or the brake in a certain manner.
  • a sequence of tactical cognitive arrows may be generated and associated, for example by using a recurrent neural network with corresponding sets of driving condition parameters which are reflective of the flooded road conditions (see: Rául Rojas (1996) . Neural networks: a systematic introduction. Springer. p. 336. ISBN 978-3-540-60505-8) .
  • These training operations may be repeated several times to collect sufficient data, including the vehicle/driving parameters and tactical cognitive arrows, for the library that is provided as input 660.
  • the input to the recurrent neural network is therefore a set of driving condition parameters (recorded while the driver is operating the vehicle under certain conditions in a training mode) and the output is the tactical cognitive arrow. Therefore, machine learning based on a recurrent neural network may be used to train the SDSC 110 to determine a tactical cognitive arrow in response to the vehicle encountering various driving conditions or scenarios while in autonomous driving mode.
  • the tactical cognitive arrows and associated sets of driving condition parameters may be stored in a dedicated memory storage unit.
  • the current driving conditions are matched to similar driving conditions in the tactical cognition memory store, and the tactical cognitive arrow that is paired with those driving conditions may be selected as tactical cognitive arrow 670.
  • various types of driving conditions are identified (e.g. bumpy road, icy road, wet road, snow-covered road, etc. ) and the driving conditions are classified based on the type.
  • Tactical cognition at step 620 may comprise recognizing the type of the current driving condition based on the real-time data collected from the vehicle sensors, and searching through the specific portion of the tactical cognition memory store that relates to that type of driving condition.
  • the computational expense required at step 620 for retrieving a tactical cognitive arrow 670 may be less, or substantially less, than that required for calculating a strategic cognitive arrow 650 at step 610.
  • the tactical cognition module and strategic cognition module are implemented on separate hardware, the modules may be implemented using different hardware tailored to the computational expense of each module.
  • Strategic cognitive arrow 650 and tactical cognitive arrow 670 are combined at block 680 to produce aggregate cognitive arrow 690.
  • block 680 comprises adding the strategic cognitive arrow 650 and tactical cognitive arrow 670.
  • the resulting aggregate cognitive arrow 690 may correspond to cognitive arrow 550 of FIG. 5 and is used to control the vehicle.
  • a steering direction for the vehicle may be indicated by the direction of the arrow 690.
  • An amplitude of an acceleration or deceleration of the vehicle may be indicated by a width or a length of the arrow 690. Acceleration and deceleration may be represented as different colors for the arrow 690.
  • Method 600 depicted in FIG. 6A produces two cognitive arrows, strategic cognitive arrow 650 and tactical cognitive arrow 670.
  • strategic cognitive arrow 650 may be generated to overtake a vehicle
  • tactical cognitive arrow 670 is generated because there is a manhole cover in front of the vehicle.
  • the manhole cover causes an edge driving condition (a bumpy road scenario) that requires certain learned technique to address.
  • SDCS 110 In order to reduce the bumpy experience for the passenger, SDCS 110 generates tactical cognitive arrow 670 directing the vehicle to decelerate (i.e. apply the brakes) .
  • an aggregate cognitive arrow 690 is generated (as described above) which results in strategic cognitive arrow 670 being set off by 670 cognitive arrow 670.
  • the resulting aggregate cognitive arrow 690 will control the vehicle to travel not as fast as indicated by strategic cognitive arrow 650, but not as slow as tactical cognitive arrow 670.
  • Particular embodiments employ deep learning to “train” the autonomous driving system 100 to drive, and more particularly, to train the cognition module 220 in its determination of the output (cognitive arrow 560) based on input received at block 510 (e.g. DSM 530 and GRP 540 in FIG. 5J) .
  • One example embodiment of a machine learning-based method 700 that may be performed by autonomous driving system 100 is shown in FIG. 7 while a model driver operates the vehicle in training mode.
  • the model driver may have extensive driving experience enabling him or her to deal with many kinds of difficulties and unexpected challenges encountered when driving.
  • the model driver may be expected to know how to operate a vehicle smoothly, safely and/or fuel efficiently.
  • Model drivers may also be expected to comply with every driving rule and regulation. Their behavior can be set as a model for training the SDCS 110.
  • a model driver operates the vehicle, while the SDCS 110 receives and processes sensor data 705 at block 710.
  • Sensor data 705 may comprise one or more of the sensor data as described above with reference to sensors 140 of FIG. 1 and may comprise Rich Point Cloud (RPC) data.
  • Sensor data 700 may be processed to produce DSM 720 at block 710 as described above with reference to FIG. 3A.
  • RPC Rich Point Cloud
  • method 700 comprises applying a learning module to generate a Program Generated Cognitive Arrow (PGCA) 740.
  • PGCA 740 may be generated according to FIG. 5A’s method 500 described above for generating a cognitive arrow, which may take into account the right of way (RoW) and route plan (as described with reference to FIG. 5J’s method 501) .
  • PGCA 740 corresponds to cognitive arrow 550 output by method 500.
  • the learning module may also generate a Human Operation Cognitive Arrow (HOCA) 750 according to the actions of the model driver.
  • HOCA 750 may be generated by receiving sensor data from the vehicle as the model driver is driving the vehicle, and processing the sensor data to determine RPC data and/or DSMs.
  • the output i.e. the cognitive decision, comprising, for example, steering direction and acceleration/deceleration amplitude
  • the output may be recorded as the HOCA 750.
  • Method 700 then proceeds to training step 760, wherein PGCA 740 is compared to HOCA 750, and unified cognitive arrow 770 is generated.
  • the difference between PGCA 740 and HOCA 750 may be used to train the algorithm used in method 500 for generating the cognitive arrow 550.
  • Cognitive arrow 770 is used to drive the vehicle. As the algorithm is trained, and differences between PGCA 740, HOCA 750 will diminish, and PCGA 740 and HOCA 750 will converge.
  • one aspect of the invention provides a method of providing driving command based on a learning module embedded in the system.
  • the method comprises: (i) recording a series of driving situations based on the refined and integrated data from one or more sensors in graphic representations; (ii) recording the output of a model human driver’s operation in a graphic command denoting an actual steering direction and an actual acceleration or deceleration; (iii) generating an autonomous driving output by the self-driving system which is represented in a similar graphic command as (ii) ; (iv) determining the differences between the human operation and the system-generated operation; (v) applying machine learning techniques for the system to adapt its Convolutional Neural Network embedded in the system, which would produce further driving commands that approach human operation commands in similar driving situations.
  • a vehicle on the road is in an open and uncertain environment.
  • a driver may encounter fog, snow, rain, hail, ice, strong wind, smoke and/or other unfavorable weather conditions.
  • one or more of the steps performed at blocks 730 and 760 of method 700 may be based on Cognitive Neural Networks (CNN) and/or Recurrent Neural Networks (RNN) .
  • the SDCS 110 may associate each input with a particular output image.
  • the input may comprise a sequence of DSMs with a route plan identifying the next positions of the autonomous vehicle along the route.
  • the output may comprise a cognitive arrow.
  • Input and output pairs (DSM sets – cognitive arrow pairs) are generated and stored in memory storage while the vehicle is being driven by a model driver.
  • the SDCS 110 may search the memory storage for a similar DSM sequence, and identify a cognitive arrow paired with the similar DSM sequence. That cognitive arrow may be then be used as the output for the DSM series encountered by an autonomous vehicle.
  • the system described herein is based on memory cognition.
  • the “memory cognition” employed by SDCS 110 provides for accumulation and continuous application of previous driving “experiences” and “memories” to make immediate decisions through the pairing of DSM sets and cognitive arrows, as explained above.
  • memory cognition may take priority over computational cognition during decision-making. This is described above with reference to FIG. 10, where the memory cognition defines rules that assist the execution module 232 to select a cognitive arrow where more than one cognitive arrow is produced, or to otherwise influence a driving decision in a particular case (e.g. slow down when approaching an intersection) .
  • Different memory sticks or storage areas may be used to store different categories of Cognitive Arrows. These may comprise, for example, a first memory stick for storing Cognitive Arrows for road-crossing, a second memory stick for storing overtaking other vehicles, and a third memory stick for merging onto a highway, etc.
  • SDCS 110 searches through the memory storages to locate existing Cognitive Arrows for the current situation.
  • “Instant memory” refers to the process whereby the SDCS receives RPC and starts to produce a DSM; the working memory is to produce the cognitive arrow to direct the car; and long-term memory stores general rules to guide the production of cognitive arrows. In many occasions, the instant memory and the working memory are working in parallel in the system and the long-term memory facilitates the working memory to produce Cognitive Arrows.
  • Memory cognition may be thought of as the convolution of the cognitive time function and the “forgetting” time function.
  • Instantaneous memory, short-term memory, and long-term memory may be expressed in different formats for perception, decision making and knowledge accumulation respectively.
  • instantaneous memory may be used to refer to the inter-frame correlation for image or radar/LIDAR RPC.
  • instantaneous memory 215 may be engaged in the process of converting the raw data from the sensors 140 into Rich Point Cloud (RPC) data and ultimately into Driving Situation Maps (DSM) 350.
  • a sequence of DSMs 350 is generated using the instantaneous memory.
  • the DSMs may be represented using log polar coordinates to facilitate cognitive recognition of the driving situation.
  • working memory 217 may be employed to generate cognitive arrows from the DSMs. For example, working memory may be engaged for searching the database for a similar DSM sequence.
  • the accumulation of knowledge, expressed as semantic rules with concepts or the scene forms, may be stored in long-term memory 219.
  • Concepts or phases are semantic annotations of pictures.
  • Concept trees may form a driving knowledge atlas. For example, in a crossroad, an experienced driver may decide that he or she would rather slow down than rush out when the situation is complex and uncertainty may appear any minute. The decision of slowing down is made rather from the immediate perception of danger than from a general law of driving.
  • Rules stored in long-term memory may be represented using semantic expressions, such as “slow down rather than change lanes when approaching an intersection; “decelerate when making a turn” , “stop instead of accelerating” .
  • the long-term memory 219 provides rules and guidelines that are applied by the SDCS 110 when generating a cognitive arrow. Such rules and guidelines may assist the SDCS 110, and in particular, the execution module 232, to select a cognitive arrow where there is more than one possible cognitive arrow produced for the control module 230.
  • the SDCS may prioritize the memory cognition in decision-making.
  • Memories may be emphasized (affirmative learning) , restrained (negative learning) , or enhanced during the process.
  • RTRP real-time route plan
  • the long-term memory is the general law of “slowing down is better than rushing out when crossing a road” .
  • the decision-making process would take the suggestion from “long-term memory” for a safer adjustment to slow down.
  • Each sensor has a limit and therefore may be imperfect. With an increased number of sensors installed in a vehicle, the data received is also increased. This may make the task of checking redundant data and merging data into RPC a burden so as to prevent the SDCS 110 from reacting or generating decisions in a timely manner. However, it is not necessary, nor is it suggested that there be an integration among sensor data, such as merging between multiple radar/LIDAR data, between multiple image data, or between radar and GPS navigation data. The feedback provided by memory cognition may avoid the space and time cost for the integration, and significantly improve cognitive performance.
  • a further advantage of this approach is to reduce the data conflict that may exist between sensors. Separation of the autonomous system into three working zones (i.e. perception, cognition, and control) may address the potential problems associated with processing a huge amount of data at once, and may improve the cognitive understanding. As a result, the SDCS 110 does not need to create a 3D perception model to understand the driving situation and make a decision.
  • the methods of SDCS as described above are realized through the systems that are installed in a fully wired automobile, which lays the foundation for complete digital control of the vehicle.
  • Digital control may be in many ways superior to manual control.
  • Digital control enables online data collection of details of the vehicle when it is being manually driven and acquires the data of a driver’s behavior for changes on turning on lights, steering wheel, accelerator, brake pedal, etc.
  • the SDCS 110 may record parameters such as speed, mileage, four-wheel or two-wheel rotating speed, slightly unbalanced degree, pitch and roll, engine torque, sideslip angle, body vibration frequency and amplitude, tire pressure, and the like.
  • Embodiments of the invention described herein may be built with a universal architecture facilitating parallel operations over multiple buses.
  • the construction of a decision bus, learning bus, and interactive bus allows the modules to work concurrently and independently. Therefore, the computation, memory and interaction modules are linked but not entangled.
  • This general architecture also leaves possibilities of extension for future add-ons like tour guide, entertainment and mobile communications.
  • Each bus is responsible for the operation and data processing of its own modules that is connected to it.
  • a virtual exchange system is provided allowing for communications between modules for each bus; no direct point-to-point communication is allowed. Messages may be exchanged in a “subscribe / publish” pattern. This may eliminate the necessity for encapsulation and data analysis. Modules may remain fairly independent by transmitting the “object” directly in between. Modules may be ready to be used and be added, stopped or moved at any point of time as required.
  • the learning bus and the interactive bus enable the SDCS 110 to learn new driver skills, and experiences and knowledge can be accumulated along with the learning process. Learning and interaction tasks do not occupy the bus bandwidth for decision-making, nor would they affect the CAN bus 150.
  • the architecture improves the real-time decision-making performance, so cognitive arrows can be sent to the CAN bus 150 in time.
  • Remote hackers on the internet can be prevented by a safety monitoring module if they attempt to hack in from the interaction bus. Such attempts will not hijack the SDCS 110 from the control of its owner or actual user.
  • a decision making bus, learning bus and interactive bus depend respectively on three physical Ethernet buses.
  • the buses are not connected to each other.
  • the communication among the working modules on each bus is implemented through the virtual switching of the modules.
  • the three virtual switching modules are not connected. Because the communication among the modules is only through the virtual switching module of that bus, the modules can be added or deleted as required for tasks.
  • Working modules may exchange information across the buses. For example, when the operator of the vehicle interactive user interface 180 provides a command to make an unplanned stop at a destination which is not part of the existing General Route Plan (e.g. stop at a particular coffee shop) , the interaction module can communicate that information to the decision bus for the production of a revised General Route Plan which includes a stop at that coffee shop.
  • the interaction module can communicate that information to the decision bus for the production of a revised General Route Plan which includes a stop at that coffee shop.
  • the SDSC 110 has passed actual road tests on various types of vehicles. These vehicles include: 64-seat bus, 5-seat sedan two-seat coup, specialized vehicles like an RV and cleaning vehicles.
  • the road tests include: open highway; closed park like an industrial working zone; city traffic etc. These tests have proved the general applicability of the architecture of SDCS 110, which can adapt to different environments and is capable with dealing with many of the uncertainties while driving.
  • the general architecture has potential for future extensions to IoT (Internet of Things) , Cloud Computing, mobile networks and big data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Traffic Control Systems (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)

Abstract

L'invention concerne des procédés et des systèmes destinés à conduire un véhicule de manière autonome. Des données de capteurs sont reçues d'un ou de plusieurs capteurs et une représentation graphique des données de capteurs, laquelle peut comprendre une carte de situation de conduite, est générée. La carte de situation de conduite peut être représentée au moyen d'un système de coordonnées polaires logarithmique comprenant une dimension d'angle et une dimension de rayon. À partir de la carte de situation de conduite, une commande de conduite graphique, laquelle peut comprendre une flèche décrite graphiquement, appelée flèche cognitive, est déterminée. Le véhicule est conduit sur la base de la commande de conduite graphique pour représenter les paramètres de contrôle désirés. Des paires de cartes de situation de conduite en tant qu'entrée avec les flèches cognitives en tant que sortie peuvent servir à l'apprentissage profond de bout en bout pour entraîner l'algorithme destiné à générer la flèche cognitive.
PCT/CN2018/071516 2018-01-05 2018-01-05 Procédés et systèmes de conduite autonome WO2019134110A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CA3087361A CA3087361A1 (fr) 2018-01-05 2018-01-05 Procedes et systemes de conduite autonome
PCT/CN2018/071516 WO2019134110A1 (fr) 2018-01-05 2018-01-05 Procédés et systèmes de conduite autonome

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/071516 WO2019134110A1 (fr) 2018-01-05 2018-01-05 Procédés et systèmes de conduite autonome

Publications (1)

Publication Number Publication Date
WO2019134110A1 true WO2019134110A1 (fr) 2019-07-11

Family

ID=67143508

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/071516 WO2019134110A1 (fr) 2018-01-05 2018-01-05 Procédés et systèmes de conduite autonome

Country Status (2)

Country Link
CA (1) CA3087361A1 (fr)
WO (1) WO2019134110A1 (fr)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110488872A (zh) * 2019-09-04 2019-11-22 中国人民解放军国防科技大学 一种基于深度强化学习的无人机实时路径规划方法
CN111178402A (zh) * 2019-12-13 2020-05-19 赛迪检测认证中心有限公司 一种自动驾驶车辆道路测试的场景分类方法及装置
CN113126620A (zh) * 2021-03-23 2021-07-16 北京三快在线科技有限公司 路径规划模型训练方法及装置
CN113568324A (zh) * 2021-06-29 2021-10-29 之江实验室 一种基于仿真演绎的知识图谱修正方法
US11432306B2 (en) 2020-08-05 2022-08-30 International Business Machines Corporation Overtaking anticipation and proactive DTCH adjustment
WO2022200217A1 (fr) 2021-03-26 2022-09-29 Volkswagen Aktiengesellschaft Analyse de conducteur basée sur un segment et assistance au conducteur individualisée
US11810364B2 (en) * 2020-08-10 2023-11-07 Volvo Car Corporation Automated road damage detection
TWI841040B (zh) 2022-11-08 2024-05-01 財團法人工業技術研究院 判定自駕車的情境資料的電子裝置及方法
US11998305B2 (en) 2020-05-11 2024-06-04 BraveHeart Wireless Inc. Systems and methods for using a wearable health monitor

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114821536B (zh) * 2022-05-13 2024-02-20 河南科技大学 一种改进yolov5的无人驾驶拖拉机田间障碍物识别方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160236682A1 (en) * 2015-02-13 2016-08-18 Toyota Jidosha Kabushiki Kaisha Vehicle driving support system
CN107097790A (zh) * 2016-02-19 2017-08-29 罗伯特·博世有限公司 用于阐明车辆的车辆周围环境的方法和设备以及车辆
CN107097792A (zh) * 2016-02-19 2017-08-29 罗伯特·博世有限公司 用于运行自动化机动车的方法和设备
CN107161141A (zh) * 2017-03-08 2017-09-15 深圳市速腾聚创科技有限公司 无人驾驶汽车系统及汽车
US20170277182A1 (en) * 2016-03-24 2017-09-28 Magna Electronics Inc. Control system for selective autonomous vehicle control

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160236682A1 (en) * 2015-02-13 2016-08-18 Toyota Jidosha Kabushiki Kaisha Vehicle driving support system
CN107097790A (zh) * 2016-02-19 2017-08-29 罗伯特·博世有限公司 用于阐明车辆的车辆周围环境的方法和设备以及车辆
CN107097792A (zh) * 2016-02-19 2017-08-29 罗伯特·博世有限公司 用于运行自动化机动车的方法和设备
US20170277182A1 (en) * 2016-03-24 2017-09-28 Magna Electronics Inc. Control system for selective autonomous vehicle control
CN107161141A (zh) * 2017-03-08 2017-09-15 深圳市速腾聚创科技有限公司 无人驾驶汽车系统及汽车

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110488872A (zh) * 2019-09-04 2019-11-22 中国人民解放军国防科技大学 一种基于深度强化学习的无人机实时路径规划方法
CN111178402A (zh) * 2019-12-13 2020-05-19 赛迪检测认证中心有限公司 一种自动驾驶车辆道路测试的场景分类方法及装置
CN111178402B (zh) * 2019-12-13 2023-04-07 赛迪检测认证中心有限公司 一种自动驾驶车辆道路测试的场景分类方法及装置
US11998305B2 (en) 2020-05-11 2024-06-04 BraveHeart Wireless Inc. Systems and methods for using a wearable health monitor
US11432306B2 (en) 2020-08-05 2022-08-30 International Business Machines Corporation Overtaking anticipation and proactive DTCH adjustment
US11810364B2 (en) * 2020-08-10 2023-11-07 Volvo Car Corporation Automated road damage detection
CN113126620A (zh) * 2021-03-23 2021-07-16 北京三快在线科技有限公司 路径规划模型训练方法及装置
CN113126620B (zh) * 2021-03-23 2023-02-24 北京三快在线科技有限公司 路径规划模型训练方法及装置
WO2022200217A1 (fr) 2021-03-26 2022-09-29 Volkswagen Aktiengesellschaft Analyse de conducteur basée sur un segment et assistance au conducteur individualisée
DE102021203057A1 (de) 2021-03-26 2022-09-29 Volkswagen Aktiengesellschaft Segmentbasierte Fahreranalyse und individualisierte Fahrerassistenz
CN113568324A (zh) * 2021-06-29 2021-10-29 之江实验室 一种基于仿真演绎的知识图谱修正方法
CN113568324B (zh) * 2021-06-29 2023-10-20 之江实验室 一种基于仿真演绎的知识图谱修正方法
TWI841040B (zh) 2022-11-08 2024-05-01 財團法人工業技術研究院 判定自駕車的情境資料的電子裝置及方法

Also Published As

Publication number Publication date
CA3087361A1 (fr) 2019-07-11

Similar Documents

Publication Publication Date Title
WO2019134110A1 (fr) Procédés et systèmes de conduite autonome
US11698638B2 (en) System and method for predictive path planning in autonomous vehicles
CN112236346B (zh) 用于模拟自主驾驶的方法和设备
US10829116B2 (en) Affecting functions of a vehicle based on function-related information about its environment
JP7433391B2 (ja) 車両とテレオペレーションシステムとの間の対話
US11535255B2 (en) Micro-weather reporting
US20180004210A1 (en) Affecting Functions of a Vehicle Based on Function-Related Information about its Environment
CN110573978A (zh) 用于自驾交通工具的动态传感器选择
JP2021524410A (ja) ドライブエンベロープの決定
US20220105959A1 (en) Methods and systems for predicting actions of an object by an autonomous vehicle to determine feasible paths through a conflicted area
CN109643118B (zh) 基于关于车辆的环境的与功能相关的信息来影响车辆的功能
CN112698645A (zh) 具有基于学习的定位校正系统的动态模型
CN116249947A (zh) 预测运动规划系统及方法
US11815891B2 (en) End dynamics and constraints relaxation algorithm on optimizing an open space trajectory
CN116249644B (zh) 由自主车辆执行出路推断以确定通过交叉路口的可行路径的方法和系统
US11774259B2 (en) Mapping off-road entries for autonomous vehicles
US20220219682A1 (en) Methods and systems for safe out-of-lane driving
CN115339437A (zh) 自主车辆的远程对象检测、定位、跟踪和分类
CN110426215B (zh) 一种用于车辆平顺性测试的模型建立方法及智能驾驶系统
CN117416344A (zh) 自主驾驶系统中校车的状态估计
CN112829762A (zh) 一种车辆行驶速度生成方法以及相关设备
US20220028293A1 (en) Systems and methods for training a driver about automated driving operation using a reliability model
CN117916682A (zh) 使用时空凸形走廊的运动规划
KR20210109615A (ko) 활동에 기초한 인지된 대상체 분류
CN112441016A (zh) 基于齿轮的车辆载荷推断系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18898347

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 3087361

Country of ref document: CA

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18898347

Country of ref document: EP

Kind code of ref document: A1