US20230202467A1 - Automatic Imposition of Vehicle Speed Restrictions Depending on Road Situation Analysis - Google Patents
Automatic Imposition of Vehicle Speed Restrictions Depending on Road Situation Analysis Download PDFInfo
- Publication number
- US20230202467A1 US20230202467A1 US18/115,433 US202318115433A US2023202467A1 US 20230202467 A1 US20230202467 A1 US 20230202467A1 US 202318115433 A US202318115433 A US 202318115433A US 2023202467 A1 US2023202467 A1 US 2023202467A1
- Authority
- US
- United States
- Prior art keywords
- vehicle
- teleoperator
- support server
- speed limit
- remote support
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title description 4
- 238000000034 method Methods 0.000 claims description 34
- 230000035484 reaction time Effects 0.000 claims description 15
- 230000000246 remedial effect Effects 0.000 claims description 15
- 230000000977 initiatory effect Effects 0.000 claims description 13
- 238000003860 storage Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 12
- 230000005540 biological transmission Effects 0.000 claims description 7
- 230000003068 static effect Effects 0.000 description 10
- 230000004044 response Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 238000010801 machine learning Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 6
- 230000001419 dependent effect Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000015654 memory Effects 0.000 description 5
- 238000004088 simulation Methods 0.000 description 5
- 230000001133 acceleration Effects 0.000 description 4
- 230000009471 action Effects 0.000 description 4
- 238000003491 array Methods 0.000 description 4
- 238000013473 artificial intelligence Methods 0.000 description 4
- 230000007423 decrease Effects 0.000 description 4
- 230000007257 malfunction Effects 0.000 description 4
- 230000007246 mechanism Effects 0.000 description 4
- 230000001052 transient effect Effects 0.000 description 4
- 230000003190 augmentative effect Effects 0.000 description 3
- 238000004590 computer program Methods 0.000 description 3
- 231100001261 hazardous Toxicity 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000008447 perception Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000011065 in-situ storage Methods 0.000 description 2
- 230000003278 mimic effect Effects 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 230000005494 condensation Effects 0.000 description 1
- 238000009833 condensation Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 230000010287 polarization Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
- B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
- B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/09—Arrangements for giving variable traffic instructions
- G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
- G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
- G08G1/096708—Systems involving transmission of highway information, e.g. weather, speed limits where the received information might be used to generate an automatic action on the vehicle control
- G08G1/096725—Systems involving transmission of highway information, e.g. weather, speed limits where the received information might be used to generate an automatic action on the vehicle control where the received information generates an automatic action on the vehicle control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
- B60W30/14—Adaptive cruise control
- B60W30/143—Speed control
- B60W30/146—Speed limiting
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units, or advanced driver assistance systems for ensuring comfort, stability and safety or drive control systems for propelling or retarding the vehicle
- B60W30/14—Adaptive cruise control
- B60W30/16—Control of distance between vehicles, e.g. keeping a distance to preceding vehicle
- B60W30/162—Speed limiting therefor
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/0011—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot associated with a remote control arrangement
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/01—Detecting movement of traffic to be counted or controlled
- G08G1/052—Detecting movement of traffic to be counted or controlled with provision for determining speed or overspeed
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/164—Centralised systems, e.g. external to vehicles
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/165—Anti-collision systems for passive traffic, e.g. including static obstacles, trees
-
- G—PHYSICS
- G08—SIGNALLING
- G08G—TRAFFIC CONTROL SYSTEMS
- G08G1/00—Traffic control systems for road vehicles
- G08G1/16—Anti-collision systems
- G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2520/00—Input parameters relating to overall vehicle dynamics
- B60W2520/10—Longitudinal speed
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2555/00—Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
- B60W2555/60—Traffic rules, e.g. speed limits or right of way
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2556/00—Input parameters relating to data
- B60W2556/45—External transmission of data to or from the vehicle
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
- B60W2720/00—Output or target parameters relating to overall vehicle dynamics
- B60W2720/10—Longitudinal speed
Definitions
- This disclosure relates generally to teleoperated vehicles, and more specifically to methods of maintaining vehicle speed within desired limits.
- teleoperation of vehicles as an auxiliary technology is employed.
- the autonomous vehicle whenever the confidence level of the machine intelligence agent controlling the vehicle declines past a certain threshold, or onboard sensors report abnormalities or certain events that require manual handling, the autonomous vehicle generates a request to be taken over by a remote human operator who then navigates the vehicle to safety using video feeds and sensor measurements provided by the vehicle.
- the teleoperation channel is usually terminated and the local machine intelligence agent continues navigating the vehicle to its destination.
- the take-over request may be fulfilled by a more capable machine intelligence agent hosted on a cloud server.
- a vehicle safety system imposes speed restrictions during a teleoperation session that enables safe teleoperation of the vehicle.
- the vehicle establishes a connection with a remote support server for enabling a teleoperation session in which the vehicle is controlled by a remote teleoperator.
- the vehicle receives, during the teleoperation system, control signals from the remote support server for controlling motion of the vehicle.
- a safety system obtains a depth map indicative of respective distances to objects in an environment of the vehicle.
- the safety system determines, based on the depth map, obstacles in the environment of the vehicle.
- the safety system receives sensor data from a sensor array indicative of a vehicle motion state and determines, based on the vehicle motion state and the one or more obstacles, a speed limit for limiting speed of the vehicle during the teleoperation session that enables safe teleoperation of the vehicle.
- the safety system determines if the vehicle is operating above the speed limit, and initiates a remedial action responsive to the vehicle operating above the speed limit.
- the remedial action may include sending a notification to the remote support server or automatically imposing speed constraints on the vehicle.
- FIG. 1 is a block diagram of teleoperated vehicle environment.
- FIG. 2 is a flowchart illustrating a method for building and maintaining a depth map.
- FIG. 3 is a flowchart illustrating a method for discriminating between mobile and static obstacles.
- FIG. 4 is a flowchart illustrating a method for evaluating vehicle speed safety.
- FIG. 5 is a flowchart illustrating a method for constructing a map of safe vehicle speeds.
- a speed limit is determined for which the vehicle can be safely teleoperated.
- a safety system senses data relating to the vehicle environment and generates a depth map from which obstacles in the vicinity of the vehicle can be identified. Based on the detected obstacles and a motion state of the vehicle, a speed limit is determined at which the vehicle is predicted to be able to implement emergency braking command (e.g., directly by the vehicle or by the teleoperator) to avoid a collision.
- the speed limit may be automatically applied to the vehicle or may be provided to the teleoperator to enable the teleoperator to adjust the vehicle speed.
- the vehicle system may maintain restrictions on vehicle kinematics to give the emergency braking subsystem sufficient time to react to prevent a collision in response to the teleoperator initiating an emergency braking command or automatically in response to a disruption in control or communications with the teleoperator.
- FIG. 1 illustrates an example embodiment of an environment 100 for teleoperating a vehicle.
- a remote support server 101 communicates with one or more connected vehicles 102 to provide teleoperation or other support services.
- a video stream capturing a vehicle environment is provided to the remote support server 101 .
- the video stream may optionally be presented at a teleoperation workstation 103 for a human teleoperator.
- other sensor data or video streams may be provided from the vehicle 102 or external sensor arrays 104 to the remote support server 101 .
- a human teleoperator at the teleoperation workstation 103 may view the video stream on a display and provide control inputs via a control input device.
- the remote support server 101 may present the video in real-time and provide real-time teleoperation control signals to control the vehicle 102 to enable the teleoperator to remotely drive the vehicle.
- the remote support server 101 may comprise an artificial intelligence agent that does not necessarily require a teleoperator workstation 103 with a display or physical controls for providing human input.
- the remote support server 101 may provide control instructions to the vehicle directly based on the processing of a real-time video feed and other sensor data without necessarily utilizing any human input.
- the remote support server 101 may comprise a semi-robotic agent that interacts with a teleoperator workstation 103 in a similar manner as a human teleoperator.
- the teleoperator workstation 103 may be coupled to the remote support server 101 via a local area network connection, a direct wired connection, or via a remote connection.
- a teleoperator workstation 103 may include a display to enable a human teleoperator to view real-time video of the vehicle environment and controls for enabling a human teleoperator to control the vehicle.
- the video may include at least a front view that mimics or approximates the view seen by a driver within the vehicle.
- the video may include additional views, such as a rear view video, side view videos, or other views that may mimic the views seen by a driver in mirrors of a traditional vehicle or may include other views not necessarily available to a driver of a traditional vehicle.
- the controls may include controls that mimic those available within a traditional vehicle such as a steering wheel, acceleration pedal, and brake pedal.
- controls may be available at the teleoperator workstation 103 such as a joystick, mouse, touch screen, voice control system, gesture control system, or other input mechanism to control one or more aspects of the vehicle.
- the teleoperator workstation 103 may be omitted.
- the connected vehicle 102 comprises a vehicle safety system 105 , a plurality of sensor arrays 106 (such as LIDAR, radar, sonar, or stereo cameras), and a drive-by-wire system 107 .
- Other components of the vehicle 102 such as a communication system for wireless communicating with the remote support server 101 or external sensor arrays 104 and an on-board computer for linking and controlling various components of the connected vehicle 102 are omitted from FIG. 1 for conciseness.
- implementation of one or more components of FIG. 1 may include a processor and a non-transitory computer-readable storage medium that stores instructions for carrying out the functions attributed to the various components described herein.
- the drive-by-wire system 107 receives control signals from the remote support server 101 and controls operation of the vehicle 102 in response to the control signals to enable teleoperation of the vehicle 102 .
- the drive-by-wire system 107 may receive steering control signals, braking control signals, acceleration control signals, or other vehicle control signals to control operation of the vehicle 102 when being teleoperated.
- the drive-by-wire system 107 may furthermore provide sensor data to the remote support server 101 to enable the remote support server 101 to generate the control signals in response to the sensed information (either based on human teleoperator controls or from an artificial intelligence agent).
- the drive-by-wire system 107 may furthermore control certain actions directly based on received sensor data and/or data from the safety system 105 , without necessarily requiring control signals from the remote support server 101 .
- the onboard sensor array 106 includes one or more sets of sensors for capturing information about the vehicle operation and/or the vehicle environment.
- the sensors may include, for example, one or more cameras for capturing images or video, an inertial measurement unit (IMU) for capturing motion data of the vehicle, a global navigation satellite system (GNSS) for capturing position data relating to the vehicle, a computational unit such as Doppler analysis unit or tire modelling unit to determine operating characteristics of the vehicle, or other sensing systems.
- the onboard sensor array 106 may furthermore comprise a network interface for receiving data from external data sources (e.g., Internet-based sources) that provide data such as current air humidity, estimates of moisture condensed on the road surface, or other information.
- the onboard sensor array 106 furthermore includes a time-of-flight ranging device such as pulse or continuous-wave LIDAR, radar, or sonar to perform depth estimation.
- the safety system 105 generates a set of safety parameters based on sensed information that may be utilized by the drive-by-wire system 107 directly or may be sent to the remote support server 101 to assist in teleoperation of the vehicle 102 .
- the safety system 105 periodically requests a data set from one or more data sources that includes information relevant to safely navigating the vehicle 102 . Responsive to receiving a data set from a source, the system 105 determines kinematic parameters such as vehicle steering angle, speed, and trajectory in substantially real time, and stores these or other parameters in transient or permanent storage onboard the connected vehicle 102 .
- the system may request data sets episodically or under certain circumstances, or utilize other mechanisms for data consumption such as publish-subscribe or push-pull.
- the safety system 105 may include a kinematic computational unit (KCU) 108 , a vehicle speed control programmatic interface 109 , a safety computational unit 110 , and a depth estimation system 111 .
- KCU kinematic computational unit
- the depth estimation system 111 provides depth estimates for objects present in the video feed captured by the onboard sensor array 106 (or external sensor array 104 ) and generates a depth map representing respective distances to detected objects.
- the depth map may comprise an array of pixels or blocks of pixels associated with an image in which a depth of each pixel or block of pixels represents a distance to the nearest object depicted at that location.
- the depth map may comprise a three-dimensional array indicating whether or not an object is detected at each three-dimensional location within a volume.
- the depth map may represent a volume in a vicinity of the vehicle 102 .
- the depth map may be limited to locations that intersect a predicted trajectory of the vehicle 102 over a certain time range. Discrete regions of the depth map that intersect the predicted trajectory of the vehicle 102 represent obstacles. Depending on the environment and the trajectory of a vehicle 102 , such obstacles may include other vehicles, pedestrians, light poles or building facades.
- the depth estimation system 111 may comprise, for example, a structured light depth camera suitable for outdoors environment mapping based on any appropriate coding scheme, such as sinusoidal, binary or de Brujin coding.
- the structured light camera uses a narrow-band light source and a narrow-band filter to remain functional in sunlight.
- the structured light camera uses a polarized light source and a polarization filter to remain functional in sunlight.
- the structured light camera uses light concentration techniques to sequentially illuminate and scan the complete angular range while reducing the time required to achieve the necessary signal-to-noise ratio at each step.
- the depth estimation system 111 may comprise an artificial neural depth estimator that includes a monocular camera and an appropriately trained machine learning (ML) system executed on an onboard computer such as a convolutional neural network (CNN).
- ML machine learning
- CNN convolutional neural network
- the ML system performs depth estimation for each frame captured by the camera.
- the ML system may additionally be trained to use the motion parallax effect to recover a depth map.
- the depth estimation system 111 receives stereo images from a stereo vision camera and estimates a depth map based on stereo disparity using state of the art computer vision (CV) methods.
- the depth estimation system 111 does not necessarily include a dedicated camera and may estimate the depth map from images and/or video received from cameras of the onboard sensor array 106 and/or external array 104 .
- the kinematic computational unit (KCU) 108 generates predictions about a time until the vehicle 102 will collide with an object absent some remedial action, based on the current vehicle state, object states, and sensed environmental conditions. For example, in an embodiment, the KCU 108 receives various kinematic parameters (e.g., position, speed, acceleration, vehicle trajectory, etc.) of the vehicle, a depth map representing distances to detected objects in a vicinity of the vehicle 102 , and time information from a system clock running with a certain degree of granularity. Based on the received information, the KCU 108 updates an estimate of the time until a collision of the vehicle 102 following the current trajectory with an obstacle, provided that a set of assumptions on the behavior of the environment is valid. These estimates may be made available to other components of the vehicle safety system 105 or may be made available to other vehicles 102 in the vicinity. For example, the time to collision may be provided via a dedicated programmatic interface or transmitted over a wireless network for consumption by remote operation software.
- the vehicle speed control programmatic interface (VSCPI) 109 generates control signals for manipulating the speed of the vehicle 102 in response to various safety considerations.
- the VSCPI 109 may directly control speed (e.g., through interaction with the drive-by-wire system 107 ) or via may control speed indirectly via a related variable such as engine power or torque.
- the VSCPI 109 may be accessed by other components of the vehicle 102 via a dedicated programmatic interface to enable the other components to provide data for controlling speed of the vehicle 102 . Additionally, the VSCPI 109 may be accessible by the remote support server 101 via a wireless network.
- the safety computational unit (SCU) 110 restricts the speed of the vehicle 102 using the VSCPI 109 in such a manner that the estimated time to collision computed by the KCU 108 exceeds the sum of the estimated remote operator reaction time and the braking time as computed using an appropriate method.
- the SCU 110 may be permanently active, or activate responsive to certain conditions being met, or activate responsive to signals emitted by other vehicle 102 components.
- the SCU 110 maintains a record of the velocity V and the steering angle ⁇ of the vehicle 102 using data from the onboard sensor array 106 , commands from the drive-by-wire system 107 , or other information sources.
- the system 105 additionally maintains a record of the expected vehicle 102 acceleration and steering angle derivative using methods such as differentiation of the velocity V and the steering angle ⁇ time profiles, analysis of the profile of drive-by-wire commands received by the vehicle 102 , or other methods.
- the SCU 110 may be designed to limit the speed in a manner that enables the vehicle to be decelerated in time to avoid a collision in response to an emergency braking signal from a teleoperator.
- the SCU 110 may be configured to automatically decelerate the vehicle as an automatic precaution in the event that the vehicle loses a connection to the remote support server during a teleoperation session.
- the speed limit may be set in a manner that enables the SCU 110 to bring the vehicle to a stop and avoid a collision in response to detecting the loss of the connection.
- the safety system 105 may be implemented wholly or partly on a computing system external to the vehicle 102 .
- the safety system 105 or components thereof may be implemented on the remote support server 101 , on a fleet management that manages a fleet of vehicles, or on another network device.
- FIG. 2 is a flowchart illustrating an embodiment of a method for building and maintaining a depth map 200 .
- the safety system 105 requests 201 a data set of sensor measurements.
- the safety system 105 may request the data from the onboard sensor arrays 106 , the external sensors 104 , other vehicles 102 , or a combination thereof.
- the sensor data may be requested on a periodic basis or in response to some event. Alternatively, sensor data may be pushed to the safety system 105 without an explicit request.
- the safety system 105 receives 202 a sensor data set responsive to the request. Responsive to receiving 202 the data set, the safety system 105 computes and updates 203 a depth map 200 representing distances to objects detected in the environment of the vehicle 102 .
- the depth map 200 may comprise, for each of a plurality of detected points in a vicinity of the vehicle 102 , a distance parameter representing a distance between the vehicle 102 and the object, and an angle parameter representing an angle between the vehicle 102 and the object.
- the depth map 200 may be generated over sufficient angular dimensions dependent on the specific environmental factors.
- the depth map 200 may be generated or updated in substantially real time (e.g., using a request-reply mechanism).
- the safety system 105 may request data sets episodically or under certain circumstances, or utilize other mechanisms for data consumption such as publish-subscribe or push-pull.
- FIG. 3 is a flowchart illustrating an example embodiment of a process for classifying obstacles detected in the depth map 200 as either mobile obstacles or static obstacles.
- the safety system 105 classifies obstacles as mobile obstacles (e.g., likely to correspond to other vehicles) if they lie within a mobile obstacle zone representing an area within a threshold lateral distance from the vehicle 102 . Otherwise, the safety system 105 classifies obstacles as static obstacles if they lie outside of the mobile obstacle zone (beyond the threshold lateral distance from the vehicle 102 ).
- the safety system 105 computes 301 edges of the mobile obstacle zone based on the information on the lane occupied by the vehicle 102 (which may be obtained based on prestored map data, real-time sensor data, or a combination thereof), the position of the vehicle 102 within that lane, and the layout of the road network in the immediate vicinity of the vehicle 102 .
- the safety system 105 For each of the points in a cluster corresponding to an obstacle, the safety system 105 transforms 303 the depth map data (which includes, for each point, an angle and distance from the vehicle 102 to the object) to a set of spatial coordinates (e.g., an x-y-z location).
- the X axis is directed parallel to the symmetry axis of the vehicle 102 (for example, coinciding with the direction of the lane)
- the Y axis is directed normally clockwise from the positive direction of the X axis
- the Z axis completes the left-handed set.
- the safety system 105 determines the (x,y) coordinate pairs by multiplying the measured depth associated with each point by the sine and cosine of the angle ⁇ between the X axis and the projection of the direction to the point on the XY plane, and determines a z coordinate by multiplying the cosine of the angle between the X axis and the projection of the direction to the point on the XZ plane.
- the system 105 then collapses 304 the points P of each obstacle 300 into a linear form corresponding to the surface closest to the vehicle 102 :
- a leading vehicle in front of the vehicle 102 may be represented as a line approximating the surface location of its rear bumper, while a trailing vehicle behind the vehicle 102 may be represented as a line approximating the surface location of its front bumper.
- the safety system 105 determines 305 the locations of the lateral edges of each obstacle. For example, the safety system 105 determines the values of the outermost coordinates of the linear form:
- the safety system 105 determines 306 whether the outer edges overlap the mobile obstacle zone. For example, the safety system 105 determines whether the y min is higher than the current left edge of the mobile obstacle zone and whether the y max is lower than the current right edge of the mobile obstacle zone. The safety system 105 then classifies 307 each obstacle as either mobile or static dependent on the overlap with the mobile obstacle zone. For example, if either of the above described overlap conditions are met, the safety system 105 classifies the obstacle as a mobile obstacle. Otherwise, the safety system 105 classifies the obstacle as a static obstacle. Steps 304 , 305 , 306 , 307 may repeat for each detected obstacle.
- system 105 may perform additional analysis for obstacles in the static zone (outside the mobile obstacle zone) to predict a likelihood of them entering the mobile obstacle zone within a relevant time interval.
- the safety system 105 may determine static obstacle zone geometry using data acquired from external data sources such as aerial or satellite imagery, cadaster records, or interfaces provided by public agencies or corporate entities.
- the system safety 105 may use the time profile of the position of a mobile obstacle m i relative to the vehicle 102 to determine its velocity v i relative to the ground, heading angle ⁇ and steering angle C.
- the safety system 105 may further use the information on dynamics of mobile obstacles during simulations to compute the position and heading of a mobile obstacle ⁇ x i (t),y i (t), ⁇ i (t) ⁇ , which in conjunction with the known geometric size of a mobile obstacle may be used by the safety system 105 to determine bounding boxes around each mobile obstacle.
- the bounding boxes may be utilized to more accurately track locations of the mobile obstacles and predict collisions.
- FIG. 4 illustrates an example embodiment of a method for evaluating vehicle speed of teleoperated vehicles to determine speed limits for operating the vehicles 102 .
- the KSU 108 may perform kinematic simulations of the vehicle 102 and its environment for multiple time points ⁇ j ⁇ [0,T] along the predicted vehicle 102 trajectory in the following manner.
- the KSU 108 may first obtain 401 sensor data.
- the sensor data may include information from a local sensor array 106 comprising sensors such as IMU, GPS, Doppler effect sensors or other devices as described above. Additionally, sensor data may be obtained from an external sensor array 104 .
- the KSU 108 may also additionally obtain 402 auxiliary information such as pending but non-executed remote operator commands, transmissions from neighboring vehicles 102 over vehicular ad-hoc networks or over wireless infrastructure that may influence the solution but is not reflected in currently observable kinematics.
- the KSU 108 then predicts 403 motion of the vehicle 102 and the detected mobile obstacles.
- the KSU 108 may adjust the interval between time points x responsive to changes in parameters describing the vehicle 102 such as onboard computer performance.
- the KSU 108 determines 404 a speed limit based on the motion data.
- the KSU 108 may detect the maximum speed at which the vehicle 102 can travel at a given time during a teleoperation session that will enable an emergency braking command to stop the vehicle in sufficient time to avoid a collision under the present conditions.
- the speed limit may be based on a time to bring the vehicle to a stop in response to an emergency braking signal from a teleoperator.
- the speed limit may be based on a time to bring the vehicle to a step as an automatic precaution in the event that the vehicle loses a connection to the remote support server during a teleoperation session.
- the speed limit may vary dependent on the conditions and may be dynamically updated over time during the teleoperation session. For example, the speed limit may decrease on more congested roadways or on roadways with a greater number of obstacles. Additionally, the speed limit may adjust dependent on whether the vehicle is being teleoperated by a human or computer agent, or may adjust dependent on varying profiles of different human operators and their respective reaction times. Further still, the speed limit may adjust dependent on current network conditions to account for varying latency that may be introduced in the time it takes for video or other sensor data to reach the teleoperator and/or the time it takes for teleoperation commands to be received an executed by the vehicle 102 .
- the KSU 108 determines the speed limit by performing simulations of the vehicle 102 trajectory and speed under the assumption that a malfunction occurs at a given time point ⁇ . The KSU 108 may then determine whether at the speed the vehicle 102 is estimated to travel at that time ⁇ , a collision of the vehicle 102 with an obstacle may occur before the vehicle 102 comes to a complete halt if an emergency slowdown command is initiated at that time.
- the system 105 represents the time ⁇ T allocated for emergency slowdown based at least in part on a time period ⁇ t b representing the practical deceleration capacity of the vehicle 102 , which in turn may depend on ABS availability, humidity of the road surface, tire model and other physical factors.
- the safety system 105 may furthermore determine the time ⁇ T based at least in part on a predefined cooldown period ⁇ t c .
- the cooldown period ⁇ t c may represent a preconfigured delay value that may be set by teleoperation administrator or adjusted automatically. Adjusting the cooldown period ⁇ t c enables calibration of the false positive rate. For example, setting a higher cooldown period ⁇ t c increases the predicted time to a collision and thus may decrease the likelihood of determining that a slowdown or emergency action is needed. Setting a lower cooldown period ⁇ t c decreases the predicted time to a collision and thus may increase the likelihood of determining that a slowdown or emergency action is needed.
- the cooldown period ⁇ t c may be helpful to calibrate the time ⁇ T allocated for emergency slowdown based on a predicted likelihood that a critical ingress communication channel such as the command and control link with the remote teleoperator or machine intelligence agent becomes severed or malfunctions.
- the time ⁇ T is based at least in part on a remote operator reaction time ⁇ t r .
- the reaction time may be determined based on, for example, the camera exposure time, video frame onboard processing time, video frame transmission time
- ⁇ ⁇ t n ⁇ 1 ⁇ rtt 2 + 8 ⁇ S 1000 ⁇ BW
- ⁇ rtt is the estimate of the network round trip time between the vehicle 102 and the teleoperation workstation 103
- S is the frame size in bytes
- BW is the estimated network throughput in megabits per second
- the video frame remote processing time (which may in turn include a safety margin of
- F is the teleoperation workstation 103 monitor refresh rate in Hz if the remote operator is an agent consuming the video feed via a visual display or a plurality thereof), estimated remote operator reaction time ⁇ t react , and the control message transmission time ⁇ t n2 determined similarly to ⁇ t n1 .
- the safety system 105 may use the remote operator reaction time ⁇ t react in scenarios where a malfunction of a critical egress communication channel such as the video feed link or relevant hardware implements is detected.
- the KSU 108 determines the maximum speed at which a collision can be avoided if emergency braking is initiated at the time point max( ⁇ + ⁇ t react , ⁇ + ⁇ t c ).
- ⁇ + ⁇ t react represents the earliest time that the remote server 101 can initiate an emergency brake
- ⁇ + ⁇ t c represents a configurable waiting time based on the cooldown period ⁇ t c .
- the KSU 108 may then transmits 405 a message indicative of the detected condition.
- the message may specify the determined speed limit itself and/or may comprise an indicator indicating whether or not the vehicle 102 is exceeding the speed limit.
- the KSU 108 may publish a message to the controlling agent (e.g., the remote support server 101 or SCU 110 ) indicating that the selected speed profile may be hazardous or make this information available to other components of the safety system 105 .
- the controlling agent e.g., the remote support server 101 or SCU 110
- the remote support server 101 may provide an indicator on the teleoperation workstation 103 that alerts a human teleoperator to the speed limit (or an indicator indicating when the teleoperated vehicle 102 is exceeding the speed limit) to enable the human teleoperator to optionally decrease the vehicle speed.
- the speed indicator may be displayed as an overlaid element on the video or may be provided via an audible alert.
- the KSU 108 may transmit the information describing the vehicle's 102 estimated future trajectory and estimated safe stopping distance to the human agent at the teleoperator workstation 103 .
- the teleoperator workstation 103 may use this information for the purpose of augmenting the visual representation displayed to the operator.
- the estimated vehicle 102 trajectory may include the predicted arc or path the vehicle is estimated to traverse in the near future, including some uncertainty or estimated error, and the estimated safe braking distance required to altogether avoid or minimize the impact of a collision.
- the estimated trajectory information may be based on the current input to the vehicle control system by either the devices capturing the operator's input or the onboard autonomous circuit.
- the teleoperator workstation 103 may present the estimated trajectory and braking distance information to the operator as an augmented reality (AR) overlay on top of a monoscopic or stereoscopic video presentation layer.
- AR augmented reality
- the vehicle 102 that is exceeding the determined speed limit may automatically adjust its speed to speed limit.
- the remote support server 101 may automatically send a control signal to adjust the vehicle's speed 102 .
- the SCU 110 may automatically adjust the vehicle speed only in the case that the remote operator has been observed to disregard warning messages.
- the safety system 105 performs the kinematic simulations under the assumption that the onboard computer only manipulates vehicle 102 speed v in an emergency slowdown mode and that the steering angle ⁇ is static, enabling the reduction of the time necessary for computing the solution and keeping the trajectory of a malfunctioning vehicle 102 predictable for agents controlling other vehicles 102 .
- the KSU 108 may compute and publish information describing a set of possible trajectories based on the vehicle's 102 current telemetry information (such as its velocity v or steering angle ⁇ ).
- the elements in the set of possible trajectories may each be classified as paths which would completely avoid a collision with any obstacle 300 , paths which would minimize the impact of a collision with an obstacle 300 , and paths which would result in a maximum-impact collision.
- the teleoperator workstation 103 may display each path to the operator in a color-coded or otherwise differentiated representation in order to provide the operator with sufficient information to make a decision regarding the trajectory to pursue.
- the teleoperator workstation 103 may present the calculated path information to the operator as an augmented reality (AR) overlay on top of a monoscopic or stereoscopic video presentation layer.
- AR augmented reality
- the teleoperator workstation 103 may provide the remote with a user interface component or application programming interface endpoint to override the decision of the SCU 111 regarding vehicle 102 speed limit adjustment.
- a user interface component or application programming interface endpoint may be used in case of sensor malfunction or data set processing errors.
- FIG. 5 is an example embodiment of a process for precomputing speed limits for a teleoperated vehicle at a remote support server 101 .
- This process enables the safety system 105 to request a pre-computed maximum vehicle speed limit data set from the remote support server 101 and use it to reduce the computational requirements of the KSU 108 .
- An example procedure of constructing such a data set may be employed as follows.
- the remote support server 101 loads 501 a regional map of static obstacles such as buildings, road edges, light pole positions or other types of objects to be avoided.
- the remote support server 101 then processes 502 the map to identify the surfaces of the road network in that region and build a polygon mesh, with each polygon corresponding to a discrete segment of the road network.
- the remote support server 101 then computes 503 an array of maximum allowed vehicle speed values using analytical calculations or numerical simulations for the entire range of possible steering angle values for each such road segment.
- the speed limit for a given road segment and steering angle may represent the maximum speed at which a vehicle traversing that road segment and being operating at the given steering angle can be decelerated by a teleoperator in sufficient time to avoid colliding with a static obstacle.
- the remote support server 101 may then adjust the determined speed limits based on the mapping between road segments and steering angles. For example, the remote support server 101 may determine 505 for each (s n ,s m ) tuple representing adjacent road segments, whether the difference between the maximum allowed vehicle 102 speed values for these segments v n ⁇ v m exceeds a threshold difference (based on, for example, the braking capabilities of the vehicle or other factors) as previously determined from the path and time available for the transition between the segments. If the difference between speed limits exceeds the threshold difference, the remote support server 101 may further adjusts v n to ensure a smooth transition between subsequent segments.
- a threshold difference based on, for example, the braking capabilities of the vehicle or other factors
- the remote support server 101 may provide the safety system 105 with a data set that includes speed limit information computed accounting for mobile obstacles moving along some trajectories and at some speeds relative to the vehicle 102 .
- a data set may be comprehensive or restricted to a range of scenarios previously requested by the safety system 105 , or restricted to a range of scenarios determined as probable to be encountered during the current trip by the remote server 101 .
- the remote support server 101 may provide the safety system 105 with a data set including manually or automatically pre-labeled geoinformation on hazardous areas.
- a data set may be solely defined by geographic coordinates of the vehicle 102 or any landmarks involved, geographic coordinates in conjunction with temporal intervals, or custom semantic rulesets.
- the remote support server 101 may provide the safety system 105 with a data set based in part on information obtained by analyzing speed limits observed by in situ human drivers or remote vehicle operators. This may allow to account for factors that are not reliably identified by machine learning algorithms but are identified by human agents with a sufficient degree of accuracy.
- the remote support server 101 may provide the safety system 105 with a data set including information on weather conditions and derivative parameters.
- the data set may include information on expected air humidity and temperature levels, and based on these characteristics the onboard computer may compute the probability and severity of fog condensation, which may in turn be used in maximum permitted speed calculations.
- the remote support server 101 may provide the safety system 105 with a data set accounting for specific vehicle model, road surface humidity level, absence or presence of an ABS system, total load of the vehicle or other parameters.
- the remote support server 101 may provide the safety system 105 with an updated data set episodically, periodically, according to a schedule or in substantial real time.
- Such updates may each comprise a full dataset package, a cumulative incremental package or a differential incremental package.
- the safety system 105 computes paths on the road network that are optimal according to a specific cost function as well as satisfy the safety conditions.
- Certain aspects of the embodiments include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the embodiments can be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems. The embodiments can also be in a computer program product which can be executed on a computing system.
- the embodiments also relate to an apparatus for performing the operations herein.
- This apparatus may be specially constructed for the purposes, e.g., a specific computer, or it may comprise a computer selectively activated or reconfigured by a computer program stored in the computer.
- a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.
- Memory can include any of the above and/or other devices that can store information/data/programs and can be transient or non-transient medium, where a non-transient or non-transitory medium can include memory/storage that stores information for more than a minimal duration.
- the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
- Coupled along with its derivatives.
- the term “coupled” as used herein is not necessarily limited to two or more elements being in direct physical or electrical contact. Rather, the term “coupled” may also encompass two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other, or are structured to provide a thermal conduction path between the elements.
- the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion.
- a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Abstract
Description
- This application is a continuation application of U.S. patent application Ser. No. 16/845,059 filed on Apr. 9, 2020, which claims the benefit of U.S. Provisional Patent Application No. 62/832,864, filed on Apr. 11, 2019, the contents of all of which are incorporated by reference herein.
- This disclosure relates generally to teleoperated vehicles, and more specifically to methods of maintaining vehicle speed within desired limits.
- Despite the substantial progress made in the past decade, no fully autonomous vehicle technology currently exists. Moreover, it is not clear whether it makes practical sense to develop self-driving technology capable of correctly navigating all theoretically possible edge cases and rare hazardous situations, since currently available machine learning technologies heavily depend on the amount of available training data. In the absence of a large data set, reliability of machine learning technologies may be inadequate, while the development of a semantic artificial intelligence applicable to this task is a scientific problem that currently has no solution.
- To resolve the issue of handling edge cases that cannot be correctly navigated by currently available and prospective machine intelligence agents, teleoperation of vehicles as an auxiliary technology is employed. According to this approach, whenever the confidence level of the machine intelligence agent controlling the vehicle declines past a certain threshold, or onboard sensors report abnormalities or certain events that require manual handling, the autonomous vehicle generates a request to be taken over by a remote human operator who then navigates the vehicle to safety using video feeds and sensor measurements provided by the vehicle. As soon as the onboard autonomous circuit or the remote operator determine that human agent intervention is no longer required, the teleoperation channel is usually terminated and the local machine intelligence agent continues navigating the vehicle to its destination. Alternatively, instead of a human teleoperator, the take-over request may be fulfilled by a more capable machine intelligence agent hosted on a cloud server.
- However, there are certain complications arising from the fact that the intelligent transport system is controlled by a remote agent for some length of time. First of all, network disruption is always a risk even if the vehicle is equipped with multiple fallback communication systems. An abrupt termination of a remote control session or a number of other communication failure modes may lead to a situation where emergency braking is not sufficient to avoid a collision with an obstacle. Additionally, it has been shown in academic literature that due to differences of in situ perception of a regular three dimensional environment from a driver's seat in a vehicle and perception of an environment projected on displays of a teleoperator workstation, multiple factors arise that can alter perception of egocentric speed and distance to obstacles. This may in turn lead to critical errors of judgment of the teleoperator, causing the teleoperated vehicle to collide with an obstacle due to a miscalculation on the part of the remote operator.
- A vehicle safety system imposes speed restrictions during a teleoperation session that enables safe teleoperation of the vehicle. The vehicle establishes a connection with a remote support server for enabling a teleoperation session in which the vehicle is controlled by a remote teleoperator. The vehicle receives, during the teleoperation system, control signals from the remote support server for controlling motion of the vehicle. A safety system obtains a depth map indicative of respective distances to objects in an environment of the vehicle. The safety system determines, based on the depth map, obstacles in the environment of the vehicle. The safety system receives sensor data from a sensor array indicative of a vehicle motion state and determines, based on the vehicle motion state and the one or more obstacles, a speed limit for limiting speed of the vehicle during the teleoperation session that enables safe teleoperation of the vehicle. The safety system determines if the vehicle is operating above the speed limit, and initiates a remedial action responsive to the vehicle operating above the speed limit. The remedial action may include sending a notification to the remote support server or automatically imposing speed constraints on the vehicle.
-
FIG. 1 is a block diagram of teleoperated vehicle environment. -
FIG. 2 is a flowchart illustrating a method for building and maintaining a depth map. -
FIG. 3 is a flowchart illustrating a method for discriminating between mobile and static obstacles. -
FIG. 4 is a flowchart illustrating a method for evaluating vehicle speed safety. -
FIG. 5 is a flowchart illustrating a method for constructing a map of safe vehicle speeds. - In a vehicle teleoperation session, a speed limit is determined for which the vehicle can be safely teleoperated. A safety system senses data relating to the vehicle environment and generates a depth map from which obstacles in the vicinity of the vehicle can be identified. Based on the detected obstacles and a motion state of the vehicle, a speed limit is determined at which the vehicle is predicted to be able to implement emergency braking command (e.g., directly by the vehicle or by the teleoperator) to avoid a collision. The speed limit may be automatically applied to the vehicle or may be provided to the teleoperator to enable the teleoperator to adjust the vehicle speed. Thus, the vehicle system may maintain restrictions on vehicle kinematics to give the emergency braking subsystem sufficient time to react to prevent a collision in response to the teleoperator initiating an emergency braking command or automatically in response to a disruption in control or communications with the teleoperator.
-
FIG. 1 illustrates an example embodiment of anenvironment 100 for teleoperating a vehicle. A remote support server 101 communicates with one or more connectedvehicles 102 to provide teleoperation or other support services. In an embodiment, upon establishing a remote support session with avehicle 102, a video stream capturing a vehicle environment is provided to the remote support server 101. The video stream may optionally be presented at ateleoperation workstation 103 for a human teleoperator. Furthermore, other sensor data or video streams may be provided from thevehicle 102 or external sensor arrays 104 to the remote support server 101. In this embodiment, a human teleoperator at theteleoperation workstation 103 may view the video stream on a display and provide control inputs via a control input device. The remote support server 101 may present the video in real-time and provide real-time teleoperation control signals to control thevehicle 102 to enable the teleoperator to remotely drive the vehicle. - In another embodiment, the remote support server 101 may comprise an artificial intelligence agent that does not necessarily require a
teleoperator workstation 103 with a display or physical controls for providing human input. Here, the remote support server 101 may provide control instructions to the vehicle directly based on the processing of a real-time video feed and other sensor data without necessarily utilizing any human input. In alternative embodiments, the remote support server 101 may comprise a semi-robotic agent that interacts with ateleoperator workstation 103 in a similar manner as a human teleoperator. - The
teleoperator workstation 103, if present, may be coupled to the remote support server 101 via a local area network connection, a direct wired connection, or via a remote connection. Ateleoperator workstation 103 may include a display to enable a human teleoperator to view real-time video of the vehicle environment and controls for enabling a human teleoperator to control the vehicle. In an embodiment, the video may include at least a front view that mimics or approximates the view seen by a driver within the vehicle. Optionally, the video may include additional views, such as a rear view video, side view videos, or other views that may mimic the views seen by a driver in mirrors of a traditional vehicle or may include other views not necessarily available to a driver of a traditional vehicle. The controls may include controls that mimic those available within a traditional vehicle such as a steering wheel, acceleration pedal, and brake pedal. Alternatively, different forms of controls may be available at theteleoperator workstation 103 such as a joystick, mouse, touch screen, voice control system, gesture control system, or other input mechanism to control one or more aspects of the vehicle. - In other embodiments, where the remote support server 101 operates entirely as an artificial intelligence agent without human intervention, the
teleoperator workstation 103 may be omitted. - The connected
vehicle 102 comprises avehicle safety system 105, a plurality of sensor arrays 106 (such as LIDAR, radar, sonar, or stereo cameras), and a drive-by-wire system 107. Other components of thevehicle 102 such as a communication system for wireless communicating with the remote support server 101 or external sensor arrays 104 and an on-board computer for linking and controlling various components of the connectedvehicle 102 are omitted fromFIG. 1 for conciseness. In an embodiment, implementation of one or more components ofFIG. 1 may include a processor and a non-transitory computer-readable storage medium that stores instructions for carrying out the functions attributed to the various components described herein. - The drive-by-
wire system 107 receives control signals from the remote support server 101 and controls operation of thevehicle 102 in response to the control signals to enable teleoperation of thevehicle 102. For example, the drive-by-wire system 107 may receive steering control signals, braking control signals, acceleration control signals, or other vehicle control signals to control operation of thevehicle 102 when being teleoperated. The drive-by-wire system 107 may furthermore provide sensor data to the remote support server 101 to enable the remote support server 101 to generate the control signals in response to the sensed information (either based on human teleoperator controls or from an artificial intelligence agent). In an embodiment, the drive-by-wire system 107 may furthermore control certain actions directly based on received sensor data and/or data from thesafety system 105, without necessarily requiring control signals from the remote support server 101. - The
onboard sensor array 106 includes one or more sets of sensors for capturing information about the vehicle operation and/or the vehicle environment. The sensors may include, for example, one or more cameras for capturing images or video, an inertial measurement unit (IMU) for capturing motion data of the vehicle, a global navigation satellite system (GNSS) for capturing position data relating to the vehicle, a computational unit such as Doppler analysis unit or tire modelling unit to determine operating characteristics of the vehicle, or other sensing systems. Theonboard sensor array 106 may furthermore comprise a network interface for receiving data from external data sources (e.g., Internet-based sources) that provide data such as current air humidity, estimates of moisture condensed on the road surface, or other information. In an embodiment, theonboard sensor array 106 furthermore includes a time-of-flight ranging device such as pulse or continuous-wave LIDAR, radar, or sonar to perform depth estimation. - The
safety system 105 generates a set of safety parameters based on sensed information that may be utilized by the drive-by-wire system 107 directly or may be sent to the remote support server 101 to assist in teleoperation of thevehicle 102. In an embodiment, thesafety system 105 periodically requests a data set from one or more data sources that includes information relevant to safely navigating thevehicle 102. Responsive to receiving a data set from a source, thesystem 105 determines kinematic parameters such as vehicle steering angle, speed, and trajectory in substantially real time, and stores these or other parameters in transient or permanent storage onboard theconnected vehicle 102. In other instances, the system may request data sets episodically or under certain circumstances, or utilize other mechanisms for data consumption such as publish-subscribe or push-pull. - The
safety system 105 may include a kinematic computational unit (KCU) 108, a vehicle speed controlprogrammatic interface 109, a safetycomputational unit 110, and adepth estimation system 111. - The
depth estimation system 111 provides depth estimates for objects present in the video feed captured by the onboard sensor array 106 (or external sensor array 104) and generates a depth map representing respective distances to detected objects. For example, the depth map may comprise an array of pixels or blocks of pixels associated with an image in which a depth of each pixel or block of pixels represents a distance to the nearest object depicted at that location. In alternative embodiments, the depth map may comprise a three-dimensional array indicating whether or not an object is detected at each three-dimensional location within a volume. In an embodiment, the depth map may represent a volume in a vicinity of thevehicle 102. In other embodiments, the depth map may be limited to locations that intersect a predicted trajectory of thevehicle 102 over a certain time range. Discrete regions of the depth map that intersect the predicted trajectory of thevehicle 102 represent obstacles. Depending on the environment and the trajectory of avehicle 102, such obstacles may include other vehicles, pedestrians, light poles or building facades. - The
depth estimation system 111 may comprise, for example, a structured light depth camera suitable for outdoors environment mapping based on any appropriate coding scheme, such as sinusoidal, binary or de Brujin coding. In one embodiment, the structured light camera uses a narrow-band light source and a narrow-band filter to remain functional in sunlight. In another embodiment, the structured light camera uses a polarized light source and a polarization filter to remain functional in sunlight. In another embodiment, the structured light camera uses light concentration techniques to sequentially illuminate and scan the complete angular range while reducing the time required to achieve the necessary signal-to-noise ratio at each step. - Alternatively, or in addition, the
depth estimation system 111 may comprise an artificial neural depth estimator that includes a monocular camera and an appropriately trained machine learning (ML) system executed on an onboard computer such as a convolutional neural network (CNN). The ML system performs depth estimation for each frame captured by the camera. In a further embodiment, the ML system may additionally be trained to use the motion parallax effect to recover a depth map. - In an embodiment, the
depth estimation system 111 receives stereo images from a stereo vision camera and estimates a depth map based on stereo disparity using state of the art computer vision (CV) methods. In an embodiment, thedepth estimation system 111 does not necessarily include a dedicated camera and may estimate the depth map from images and/or video received from cameras of theonboard sensor array 106 and/or external array 104. - The kinematic computational unit (KCU) 108 generates predictions about a time until the
vehicle 102 will collide with an object absent some remedial action, based on the current vehicle state, object states, and sensed environmental conditions. For example, in an embodiment, theKCU 108 receives various kinematic parameters (e.g., position, speed, acceleration, vehicle trajectory, etc.) of the vehicle, a depth map representing distances to detected objects in a vicinity of thevehicle 102, and time information from a system clock running with a certain degree of granularity. Based on the received information, theKCU 108 updates an estimate of the time until a collision of thevehicle 102 following the current trajectory with an obstacle, provided that a set of assumptions on the behavior of the environment is valid. These estimates may be made available to other components of thevehicle safety system 105 or may be made available toother vehicles 102 in the vicinity. For example, the time to collision may be provided via a dedicated programmatic interface or transmitted over a wireless network for consumption by remote operation software. - The vehicle speed control programmatic interface (VSCPI) 109 generates control signals for manipulating the speed of the
vehicle 102 in response to various safety considerations. TheVSCPI 109 may directly control speed (e.g., through interaction with the drive-by-wire system 107) or via may control speed indirectly via a related variable such as engine power or torque. TheVSCPI 109 may be accessed by other components of thevehicle 102 via a dedicated programmatic interface to enable the other components to provide data for controlling speed of thevehicle 102. Additionally, theVSCPI 109 may be accessible by the remote support server 101 via a wireless network. - The safety computational unit (SCU) 110 restricts the speed of the
vehicle 102 using theVSCPI 109 in such a manner that the estimated time to collision computed by theKCU 108 exceeds the sum of the estimated remote operator reaction time and the braking time as computed using an appropriate method. TheSCU 110 may be permanently active, or activate responsive to certain conditions being met, or activate responsive to signals emitted byother vehicle 102 components. - In an embodiment, the
SCU 110 maintains a record of the velocity V and the steering angle ϕ of thevehicle 102 using data from theonboard sensor array 106, commands from the drive-by-wire system 107, or other information sources. In an embodiment, thesystem 105 additionally maintains a record of the expectedvehicle 102 acceleration and steering angle derivative using methods such as differentiation of the velocity V and the steering angle ϕ time profiles, analysis of the profile of drive-by-wire commands received by thevehicle 102, or other methods. - The
SCU 110 may be designed to limit the speed in a manner that enables the vehicle to be decelerated in time to avoid a collision in response to an emergency braking signal from a teleoperator. In another embodiment, theSCU 110 may be configured to automatically decelerate the vehicle as an automatic precaution in the event that the vehicle loses a connection to the remote support server during a teleoperation session. Here, the speed limit may be set in a manner that enables theSCU 110 to bring the vehicle to a stop and avoid a collision in response to detecting the loss of the connection. - In alternative embodiments, the
safety system 105 may be implemented wholly or partly on a computing system external to thevehicle 102. For example, thesafety system 105 or components thereof may be implemented on the remote support server 101, on a fleet management that manages a fleet of vehicles, or on another network device. -
FIG. 2 is a flowchart illustrating an embodiment of a method for building and maintaining adepth map 200. Thesafety system 105 requests 201 a data set of sensor measurements. Thesafety system 105 may request the data from theonboard sensor arrays 106, the external sensors 104,other vehicles 102, or a combination thereof. The sensor data may be requested on a periodic basis or in response to some event. Alternatively, sensor data may be pushed to thesafety system 105 without an explicit request. Thesafety system 105 receives 202 a sensor data set responsive to the request. Responsive to receiving 202 the data set, thesafety system 105 computes and updates 203 adepth map 200 representing distances to objects detected in the environment of thevehicle 102. Thedepth map 200 may comprise, for each of a plurality of detected points in a vicinity of thevehicle 102, a distance parameter representing a distance between thevehicle 102 and the object, and an angle parameter representing an angle between thevehicle 102 and the object. In an embodiment, thedepth map 200 may be generated over sufficient angular dimensions dependent on the specific environmental factors. Furthermore, in an embodiment, thedepth map 200 may be generated or updated in substantially real time (e.g., using a request-reply mechanism). In other instances, thesafety system 105 may request data sets episodically or under certain circumstances, or utilize other mechanisms for data consumption such as publish-subscribe or push-pull. -
FIG. 3 is a flowchart illustrating an example embodiment of a process for classifying obstacles detected in thedepth map 200 as either mobile obstacles or static obstacles. In this embodiment, thesafety system 105 classifies obstacles as mobile obstacles (e.g., likely to correspond to other vehicles) if they lie within a mobile obstacle zone representing an area within a threshold lateral distance from thevehicle 102. Otherwise, thesafety system 105 classifies obstacles as static obstacles if they lie outside of the mobile obstacle zone (beyond the threshold lateral distance from the vehicle 102). - In the example process, the
safety system 105 computes 301 edges of the mobile obstacle zone based on the information on the lane occupied by the vehicle 102 (which may be obtained based on prestored map data, real-time sensor data, or a combination thereof), the position of thevehicle 102 within that lane, and the layout of the road network in the immediate vicinity of thevehicle 102. Thesafety system 105 identifies 302 discrete obstacles as clusters of adjacent points P={i∈{1,2, . . . , n}|pi} in thedepth map 200. Clusters may be identified using any suitable clustering algorithm. For each of the points in a cluster corresponding to an obstacle, thesafety system 105transforms 303 the depth map data (which includes, for each point, an angle and distance from thevehicle 102 to the object) to a set of spatial coordinates (e.g., an x-y-z location). Here, the X axis is directed parallel to the symmetry axis of the vehicle 102 (for example, coinciding with the direction of the lane), the Y axis is directed normally clockwise from the positive direction of the X axis, and the Z axis completes the left-handed set. In an example, the z=0 level may be chosen in a manner so that the ranging device onboard thevehicle 102 lies in the XY plane. For each point in thedepth map 200, thesafety system 105 determines the (x,y) coordinate pairs by multiplying the measured depth associated with each point by the sine and cosine of the angle α between the X axis and the projection of the direction to the point on the XY plane, and determines a z coordinate by multiplying the cosine of the angle between the X axis and the projection of the direction to the point on the XZ plane. - The
system 105 then collapses 304 the points P of each obstacle 300 into a linear form corresponding to the surface closest to the vehicle 102: -
- For example, a leading vehicle in front of the
vehicle 102 may be represented as a line approximating the surface location of its rear bumper, while a trailing vehicle behind thevehicle 102 may be represented as a line approximating the surface location of its front bumper. Thesafety system 105 then determines 305 the locations of the lateral edges of each obstacle. For example, thesafety system 105 determines the values of the outermost coordinates of the linear form: -
- for each obstacle 300. The
safety system 105 determines 306 whether the outer edges overlap the mobile obstacle zone. For example, thesafety system 105 determines whether the ymin is higher than the current left edge of the mobile obstacle zone and whether the ymax is lower than the current right edge of the mobile obstacle zone. Thesafety system 105 then classifies 307 each obstacle as either mobile or static dependent on the overlap with the mobile obstacle zone. For example, if either of the above described overlap conditions are met, thesafety system 105 classifies the obstacle as a mobile obstacle. Otherwise, thesafety system 105 classifies the obstacle as a static obstacle.Steps - In an embodiment, the
system 105 may perform additional analysis for obstacles in the static zone (outside the mobile obstacle zone) to predict a likelihood of them entering the mobile obstacle zone within a relevant time interval. - In an embodiment, the
safety system 105 may determine static obstacle zone geometry using data acquired from external data sources such as aerial or satellite imagery, cadaster records, or interfaces provided by public agencies or corporate entities. - In an embodiment, the
system safety 105 may use the time profile of the position of a mobile obstacle mi relative to thevehicle 102 to determine its velocity vi relative to the ground, heading angle θ and steering angle C. Thesafety system 105 may further use the information on dynamics of mobile obstacles during simulations to compute the position and heading of a mobile obstacle {xi (t),yi(t),θi(t)}, which in conjunction with the known geometric size of a mobile obstacle may be used by thesafety system 105 to determine bounding boxes around each mobile obstacle. The bounding boxes may be utilized to more accurately track locations of the mobile obstacles and predict collisions. -
FIG. 4 illustrates an example embodiment of a method for evaluating vehicle speed of teleoperated vehicles to determine speed limits for operating thevehicles 102. Here, theKSU 108 may perform kinematic simulations of thevehicle 102 and its environment for multiple time points τj∈[0,T] along the predictedvehicle 102 trajectory in the following manner. TheKSU 108 may first obtain 401 sensor data. The sensor data may include information from alocal sensor array 106 comprising sensors such as IMU, GPS, Doppler effect sensors or other devices as described above. Additionally, sensor data may be obtained from an external sensor array 104. In an embodiment, theKSU 108 may also additionally obtain 402 auxiliary information such as pending but non-executed remote operator commands, transmissions from neighboringvehicles 102 over vehicular ad-hoc networks or over wireless infrastructure that may influence the solution but is not reflected in currently observable kinematics. TheKSU 108 then predicts 403 motion of thevehicle 102 and the detected mobile obstacles. For example, theKSU 108 may analytically or numerically solve a motion equation for thevehicle 102 and the set of known mobile obstacles 300 M={k∈{1,2, . . . , K}|mi}, that predicts the values for vehicle and mobile obstacle positions and orientations for all the desired time points τj. TheKSU 108 may adjust the interval between time points x responsive to changes in parameters describing thevehicle 102 such as onboard computer performance. TheKSU 108 determines 404 a speed limit based on the motion data. Here, theKSU 108 may detect the maximum speed at which thevehicle 102 can travel at a given time during a teleoperation session that will enable an emergency braking command to stop the vehicle in sufficient time to avoid a collision under the present conditions. In an embodiment, the speed limit may be based on a time to bring the vehicle to a stop in response to an emergency braking signal from a teleoperator. In another embodiment, the speed limit may be based on a time to bring the vehicle to a step as an automatic precaution in the event that the vehicle loses a connection to the remote support server during a teleoperation session. - The speed limit may vary dependent on the conditions and may be dynamically updated over time during the teleoperation session. For example, the speed limit may decrease on more congested roadways or on roadways with a greater number of obstacles. Additionally, the speed limit may adjust dependent on whether the vehicle is being teleoperated by a human or computer agent, or may adjust dependent on varying profiles of different human operators and their respective reaction times. Further still, the speed limit may adjust dependent on current network conditions to account for varying latency that may be introduced in the time it takes for video or other sensor data to reach the teleoperator and/or the time it takes for teleoperation commands to be received an executed by the
vehicle 102. - In an embodiment, the
KSU 108 determines the speed limit by performing simulations of thevehicle 102 trajectory and speed under the assumption that a malfunction occurs at a given time point τ. TheKSU 108 may then determine whether at the speed thevehicle 102 is estimated to travel at that time τ, a collision of thevehicle 102 with an obstacle may occur before thevehicle 102 comes to a complete halt if an emergency slowdown command is initiated at that time. In an embodiment, thesystem 105 represents the time ΔT allocated for emergency slowdown based at least in part on a time period Δtb representing the practical deceleration capacity of thevehicle 102, which in turn may depend on ABS availability, humidity of the road surface, tire model and other physical factors. Additionally, thesafety system 105 may furthermore determine the time ΔT based at least in part on a predefined cooldown period Δtc. Here, the cooldown period Δtc may represent a preconfigured delay value that may be set by teleoperation administrator or adjusted automatically. Adjusting the cooldown period Δtc enables calibration of the false positive rate. For example, setting a higher cooldown period Δtc increases the predicted time to a collision and thus may decrease the likelihood of determining that a slowdown or emergency action is needed. Setting a lower cooldown period Δtc decreases the predicted time to a collision and thus may increase the likelihood of determining that a slowdown or emergency action is needed. The cooldown period Δtc may be helpful to calibrate the time ΔT allocated for emergency slowdown based on a predicted likelihood that a critical ingress communication channel such as the command and control link with the remote teleoperator or machine intelligence agent becomes severed or malfunctions. - In another embodiment, the time ΔT is based at least in part on a remote operator reaction time Δtr. The reaction time may be determined based on, for example, the camera exposure time, video frame onboard processing time, video frame transmission time
-
- (where τrtt is the estimate of the network round trip time between the
vehicle 102 and theteleoperation workstation 103, S is the frame size in bytes and BW is the estimated network throughput in megabits per second), the video frame remote processing time (which may in turn include a safety margin of -
- where F is the
teleoperation workstation 103 monitor refresh rate in Hz if the remote operator is an agent consuming the video feed via a visual display or a plurality thereof), estimated remote operator reaction time Δtreact, and the control message transmission time Δtn2 determined similarly to Δtn1. For example, thesafety system 105 may use the remote operator reaction time Δtreact in scenarios where a malfunction of a critical egress communication channel such as the video feed link or relevant hardware implements is detected. In an embodiment, theKSU 108 determines the maximum speed at which a collision can be avoided if emergency braking is initiated at the time point max(τ+Δtreact,τ+Δtc). Here, τ+Δtreact represents the earliest time that the remote server 101 can initiate an emergency brake, while τ+Δtc represents a configurable waiting time based on the cooldown period Δtc. Thus, if thesafety system 105 determines to initiate an emergency braking but the command does not arrive within the time window τ+Δtc, theKSU 108 can determine that the network communication or remote teleoperator is malfunctioning. - The
KSU 108 may then transmits 405 a message indicative of the detected condition. Here, the message may specify the determined speed limit itself and/or may comprise an indicator indicating whether or not thevehicle 102 is exceeding the speed limit. For example, theKSU 108 may publish a message to the controlling agent (e.g., the remote support server 101 or SCU 110) indicating that the selected speed profile may be hazardous or make this information available to other components of thesafety system 105. - In an embodiment, the remote support server 101 may provide an indicator on the
teleoperation workstation 103 that alerts a human teleoperator to the speed limit (or an indicator indicating when theteleoperated vehicle 102 is exceeding the speed limit) to enable the human teleoperator to optionally decrease the vehicle speed. For example, the speed indicator may be displayed as an overlaid element on the video or may be provided via an audible alert. In another embodiment, theKSU 108 may transmit the information describing the vehicle's 102 estimated future trajectory and estimated safe stopping distance to the human agent at theteleoperator workstation 103. For example, theteleoperator workstation 103 may use this information for the purpose of augmenting the visual representation displayed to the operator. For example, the estimatedvehicle 102 trajectory may include the predicted arc or path the vehicle is estimated to traverse in the near future, including some uncertainty or estimated error, and the estimated safe braking distance required to altogether avoid or minimize the impact of a collision. The estimated trajectory information may be based on the current input to the vehicle control system by either the devices capturing the operator's input or the onboard autonomous circuit. Theteleoperator workstation 103 may present the estimated trajectory and braking distance information to the operator as an augmented reality (AR) overlay on top of a monoscopic or stereoscopic video presentation layer. - In an embodiment, the
vehicle 102 that is exceeding the determined speed limit may automatically adjust its speed to speed limit. Alternatively, the remote support server 101 may automatically send a control signal to adjust the vehicle'sspeed 102. In another embodiment, theSCU 110 may automatically adjust the vehicle speed only in the case that the remote operator has been observed to disregard warning messages. - In an embodiment, the
safety system 105 performs the kinematic simulations under the assumption that the onboard computer only manipulatesvehicle 102 speed v in an emergency slowdown mode and that the steering angle ϕ is static, enabling the reduction of the time necessary for computing the solution and keeping the trajectory of a malfunctioningvehicle 102 predictable for agents controllingother vehicles 102. - In an embodiment, the
KSU 108 may compute and publish information describing a set of possible trajectories based on the vehicle's 102 current telemetry information (such as its velocity v or steering angle ϕ). For example, the elements in the set of possible trajectories may each be classified as paths which would completely avoid a collision with any obstacle 300, paths which would minimize the impact of a collision with an obstacle 300, and paths which would result in a maximum-impact collision. Theteleoperator workstation 103 may display each path to the operator in a color-coded or otherwise differentiated representation in order to provide the operator with sufficient information to make a decision regarding the trajectory to pursue. Theteleoperator workstation 103 may present the calculated path information to the operator as an augmented reality (AR) overlay on top of a monoscopic or stereoscopic video presentation layer. - In an embodiment, the
teleoperator workstation 103 may provide the remote with a user interface component or application programming interface endpoint to override the decision of theSCU 111 regardingvehicle 102 speed limit adjustment. For example, such an embodiment may be used in case of sensor malfunction or data set processing errors. -
FIG. 5 is an example embodiment of a process for precomputing speed limits for a teleoperated vehicle at a remote support server 101. This process enables thesafety system 105 to request a pre-computed maximum vehicle speed limit data set from the remote support server 101 and use it to reduce the computational requirements of theKSU 108. An example procedure of constructing such a data set may be employed as follows. The remote support server 101 loads 501 a regional map of static obstacles such as buildings, road edges, light pole positions or other types of objects to be avoided. The remote support server 101 then processes 502 the map to identify the surfaces of the road network in that region and build a polygon mesh, with each polygon corresponding to a discrete segment of the road network. The remote support server 101 then computes 503 an array of maximum allowed vehicle speed values using analytical calculations or numerical simulations for the entire range of possible steering angle values for each such road segment. The speed limit for a given road segment and steering angle may represent the maximum speed at which a vehicle traversing that road segment and being operating at the given steering angle can be decelerated by a teleoperator in sufficient time to avoid colliding with a static obstacle. At the next stage, the remote support server 101 determines 504 a mapping of relations between individual road segments and steering angles. In other words, it establishes that applying a steering angle ϕ0 while traversing the road segment sn thevehicle 102 will travel to the road segment sm=S(n,ϕ0). The remote support server 101 may then adjust the determined speed limits based on the mapping between road segments and steering angles. For example, the remote support server 101 may determine 505 for each (sn,sm) tuple representing adjacent road segments, whether the difference between the maximum allowedvehicle 102 speed values for these segments vn−vm exceeds a threshold difference (based on, for example, the braking capabilities of the vehicle or other factors) as previously determined from the path and time available for the transition between the segments. If the difference between speed limits exceeds the threshold difference, the remote support server 101 may further adjusts vn to ensure a smooth transition between subsequent segments. - In an embodiment, the remote support server 101 may provide the
safety system 105 with a data set that includes speed limit information computed accounting for mobile obstacles moving along some trajectories and at some speeds relative to thevehicle 102. Such a data set may be comprehensive or restricted to a range of scenarios previously requested by thesafety system 105, or restricted to a range of scenarios determined as probable to be encountered during the current trip by the remote server 101. - In an embodiment, the remote support server 101 may provide the
safety system 105 with a data set including manually or automatically pre-labeled geoinformation on hazardous areas. Such a data set may be solely defined by geographic coordinates of thevehicle 102 or any landmarks involved, geographic coordinates in conjunction with temporal intervals, or custom semantic rulesets. - In another embodiment, the remote support server 101 may provide the
safety system 105 with a data set based in part on information obtained by analyzing speed limits observed by in situ human drivers or remote vehicle operators. This may allow to account for factors that are not reliably identified by machine learning algorithms but are identified by human agents with a sufficient degree of accuracy. - In another embodiment, the remote support server 101 may provide the
safety system 105 with a data set including information on weather conditions and derivative parameters. For example, the data set may include information on expected air humidity and temperature levels, and based on these characteristics the onboard computer may compute the probability and severity of fog condensation, which may in turn be used in maximum permitted speed calculations. - In another embodiment, the remote support server 101 may provide the
safety system 105 with a data set accounting for specific vehicle model, road surface humidity level, absence or presence of an ABS system, total load of the vehicle or other parameters. - In another embodiment, the remote support server 101 may provide the
safety system 105 with an updated data set episodically, periodically, according to a schedule or in substantial real time. Such updates may each comprise a full dataset package, a cumulative incremental package or a differential incremental package. - In another embodiment, the
safety system 105 computes paths on the road network that are optimal according to a specific cost function as well as satisfy the safety conditions. - Reference in the specification to “one embodiment” or to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment. The appearances of the phrase “in one embodiment” or “an embodiment” in various places in the specification are not necessarily all referring to the same embodiment.
- Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps (instructions) leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical, magnetic or optical signals capable of being stored, transferred, combined, compared and otherwise manipulated. It is convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. Furthermore, it is also convenient at times, to refer to certain arrangements of steps requiring physical manipulations or transformation of physical quantities or representations of physical quantities as modules or code devices, without loss of generality.
- However, all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or “determining” or the like, refer to the action and processes of a computer system, or similar electronic computing device (such as a specific computing machine), that manipulates and transforms data represented as physical (electronic) quantities within the computer system memories or registers or other such information storage, transmission or display devices.
- Certain aspects of the embodiments include process steps and instructions described herein in the form of an algorithm. It should be noted that the process steps and instructions of the embodiments can be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by a variety of operating systems. The embodiments can also be in a computer program product which can be executed on a computing system.
- The embodiments also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the purposes, e.g., a specific computer, or it may comprise a computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Memory can include any of the above and/or other devices that can store information/data/programs and can be transient or non-transient medium, where a non-transient or non-transitory medium can include memory/storage that stores information for more than a minimal duration. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
- The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the method steps. The structure for a variety of these systems will appear from the description herein. In addition, the embodiments are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the embodiments as described herein, and any references herein to specific languages are provided for disclosure of enablement and best mode.
- Throughout this specification, some embodiments have used the expression “coupled” along with its derivatives. The term “coupled” as used herein is not necessarily limited to two or more elements being in direct physical or electrical contact. Rather, the term “coupled” may also encompass two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other, or are structured to provide a thermal conduction path between the elements.
- Likewise, as used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
- In addition, use of the “a” or “an” are employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of embodiments. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise. The use of the term and/or is intended to mean any of: “both”, “and”, or “or.”
- In addition, the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the embodiments.
- While particular embodiments and applications have been illustrated and described herein, it is to be understood that the embodiments are not limited to the precise construction and components disclosed herein and that various modifications, changes, and variations may be made in the arrangement, operation, and details of the methods and apparatuses of the embodiments without departing from the scope of the embodiments.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US18/115,433 US20230202467A1 (en) | 2019-04-11 | 2023-02-28 | Automatic Imposition of Vehicle Speed Restrictions Depending on Road Situation Analysis |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201962832864P | 2019-04-11 | 2019-04-11 | |
US16/845,059 US11618439B2 (en) | 2019-04-11 | 2020-04-09 | Automatic imposition of vehicle speed restrictions depending on road situation analysis |
US18/115,433 US20230202467A1 (en) | 2019-04-11 | 2023-02-28 | Automatic Imposition of Vehicle Speed Restrictions Depending on Road Situation Analysis |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/845,059 Continuation US11618439B2 (en) | 2019-04-11 | 2020-04-09 | Automatic imposition of vehicle speed restrictions depending on road situation analysis |
Publications (1)
Publication Number | Publication Date |
---|---|
US20230202467A1 true US20230202467A1 (en) | 2023-06-29 |
Family
ID=72749478
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/845,059 Active 2040-08-19 US11618439B2 (en) | 2019-04-11 | 2020-04-09 | Automatic imposition of vehicle speed restrictions depending on road situation analysis |
US18/115,433 Pending US20230202467A1 (en) | 2019-04-11 | 2023-02-28 | Automatic Imposition of Vehicle Speed Restrictions Depending on Road Situation Analysis |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/845,059 Active 2040-08-19 US11618439B2 (en) | 2019-04-11 | 2020-04-09 | Automatic imposition of vehicle speed restrictions depending on road situation analysis |
Country Status (1)
Country | Link |
---|---|
US (2) | US11618439B2 (en) |
Families Citing this family (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10209718B2 (en) | 2017-03-14 | 2019-02-19 | Starsky Robotics, Inc. | Vehicle sensor system and method of use |
US11295615B2 (en) | 2018-10-29 | 2022-04-05 | Here Global B.V. | Slowdown events |
US11100794B2 (en) * | 2019-04-15 | 2021-08-24 | Here Global B.V. | Autonomous driving and slowdown patterns |
US11475772B2 (en) * | 2019-05-01 | 2022-10-18 | Ottopia Technologies Ltd. | System and method for remote operator assisted driving through collision warning |
US11448744B2 (en) * | 2019-12-31 | 2022-09-20 | Woven Planet North America, Inc. | Sequential doppler focusing |
US20210295171A1 (en) * | 2020-03-19 | 2021-09-23 | Nvidia Corporation | Future trajectory predictions in multi-actor environments for autonomous machine applications |
US11892838B1 (en) * | 2020-07-27 | 2024-02-06 | Nuro, Inc. | Framework for validating autonomy and teleoperations systems |
EP3961255A1 (en) * | 2020-08-28 | 2022-03-02 | Aptiv Technologies Limited | Driver assistance system for a vehicle, vehicle and a driver assistance method implementable by the system |
JP2022150903A (en) * | 2021-03-26 | 2022-10-07 | トヨタ自動車株式会社 | Driving support method, driving support device, driving support system, and computer program |
EP4083735A1 (en) * | 2021-04-26 | 2022-11-02 | Robert Bosch GmbH | Method and device for remote driving |
US11885102B2 (en) * | 2021-12-20 | 2024-01-30 | Caterpillar Inc. | Visual overlays for indicating stopping distances of remote controlled machines |
DE102022107187A1 (en) * | 2022-03-28 | 2023-09-28 | Valeo Schalter Und Sensoren Gmbh | Driving a vehicle in a specified drivable area |
KR102639020B1 (en) * | 2023-09-14 | 2024-02-21 | 에이티씨앤에스 주식회사 | A system for controlling vehicles and aircraft in the airside of an airport |
Citations (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5410304A (en) * | 1992-05-05 | 1995-04-25 | Daimler-Benz Ag | Method and apparatus for displaying the impending danger due to speed associated with the driving situation of a vehicle |
US20070213883A1 (en) * | 2006-03-07 | 2007-09-13 | Tdk Corporation | Vehicle over speed indicator |
JP4221840B2 (en) * | 1999-09-27 | 2009-02-12 | 株式会社エクォス・リサーチ | Vehicle control device |
US9008890B1 (en) * | 2013-03-15 | 2015-04-14 | Google Inc. | Augmented trajectories for autonomous vehicles |
US20160027300A1 (en) * | 2014-07-28 | 2016-01-28 | Econolite Group, Inc. | Self-configuring traffic signal controller |
US20160133131A1 (en) * | 2014-11-12 | 2016-05-12 | GM Global Technology Operations LLC | Use of participative sensing systems to enable enhanced road friction estimation |
US9517767B1 (en) * | 2015-11-04 | 2016-12-13 | Zoox, Inc. | Internal safety systems for robotic vehicles |
US20170341660A1 (en) * | 2016-05-25 | 2017-11-30 | Here Global B.V. | Determining speed information |
JP2018016189A (en) * | 2016-07-27 | 2018-02-01 | アイシン精機株式会社 | Speed control apparatus |
US20180096605A1 (en) * | 2013-08-02 | 2018-04-05 | Honda Motor Co., Ltd. | Vehicle pedestrian safety system and methods of use and manufacture thereof |
US20180286242A1 (en) * | 2017-03-31 | 2018-10-04 | Ford Global Technologies, Llc | Steering wheel actuation |
WO2019023589A1 (en) * | 2017-07-28 | 2019-01-31 | Nuro, Inc. | Systems and methods for augmented capabilities for remote operation of robot vehicles |
US20190143974A1 (en) * | 2017-11-16 | 2019-05-16 | Volkswagen Aktiengesellschaft | Method for adapting a predetermined reference line for a transportation vehicle and device |
US20190308617A1 (en) * | 2018-04-10 | 2019-10-10 | Valeo Schalter Und Sensoren Gmbh | Tailgating situation handling by an automated driving vehicle |
US20190318620A1 (en) * | 2018-04-13 | 2019-10-17 | Toyota Jidosha Kabushiki Kaisha | Remote Vehicle Control at Intersections |
US20190354111A1 (en) * | 2018-05-16 | 2019-11-21 | Direct Current Capital LLC | Method for dynamically querying a remote operator for assistance |
US20200164870A1 (en) * | 2017-08-10 | 2020-05-28 | Hitachi Automotive Systems, Ltd. | Device and method for controlling vehicle movement, and device and method for generating target course |
US20200273197A1 (en) * | 2019-02-22 | 2020-08-27 | Toyota Jidosha Kabushiki Kaisha | Vehicle Localization Using Marker Devices |
US10831210B1 (en) * | 2018-09-28 | 2020-11-10 | Zoox, Inc. | Trajectory generation and optimization using closed-form numerical integration in route-relative coordinates |
US10977882B1 (en) * | 2018-10-17 | 2021-04-13 | Lytx, Inc. | Driver health profile |
US20210116907A1 (en) * | 2018-03-18 | 2021-04-22 | Driveu Tech Ltd. | Device, System, and Method of Autonomous Driving and Tele-Operated Vehicles |
US20210291900A1 (en) * | 2018-04-16 | 2021-09-23 | Volvo Truck Corporation | Method for estimating a maximum safe articulation angle to be used in reversing of a vehicle combination |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102421645B (en) * | 2009-05-07 | 2015-06-24 | 大陆-特韦斯贸易合伙股份公司及两合公司 | Method and apparatus for regulating or controlling the driving stability of a vehicle |
JP5143103B2 (en) * | 2009-09-30 | 2013-02-13 | 日立オートモティブシステムズ株式会社 | Vehicle motion control device |
WO2011075392A1 (en) * | 2009-12-18 | 2011-06-23 | Honda Motor Co., Ltd. | A predictive human-machine interface using eye gaze technology, blind spot indicators and driver experience |
US9424468B2 (en) * | 2010-09-08 | 2016-08-23 | Toyota Jidosha Kabushiki Kaisha | Moving object prediction device, hypothetical movable object prediction device, program, moving object prediction method and hypothetical movable object prediction method |
JP5729416B2 (en) * | 2013-04-26 | 2015-06-03 | 株式会社デンソー | Collision determination device and collision mitigation device |
JP6532786B2 (en) * | 2015-08-07 | 2019-06-19 | 株式会社日立製作所 | Vehicle travel control device and speed control method |
US9630619B1 (en) * | 2015-11-04 | 2017-04-25 | Zoox, Inc. | Robotic vehicle active safety systems and methods |
US10048080B2 (en) * | 2016-03-22 | 2018-08-14 | Toyota Motor Engineering & Manufacturing North America, Inc. | Autonomous vehicle virtual reality navigation system |
KR20230017365A (en) * | 2016-06-27 | 2023-02-03 | 모빌아이 비젼 테크놀로지스 엘티디. | controlling host vehicle based on detected spacing between stationary vehicles |
US10317899B2 (en) | 2017-06-16 | 2019-06-11 | nuTonomy Inc. | Intervention in operation of a vehicle having autonomous driving capabilities |
US10740988B2 (en) | 2017-06-16 | 2020-08-11 | nuTonomy Inc. | Intervention in operation of a vehicle having autonomous driving capabilities |
US10761542B1 (en) * | 2017-07-11 | 2020-09-01 | Waymo Llc | Methods and systems for keeping remote assistance operators alert |
-
2020
- 2020-04-09 US US16/845,059 patent/US11618439B2/en active Active
-
2023
- 2023-02-28 US US18/115,433 patent/US20230202467A1/en active Pending
Patent Citations (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5410304A (en) * | 1992-05-05 | 1995-04-25 | Daimler-Benz Ag | Method and apparatus for displaying the impending danger due to speed associated with the driving situation of a vehicle |
JP4221840B2 (en) * | 1999-09-27 | 2009-02-12 | 株式会社エクォス・リサーチ | Vehicle control device |
US20070213883A1 (en) * | 2006-03-07 | 2007-09-13 | Tdk Corporation | Vehicle over speed indicator |
US9008890B1 (en) * | 2013-03-15 | 2015-04-14 | Google Inc. | Augmented trajectories for autonomous vehicles |
US20180096605A1 (en) * | 2013-08-02 | 2018-04-05 | Honda Motor Co., Ltd. | Vehicle pedestrian safety system and methods of use and manufacture thereof |
US20160027300A1 (en) * | 2014-07-28 | 2016-01-28 | Econolite Group, Inc. | Self-configuring traffic signal controller |
US20160133131A1 (en) * | 2014-11-12 | 2016-05-12 | GM Global Technology Operations LLC | Use of participative sensing systems to enable enhanced road friction estimation |
US9517767B1 (en) * | 2015-11-04 | 2016-12-13 | Zoox, Inc. | Internal safety systems for robotic vehicles |
US20170341660A1 (en) * | 2016-05-25 | 2017-11-30 | Here Global B.V. | Determining speed information |
JP2018016189A (en) * | 2016-07-27 | 2018-02-01 | アイシン精機株式会社 | Speed control apparatus |
US20180286242A1 (en) * | 2017-03-31 | 2018-10-04 | Ford Global Technologies, Llc | Steering wheel actuation |
WO2019023589A1 (en) * | 2017-07-28 | 2019-01-31 | Nuro, Inc. | Systems and methods for augmented capabilities for remote operation of robot vehicles |
US20200164870A1 (en) * | 2017-08-10 | 2020-05-28 | Hitachi Automotive Systems, Ltd. | Device and method for controlling vehicle movement, and device and method for generating target course |
US20190143974A1 (en) * | 2017-11-16 | 2019-05-16 | Volkswagen Aktiengesellschaft | Method for adapting a predetermined reference line for a transportation vehicle and device |
US20210116907A1 (en) * | 2018-03-18 | 2021-04-22 | Driveu Tech Ltd. | Device, System, and Method of Autonomous Driving and Tele-Operated Vehicles |
US20190308617A1 (en) * | 2018-04-10 | 2019-10-10 | Valeo Schalter Und Sensoren Gmbh | Tailgating situation handling by an automated driving vehicle |
US20190318620A1 (en) * | 2018-04-13 | 2019-10-17 | Toyota Jidosha Kabushiki Kaisha | Remote Vehicle Control at Intersections |
US20210291900A1 (en) * | 2018-04-16 | 2021-09-23 | Volvo Truck Corporation | Method for estimating a maximum safe articulation angle to be used in reversing of a vehicle combination |
US20190354111A1 (en) * | 2018-05-16 | 2019-11-21 | Direct Current Capital LLC | Method for dynamically querying a remote operator for assistance |
US10831210B1 (en) * | 2018-09-28 | 2020-11-10 | Zoox, Inc. | Trajectory generation and optimization using closed-form numerical integration in route-relative coordinates |
US10977882B1 (en) * | 2018-10-17 | 2021-04-13 | Lytx, Inc. | Driver health profile |
US20200273197A1 (en) * | 2019-02-22 | 2020-08-27 | Toyota Jidosha Kabushiki Kaisha | Vehicle Localization Using Marker Devices |
Also Published As
Publication number | Publication date |
---|---|
US20200324761A1 (en) | 2020-10-15 |
US11618439B2 (en) | 2023-04-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11618439B2 (en) | Automatic imposition of vehicle speed restrictions depending on road situation analysis | |
US11390300B2 (en) | Method for using lateral motion to optimize trajectories for autonomous vehicles | |
US10832502B2 (en) | Calibration for autonomous vehicle operation | |
JP7381539B2 (en) | Remote control system and method for trajectory correction of autonomous vehicles | |
US11314249B2 (en) | Teleoperation system and method for trajectory modification of autonomous vehicles | |
US10627812B2 (en) | Risk based driver assistance for approaching intersections of limited visibility | |
US11022974B2 (en) | Sensor-based object-detection optimization for autonomous vehicles | |
US10599150B2 (en) | Autonomous vehicle: object-level fusion | |
US10377375B2 (en) | Autonomous vehicle: modular architecture | |
WO2020147311A1 (en) | Vehicle driving guarantee method and apparatus, device, and readable storage medium | |
US20180056998A1 (en) | System and Method for Multi-Vehicle Path Planning Technical Field | |
WO2019112799A1 (en) | Improving safety of autonomous vehicles using a virtual augmented support environment | |
US20120136510A1 (en) | Apparatus and method for detecting vehicles using laser scanner sensors | |
US11724708B2 (en) | Fail-safe handling system for autonomous driving vehicle | |
US20220284619A1 (en) | Offline optimization of sensor data for agent trajectories | |
US11613254B2 (en) | Method to monitor control system of autonomous driving vehicle with multiple levels of warning and fail operations | |
JP2019530609A (en) | Modular architecture autonomous vehicle | |
US20220017112A1 (en) | Systems and Methods for Generating Vehicle Corridors to Improve Path Planning Efficiency | |
US20220289198A1 (en) | Automated emergency braking system | |
WO2021153176A1 (en) | Autonomous movement device, autonomous movement control method, and program | |
CN115223131A (en) | Adaptive cruise following target vehicle detection method and device and automobile | |
EP4006680B1 (en) | Systems and methods for controlling a robotic vehicle | |
US20220340165A1 (en) | Systems and methods for vehicular safety control | |
US20230192090A1 (en) | Precedence determination at multi-way stops | |
US20220198714A1 (en) | Camera to camera calibration |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: THE BOEING COMPANY, VIRGINIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:LARSEN, TY A.;MITCHELL, BRADLEY J.;MARTIN, DAMIEN O.;AND OTHERS;SIGNING DATES FROM 20221104 TO 20230218;REEL/FRAME:062831/0967 |
|
AS | Assignment |
Owner name: PHANTOM AUTO INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MAGZIMOF, SHAY;PARUNANKIAN, DAVID;GLAVTCHEV, VLADIMIR ROUMENOV;REEL/FRAME:062863/0275 Effective date: 20200414 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
AS | Assignment |
Owner name: TRIPLEPOINT PRIVATE VENTURE CREDIT INC., CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:PHANTOM AUTO INC.;REEL/FRAME:066439/0180 Effective date: 20240209 |