CN111615722A

CN111615722A - Method and system for risk control in switching driving modes

Info

Publication number: CN111615722A
Application number: CN201780097092.1A
Authority: CN
Inventors: H·郑; T·P·小戴利; D·W·刘
Original assignee: Zhijia Technology Co ltd
Current assignee: Zhijia Technology Co ltd; PlusAI Corp
Priority date: 2017-12-19
Filing date: 2017-12-19
Publication date: 2020-09-01
Also published as: EP3729398A4; WO2019122968A1; EP3729398A1

Abstract

The present teachings relate to methods, systems, and media for switching modes of a vehicle. Real-time data relating to the status of the vehicle and the driver in the vehicle is received and used to determine a set of tasks to be performed in order to switch the vehicle from the current mode to the different mode, in accordance with a task model. A first duration of a handover requirement is determined based on a first risk associated with a current mode. The task duration required to complete each task is estimated. A second risk associated with the handover is determined based on the required first duration and a total duration determined based on a task duration required to complete the set of tasks. If the second risk satisfies a criterion, the vehicle mode is switched.

Description

Method and system for risk control in switching driving modes

Cross Reference to Related Applications

This application is related to U.S. patent application 15/846,851 filed on 12/19/2017, the entire contents of which are incorporated herein by reference.

Technical Field

The present teachings relate generally to the field of automotive vehicles and hybrid vehicles. In particular, the present teachings relate to a mechanism for operating a vehicle and its augmented behavior planning interface (augmented behavor planning interface).

Background

Automated vehicles employ a variety of computational approaches to assist in automated vehicle operation. Recently, in the automotive industry, much focus has been placed on operating a vehicle in an automatic mode in a safe manner. Autonomous vehicles allow an occupant (driver) to manually switch from a human driving mode (i.e., a mode in which the operator fully performs vehicle control) to an automatic mode (i.e., a mode in which the vehicle is essentially self-driving). Vehicles with two operating mode (automatic and manual) capabilities are hybrid drive vehicles.

In the past, switching between automatic and manual driving modes was done manually. In operation, in some cases, the handover itself may present a hazard if it is done in a particular situation. In addition, even when the autonomous vehicle detects a potential hazard and decides to switch control to the human driver, the manner in which the human driver takes over control in the manual mode may depend on the state of the occupant. For example, in an autonomous driving mode, the occupant may have given less attention, or may even fall into the dream. These real situations in autonomous driving have not been addressed in the past.

Another aspect relates to automatic switching of a hybrid driving mode vehicle in the opposite direction, i.e. from a human driving mode to an automatic driving mode. Conventional methods do not address this direction switch, no mention how to do it in a safe manner.

Therefore, there is a need for a solution to these problems.

Disclosure of Invention

The teachings disclosed herein relate to methods, systems, and programming for augmented behavior planning in automated and hybrid vehicle driving environments. In particular, the present teachings relate to methods, systems, and programming for a mechanism to perform handoff (handover) operations associated with different operating modes of a vehicle.

By one aspect of the present disclosure, a method is disclosed that is implemented on a machine having at least one processor, memory, and a communication platform connectable to a network for switching a vehicle from a current mode to a different mode. The method comprises the steps of: real-time data relating to the vehicle and the status of a driver in the vehicle is received, and a set of tasks to be performed to switch the vehicle from a current mode to the different mode is determined based on the real-time data according to a task model. The method further comprises the following steps: a first duration required to switch from the current mode to the different mode is determined based on a first risk associated with the current mode, and a task duration required to complete the task is estimated for each of the set of tasks. The method infers a second risk of switching from the current mode to the different mode based on the requested first duration and a total duration determined based on a duration of the tasks required to complete the set of tasks, and switches the vehicle from the current mode to the different mode in response to the second risk satisfying a criterion.

By one aspect of the present disclosure, a system for switching a vehicle from a current mode to a different mode is disclosed. The system includes a switching risk estimator configured to receive real-time data relating to a state of the vehicle and a driver in the vehicle, determine a set of tasks to be performed to switch the vehicle from a current mode to the different mode based on the real-time data according to a task model. The system also includes a response time estimator configured to determine a first duration required to switch from the current mode to the different mode based on a first risk associated with the current mode, and estimate, for each of the set of tasks, a task duration required to complete the task. The system also includes a switch risk estimator configured to estimate a second risk of switching from the current mode to the different mode based on the required first duration and a total duration determined based on a duration of tasks required to complete the set of tasks. The switching risk estimator switches from the current mode of the vehicle to the different mode in response to the second risk satisfying a criterion.

Other concepts relate to software for implementing the present teachings with respect to developing vehicle systems. According to this concept, a software product includes at least one machine-readable non-transitory medium and information carried by the medium. The information carried by the medium may be executable program code data, parameters associated with the executable program code and/or information related to the user, requests, content or information related to social groups, etc.

In one example, a machine-readable medium having information stored thereon for switching a vehicle from a current mode to a different mode is disclosed, wherein the information, when read by a machine, causes the machine to perform the steps of: receiving real-time data relating to a vehicle and a status of a driver in the vehicle; a set of tasks to be performed to switch the vehicle from the current mode to the different mode is determined based on the real-time data according to the task model. The steps further include determining a first duration required to switch from the current mode to the different mode based on a first risk associated with the current mode, and for each of the set of tasks, inferring a task duration required to complete the task. Further, the steps include inferring a second risk of switching from the current mode to the different mode based on the requested first duration and a total duration determined based on a duration of tasks required to complete the set of tasks, and switching the vehicle from the current mode to the different mode in response to the second risk satisfying a criterion.

Additional advantages and novel features will be set forth in part in the description which follows and in part will become apparent to those skilled in the art upon examination of the following description and drawings or may be learned by manufacture or operation of the examples. The advantages of the present teachings may be realized and attained by practice and application of the various aspects of the methods, apparatus, and combinations particularly pointed out in the detailed examples discussed below.

Drawings

The methods, systems, and/or programming described herein are further described in terms of exemplary embodiments. These exemplary embodiments are described in detail with reference to the accompanying drawings. These embodiments are non-limiting exemplary embodiments, in which like reference numerals represent like structures throughout the several views of the drawings, and in which:

FIG. 1 illustrates an exemplary diagram showing switching operating modes among vehicles, in accordance with one embodiment of the present teachings;

FIG. 2 illustrates an exemplary block diagram of a vehicle driving mode switching unit according to one embodiment of the present teachings;

FIG. 3 shows an illustrative flow chart of an exemplary process performed by a vehicle driving mode switching unit according to one embodiment of the present teachings;

FIG. 4 illustrates a chart that presents information associated with real-time intrinsic and extrinsic data in accordance with various embodiments of the present teachings;

FIG. 5 illustrates a chart presenting information associated with real-time vehicle data in accordance with various embodiments of the present teachings;

FIG. 6 illustrates a chart presenting information associated with a real-time human state in accordance with various embodiments of the present teachings;

FIG. 7 illustrates an exemplary block diagram of a risk evaluator included in a vehicle driving mode switching unit, according to an embodiment of the present teachings;

FIG. 8 illustrates an exemplary chart that sets forth the types of risk assessments according to one embodiment of the present teachings;

FIG. 9 shows an illustrative flow chart summarizing an exemplary process performed by a risk evaluator contained in a driving mode switching unit of a vehicle in accordance with an embodiment of the present teachings;

FIG. 10 illustrates an exemplary block diagram of a driver status analyzer included in a risk evaluator in accordance with an embodiment of the present teachings;

FIG. 11 shows an illustrative flow chart summarizing an exemplary process performed by a driver status analyzer contained in a risk evaluator unit of a vehicle in accordance with an embodiment of the present teachings;

FIG. 12 illustrates an exemplary block diagram of a switching risk determiner included in a driving mode switching unit, in accordance with an embodiment of the present teachings;

FIG. 13 depicts an illustrative flow chart summarizing an exemplary process performed by a handover risk determiner according to an embodiment of the present teachings;

FIG. 14 illustrates an exemplary block diagram of an automatic mode-to-human driver (A-H) mode switching risk determiner, according to an embodiment of the present teachings;

FIG. 15 shows an illustrative flow chart summarizing an exemplary process performed by an (A-H) mode switch risk determiner according to an embodiment of the present teachings;

FIG. 16 illustrates a chart that presents information associated with response times in accordance with various embodiments of the present teachings;

FIG. 17A illustrates an exemplary block diagram of a human driver mode to automatic (H-A) mode switching risk determiner, according to an embodiment of the present teachings;

FIG. 17B illustrates an exemplary table summarizing a plurality of vehicle operating modes according to an embodiment of the present teachings;

FIG. 18 depicts an exemplary flowchart outlining an exemplary process performed by an (H-A) mode switch risk determiner according to one embodiment of the present teachings;

FIG. 19 illustrates an exemplary block diagram of a handover alert control unit according to one embodiment of the present teachings;

FIG. 20A shows an illustrative flow chart summarizing an exemplary process performed by a handover alert control unit according to one embodiment of the present teachings;

FIG. 20B illustrates an exemplary mission plan presenting different mediums for alerting a driver of a vehicle in accordance with an embodiment of the present teachings;

FIG. 21 illustrates an exemplary block diagram of a multi-modal switching alert unit in accordance with an embodiment of the present teachings;

FIG. 22 shows an illustrative flow chart summarizing an exemplary process performed by a multimodal switch alert unit in accordance with an embodiment of the present teachings;

FIG. 23 illustrates the architecture of a mobile device that can be used to implement a dedicated system incorporating the present teachings; and is

FIG. 24 illustrates the architecture of a computer, which can be used to implement a specific purpose system incorporating the present teachings.

Detailed Description

In the following detailed description, by way of example, numerous specific details are set forth in order to provide a thorough understanding of the relevant teachings. However, it will be apparent to one skilled in the art that the present teachings may be practiced without these specific details. In other instances, well-known methods, procedures, components, and/or circuits have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

An autonomous vehicle is a vehicle that is capable of detecting its environment and navigating through the environment without human input. Automotive vehicles detect their surroundings using a variety of techniques, such as radar, laser, GPS, odometers, and computer vision. The advanced control system interprets the sensor information to identify the appropriate flight path as well as the obstacles and associated signs. Automated vehicles include a control system that is capable of analyzing sensor data in order to distinguish between different vehicles on a roadway. Potential benefits of automotive vehicles include reduced mobility and infrastructure costs, increased safety, increased mobility, increased user satisfaction, reduced crime rates. Specifically, significant reductions in traffic collisions and associated costs due to autonomous vehicles, including less need for insurance. Automated vehicles are expected to increase traffic flow, free travelers from driving and navigation chores, and reduce fuel consumption.

However, there are certain obstacles to the widespread adoption of automotive vehicles. For example, disputes about liability, the period of time required to replace an existing vehicle inventory, individual conflict with out-of-control, consumer safety concerns, implementation of a viable legal framework, establishment of government regulations, loss of privacy risks, and safety concerns are certain factors that prevent adoption of automated vehicles. Thus, the following describes a framework of vehicle systems that use an automatic mode of operation as well as a human driver mode of operation. Note that in this disclosure, the terms "automatic mode" and "automatic driving mode" are used interchangeably to correspond to automatic operation of the vehicle, while the terms "human driver mode" and "manual mode" are used interchangeably to correspond to operation of the vehicle by a human driver.

FIG. 1 shows an illustrative view showing operating mode switching among vehicles, according to an embodiment of the present teachings. As shown in fig. 1, the vehicle includes a driving mode switching unit 130 and a multi-mode switching warning unit 140. The vehicle may operate in an automatic mode 150 or a human driver mode 160. The driving mode switching unit 130 is configured to switch the vehicle operation mode between the automatic mode 150 and the human driver mode 160, respectively.

By one embodiment, the driving mode switching unit 130 initially determines the current mode in which the vehicle is operating. In determining the current operation mode, the driving mode switching unit 130 determines whether a transition to a different mode is required based on a specific criterion (described below with reference to fig. 2). For example, if the current operation mode is determined to be the automatic mode (or, alternatively, the human driver mode), the driving mode switching unit 130 determines whether a transition to the human driver mode (or, alternatively, the automatic mode) is required based on a criterion associated with operating the vehicle in the current mode. Upon a positive determination to switch the current running mode, the driving mode switching unit 130 provides an instruction to the multimodal switching alert unit 140, which indicates an intention to switch the running mode of the vehicle.

Upon receiving an instruction indicating an intention to perform the vehicle operation mode switching, the multimodal switching alert unit 140 transmits an alert signal (described below with reference to fig. 21) to an operator (e.g., a driver) of the vehicle. In addition, the alert signal may include such specific tasks: which is to be performed by the driver of the vehicle in order to switch the vehicle operation mode in an efficient manner. Thus, in one embodiment, the multimodal switch alert unit 140 monitors the driver to determine if the tasks are being performed in time (real time). Specifically, the multimodal switch alert unit 140 monitors various tasks assigned to the vehicle driver to ensure that the assigned tasks are being performed in due time. The driving mode switching unit 130 receives a feedback signal indicating the state of each task assigned to the vehicle driver. The driving mode switching unit 130 performs switching of the vehicle running mode based on successful completion of each task. By way of an example, if a task is not completed in due time, the driving mode switching unit 130 receives a feedback signal indicating that an exception handling process is to be performed. Details regarding the operation of the multimodal switch alert unit 140 and the driving mode switch unit 130 are described in the remainder of the disclosure.

FIG. 2 illustrates an exemplary block diagram of the driving mode switching unit 130 according to an embodiment of the present teachings. The driving mode switching unit 130 includes a current risk evaluator 230, a switching risk determiner 240, a switching alert control unit 250, a switching executor 280, a vehicle control system 260, a driver profile 210, and a map/road configuration model 220.

The current risk evaluator 230 is configured to determine a risk of operating the vehicle in the current operating mode. By one embodiment, to determine the risk, the current risk assessor 230 receives as input real-time vehicle data, real-time intrinsic and extrinsic data, sensor data, the driver profile 210, and map/road configuration data 220. The map/road configuration data 220 provides information about the geographic location where the vehicle is currently located. Additionally, the map/road configuration data 220 may include information corresponding to traffic (traffic) within the vehicle's geographic location. The driver profile 210 data includes information about the driver's driving history. Such information may include, for example, the number of times the driver has been involved in a violation within a predetermined period of time, the model of the vehicle the driver is operating, characteristics of the driver (e.g., whether the driver is disabled, whether the driver is myopic, whether the law requires the driver to wear a device that assists the driver in operating the vehicle (e.g., prescription glasses), weather conditions under which the driver is preferably recommended not to operate the vehicle, etc.).

Turning to FIG. 5, exemplary real-time vehicle data input to current risk evaluator 230 is shown. In particular, FIG. 5 shows an illustrative chart that presents information associated with real-time vehicle data in accordance with various embodiments of the present teachings. For example, the real-time vehicle data may include information about the current location of the vehicle (e.g., latitude and longitude locations), information about the surroundings of the vehicle, the current visibility experienced by the driver of the vehicle (e.g., amount of sun glare based on the current time), controls specific to the weight or quality of the vehicle (e.g., the current weight of occupants in the vehicle and other control parameters related to vehicle operation), and information about vehicle maintenance (e.g., current fuel volume, tire conditions, brakes, wipers), and other important parameters that determine whether the vehicle should be operated in a safe manner.

The present risk assessor 230 also receives real-time data (extrinsic and intrinsic vehicle performance) that is used in determining the risk of the vehicle's current operating mode. Turning to FIG. 4, an illustrative chart is shown that presents information associated with real-time intrinsic and extrinsic data in accordance with various embodiments of the present teachings. As shown in fig. 4, the data related to the intrinsic performance of the vehicle may include information about vehicle operating parameters, such as a supported speed of the vehicle, a supported safety level, a defect in the vehicle (e.g., low tire pressure, defective brake pads, defective headlights/taillights, etc.), and a reliability factor of the vehicle. The data relating to the vehicle's external performance may include information external to the vehicle, such as the type and condition of the road the vehicle is traversing (e.g., the steepness/slope of the road, the speed limit permitted for the road, the current vehicle and human traffic on the road, and road surface conditions such as whether the road is slippery), weather information corresponding to the vehicle environment (e.g., the amount of ice, snow, or precipitation on the road the vehicle is traversing), visibility into the vehicle environment, external events such as whether there is an accident on the vehicle's path, whether the vehicle is traversing a restricted zone (e.g., a hospital or school), or whether there is construction on the vehicle's path that results in a detour from the vehicle's originally planned path, and so forth.

A further input to the present risk assessor 230 is real-time sensor data that is used to determine the current status of the vehicle driver. Turning to FIG. 6, an illustrative chart is shown that presents information associated with a vehicle driver state in accordance with various embodiments of the present teachings. Such information may include data corresponding to the attitude of the driver and/or occupant (e.g., detected position, sitting position, whether the driver is wearing a seat belt, etc.) and the position of the driver and occupant within the vehicle. Additional information associated with the driver status may include information about the health of the driver, the functional status of the driver (i.e., whether the driver is drowsy, asleep, or drowsy, and whether the driver is intoxicated), the mental status of the driver (e.g., the level of alertness of the driver), and information about the nature of the journey made by the driver (e.g., whether the driver and/or occupants are severely injured, and whether the vehicle is traveling on the road to a hospital, etc.). The sensor data used by the current risk assessor 230 to detect the driver state described above is described below with reference to FIG. 10.

Upon receiving the above-described input, risk assessor 230 determines the risk involved in the current operating mode of the vehicle. Specifically, the risk assessor 230 determines whether it is safe to operate the vehicle in the current operating mode. Details regarding the calculation of this risk are described below with reference to fig. 7. In calculating the risk, the current risk assessor 230 determines whether the calculated risk satisfies a particular criterion (e.g., whether the calculated risk is above or below a predetermined first threshold risk level). The predetermined first threshold risk level associated with a particular operating mode of the vehicle may correspond to a safety level for operating the vehicle in the particular operating mode of the vehicle. In particular, if the calculated risk is above a first predetermined threshold risk level, it may be deemed unsafe to operate the vehicle in the current mode, and if the calculated risk is below the first predetermined threshold risk level, it may be deemed that the vehicle may continue to operate in the current mode. Based on the calculated risk satisfying (or not satisfying) the criterion, the risk evaluator 230 may determine whether it is safe to operate the vehicle in the current operating mode or whether a switch from the current mode to a different mode of operating the vehicle is to be performed.

By one embodiment, if the calculated risk associated with the current operating mode is within the allowable limit (e.g., the risk is below a first predetermined threshold risk level), the vehicle remains in the current vehicle operating mode (i.e., no transition to the vehicle operating mode is performed). However, if the calculated risk does not satisfy the criterion (e.g., the calculated risk is above a first predetermined threshold risk level), the current risk evaluator 230 actuates the switch risk determiner 240 to determine whether it is feasible to switch the vehicle operating mode to a different mode.

The switch risk determiner 240 receives real-time intrinsic and extrinsic data, information about the current state of the driver, the current mode in which the vehicle is operating. By one embodiment, the switch risk determiner 240 is configured to determine a risk associated with switching the vehicle to operate in a different operating mode. For example, if the vehicle is currently operating in the automatic mode, the switching risk determiner 240 calculates a risk of switching the vehicle to operate in the human driving mode (i.e., the manual mode), and similarly, if the vehicle is currently operating in the human driving mode, the switching risk determiner 240 calculates a risk of switching the vehicle to operate in the automatic mode. The determination of the risk incurred to switch vehicle operating modes is calculated based on several factors as described below with reference to fig. 12.

By one embodiment, the current risk evaluator 240 determines a current risk associated with a current operating mode of the vehicle. If the current risk is determined to be abnormally high, i.e., greater than a second predetermined threshold associated with the current mode (significantly greater than the first predetermined threshold (e.g., a safety level), the risk determiner 240 may not initialize the switch risk determiner to determine the risk associated with switching the vehicle operating mode to a different mode.

Instead, the current risk assessor 230 may perform exception handling procedures, such as stopping the vehicle in a safe manner. For example, consider that the current vehicle operating mode is a human driving mode. If the risk determiner 230 determines that, for example, the driver of the vehicle is experiencing a disease emergency or life threatening event (detected via a driver status analyzer included in the risk evaluator 230 and described below with reference to FIG. 7), then the risk determiner 230 may not initialize the switch risk determiner 240 to determine the risk associated with switching vehicle operating modes. Instead, the present risk determiner 230 may perform exception handling procedures, such as immediately bringing the vehicle to a stop, initiating a call for government regulatory assistance (e.g., automatically dialing 911 or calling roadside assistance via a mobile phone associated with the vehicle), sending the GPS location of the vehicle to a server that monitors the operation of a particular vehicle, and other similar vehicles. It must be understood that performing the function of completely stopping the vehicle as described above may comprise the processing steps of: such as determining the current speed of the vehicle, the current lane in which the vehicle is traveling, adjacent traffic, identifying shoulder areas on the road (e.g., side-to-side parking zones), and gradually bringing the vehicle to a complete stop in a safe manner. Thus, as shown in FIG. 2, current risk determiner 230 may bypass switch risk determiner 240 and signal switch actuator 280 directly to indicate an impending vehicle operation change to vehicle control 260, such as bringing the vehicle to a stop.

By way of an embodiment, if current risk evaluator 230 determines that the current risk associated with the current vehicle operating mode is above a first predetermined threshold associated with the current mode (the safety level of the current mode) and below a second predetermined threshold level associated with the current mode, the current risk evaluator actuates switch task determiner 240 to determine the risk associated with switching the vehicle to a different mode. If the switch risk determiner 240 determines that a risk associated with switching the vehicle operating mode to a different mode is feasible, i.e., the switch risk determiner 240 determines that the vehicle can operate in the different mode in an efficient manner, the switch risk determiner 240 controls the switch alert control unit 250 to send a signal to the multimodal switch alert unit 140 that provides an indication to the driver (e.g., via visual and audio means as described below with reference to FIG. 19) to perform a particular set of tasks so that the vehicle can efficiently switch to the different mode. Specifically, the multimodal switch alert unit 140 provides an indication to the driver that: there is a risk that the current operating mode of the vehicle will be changed over to a different operating mode of the vehicle, and therefore a switch to a different operating mode of the vehicle will occur urgently based on the driver performing the set of designated tasks. Details regarding the operations involved in switching the vehicle operating mode to a different operating mode are described below with reference to fig. 19.

Conversely, if the risk associated with switching the vehicle to a different mode is determined to be high by the switching risk determiner 240, the switching risk determiner 240 may control the switching actuator 280 to instruct the vehicle control system 260 to implement the exception handling process previously described.

FIG. 3 shows an illustrative flow chart of an exemplary process performed by the driving mode switching unit 130 of the vehicle in accordance with an embodiment of the present teachings. The process begins in step 310, where the driving mode switching unit 130 receives real-time information (i.e., real-time vehicle data, real-time intrinsic and extrinsic data) and sensor data.

Further, in step 320, based on the information received in step 310, the process determines the current driving mode of the vehicle and the current status of the driver operating the vehicle. Details regarding the detection of the current state of the driver operating the vehicle are described below with reference to fig. 10, while details regarding the determination of the current operating mode of the vehicle are described below with reference to fig. 7.

Further, in step 330, the process assesses a risk associated with the current operating mode of the vehicle. Thereafter, in step 340, the process executes a query to determine if the risk of operating the vehicle in the current mode, if assessed, is abnormally high. For example, as previously described, the process determines whether the assessed risk is above a second predetermined threshold associated with the current operating mode of the vehicle. If the response to the query is positive, the process moves to step 395, where the exception handling process is performed as previously described. However, if the response to the query is not qualitative, the process moves to step 350.

In step 350, the process determines a risk of a switch, i.e., a risk associated with transitioning the vehicle to a different vehicle operating mode. By way of an embodiment, and as described below with reference to fig. 12, the risk of switching may be calculated based on a driver of the vehicle performing at least one task within an estimated time of performing the at least one task, wherein the estimated time is determined based on a state of the driver. In addition, it must be understood that upon switching from the current mode to a different operating mode of the vehicle, the risk of the different operating mode of the vehicle does not exceed the risk threshold level associated with the different mode.

The process then moves to step 360 where a query is performed to determine if the risk of handover assessed in step 350 is high. By way of an example, the process may determine whether there are other switching risks associated with transitioning the vehicle to the different mode. If the response to the query in step 360 is positive, the process moves to step 395 and an exception handling process is performed, otherwise the process moves to step 370.

In step 370, the process selects at least one alert medium from a plurality of alert medium sources to alert a driver of the vehicle of an emergency switch in operating modes of the vehicle. By one embodiment, the alert medium is selected based on the risk of handover assessed in step 350. In addition, with one aspect of the present disclosure, a warning medium may be selected based on the driver's status, wherein the warning medium warns the driver to perform a specific task (described below with reference to fig. 21) in order to efficiently switch/switch the operating mode of the vehicle.

In step 380, the process performs warning based on the selected medium, and thereafter in step 390, the driving mode switching unit 130 performs vehicle running mode switching.

Turning now to fig. 7, an exemplary block diagram of a risk evaluator 230 included in a driving mode switching unit 130 of a vehicle is shown, according to an embodiment. As previously described, the risk assessor 230 is configured to determine a risk associated with a current mode of vehicle operation. As shown in fig. 8, with one embodiment of the present disclosure, risk assessment is performed in an automatic mode of operation and a manual mode of operation, respectively. In addition, associated with each operating mode is a risk assessment corresponding to the risk incurred when switching from the current operating mode to a different mode. It must be understood that the number of modes in which the vehicle can operate is not limited to the two modes shown in fig. 8. Additionally, the vehicle can be operated in a variety of modes as described below with reference to FIG. 17B.

The risk assessor 230 as shown in FIG. 7 includes a driver status analyzer 710, a driver risk pattern 720, a current pattern detector 760, an automatic pattern risk detector 730, a manual pattern risk detector 770, a current pattern risk report generator 740, an abnormal risk processor 750, and a driver profile 780.

By one embodiment, the driver state analyzer 710 receives data from a plurality of sensors configured to capture information about a variety of characteristics of the driver and/or occupants within the vehicle. For example, the vehicle system may include a plurality of sensors, including visual sensors, auditory sensors, olfactory sensors, illumination sensors, etc., that enable risk assessor 230 to determine the status of the driver. Details regarding the various sensors and the information captured by them to detect the driver's state are described below with reference to FIG. 10.

The current mode detector 760 is configured to determine a current operating state of the vehicle. By way of one embodiment, the real-time vehicle data previously described with reference to FIG. 5 can be used to determine a vehicle operating mode. For example, sensors disposed within the vehicle may be configured to determine whether the driver is operating the vehicle by detecting, for example, whether a steering wheel is being operated by the driver, whether an accelerator and/or brake pedal is being controlled by the driver, and the like. It must be understood that a variety of mechanisms can be used to determine whether the vehicle is in the human driver operating mode or the automatic operating mode.

Based on the determination of the current vehicle operating mode, current mode detector 760 activates either automatic mode risk detector 730 or manual mode risk detector 770 to determine the risk associated with the respective vehicle operating mode. As inputs, the automatic mode risk detector 730 receives the real-time data and the real-time vehicle data described above with reference to fig. 4 and 5. As inputs, the manual mode risk detector 770 receives real-time data and real-time vehicle data, driver status, and information contained in the driver profile 780.

By way of an embodiment, the automatic mode risk detector 730 may determine a risk event based on environmental conditions. For example, in foggy weather, the automatic mode risk detector 730 may determine that the vehicle is not safe to operate automatically, e.g., a collision detector of the vehicle may have difficulty accurately detecting a nearby vehicle. In a similar manner, when the vehicle is traversing a steep icy road, it may be feasible to operate the vehicle under human control, i.e., the automatic mode may not be suitable for controlling the vehicle under severe (icy) road conditions. In addition, there may also be certain geographic areas where autonomous driving is prohibited or deemed unsafe, such as acceleration lanes, drive-off lanes, toll booths, known construction zones, school zones, and portions of roads near these areas. The risk associated with these events is determined by the automatic mode risk detector 730.

Similar to the scenario of the dangerous autopilot scenario described above, by one embodiment, the manual mode risk detector 770 may determine the situation: where it may be dangerous to run the vehicle under manual control. For example, the manual mode risk detector 770 may determine that the driver is likely to encounter an accident in the imminent future based on the detected driver state and the driver profile 780. Such a determination may be based on the traffic density in the vicinity of the vehicle under consideration. As another example, the risk associated with a manual mode of operation of the vehicle may correspond to detecting that the vehicle has deviated from its predetermined path several times (within a predetermined time window). For example, an in-vehicle sensor may detect that the vehicle is being operated by the driver in such a manner: the vehicle is off center in the lane and enters the adjacent lane. The manual mode risk detector 770 may treat the occurrence of such an event as a risky event based on the amount of traffic in the vicinity of the vehicle. Similarly, the driver state detector may determine that the driver is in a drowsy, and/or yawned state, and running the vehicle in the human driving mode may be at risk.

According to the driving risk model 720, the automatic mode risk detector 730 and the manual mode risk detector 770 detect respective risks associated with operating the vehicle in the automatic mode or the manual mode, respectively. Specifically, based on the respective input information, the risk patterns 720 are used by the automatic pattern risk detector 730 and the manual pattern risk detector 770 to assess the risk of operating the vehicle in the respective modes.

By one embodiment, the automatic mode risk detector 730 and the manual mode risk detector 770 input information about the driver and/or the detected scenarios about the geographic location through which the vehicle is traversing to the risk report generator 740. The risk report generator 740 uses the driving risk model 720 to generate a risk report associated with an automatic mode or a manual mode of vehicle operation, respectively, and outputs a current risk associated with the operating mode of the vehicle.

Additionally, with one embodiment of the present disclosure, the automatic mode risk detector 730 and the manual mode risk detector 770 provide inputs (indicative of the separately detected risks) to the anomaly risk processor 750. As described below with reference to FIG. 12, with one embodiment, upon detecting a risk in a current operating mode of the vehicle, the vehicle system may determine a risk associated with switching the vehicle to a different operating mode. If the risk associated with the different operating modes is within the allowable limit, the vehicle system may perform an operation to switch the operating mode of the vehicle.

However, if the risk associated with the different vehicle operating mode is not acceptable, the current risk evaluator 230 may activate an abnormal risk processor 750, which may be configured to perform safety-related functions as previously described based on the driver profile. For example, the abnormal risk processor 750 may perform processing such as bringing the vehicle to an immediate stop, initiating a call for government regulatory assistance (e.g., auto-dialing 911 or calling roadside assistance via a mobile phone associated with the vehicle), sending the vehicle GPS address to a server that monitors the operation of a particular vehicle and other similar vehicles. It must be understood that in order to perform the function of bringing the vehicle into a complete stop described above, it may comprise the processing steps of: for example, the current speed of the vehicle, the current lane in which the vehicle is traveling, nearby traffic, identifying the shoulder area of the road (e.g., the side parking area), and gradually bringing the vehicle to a complete stop in a safe manner.

FIG. 9 shows an illustrative flow chart outlining an exemplary process performed by risk evaluator 230 according to one embodiment of the present teachings. The process begins at step 905, where the activity of the driver and/or occupant of the vehicle under consideration (described below with reference to FIG. 10) is detected via various sensors.

Based on the information detected by the sensors and a driver detection model (described below with reference to FIG. 10), the process in step 910 analyzes the driver/occupant status. In step 915, the current operating state of the vehicle is determined. For example, the current mode detector 760 shown in fig. 7 may detect a current operating mode of the vehicle.

The process further proceeds to step 920 where a query is performed to determine if the vehicle is operating in the automatic mode. If the response to the query is positive, the process moves to step 925, otherwise, if the vehicle is operating in manual mode, the process moves to step 930.

In step 930, the process detects a risk associated with a vehicle operating in a manual mode. As introduced previously, the determination of such risk may be based at least on the detected state of the driver, information contained in the driver profile, real-time information about the vehicle the driver is operating. It must be understood that the information contained in the driver profile is static driver information corresponding to the characteristics of the driver, such as whether the driver is disabled, myopic, medical information of the driver, etc., and the information related to the detected driver's state is dynamic information based on the current behavior of the driver.

Returning to step 920, if it is determined that the vehicle is operating in the automatic mode, the process moves to step 925 and obtains the operating state of the vehicle in the automatic mode. For example, the process may determine the location of the vehicle, the current speed at which the vehicle is traveling, and similar information. By way of example, the process may be configured to determine a level of an automatic mode (level) at which the vehicle is operating (described below with reference to FIG. 17B).

Upon determining the respective risk of operating the vehicle in the automatic mode (step 935) or in the human driving mode (step 930), the process moves to step 940. In step 940, a query is performed to determine whether exception handling is required, i.e., the risk of the determined automatic mode or human driving mode of the vehicle is abnormally high. If the response to the query is positive, the process moves to step 950, where the vehicle is operated in an exception handling mode, performing at least one of the exception handling functions previously described.

However, if the response to the query in step 940 is negative, the process moves to step 955, where a risk assessment report is generated. Additionally, the generated risk assessment report may be displayed on a display panel included within the vehicle and/or transmitted to a remote server for monitoring purposes, as shown in step 960.

FIG. 10 illustrates an exemplary block diagram of a driving state analyzer 710 according to an embodiment of the present teachings. The driver status analyzer 710 includes a sensor actuator 1010, a variety of in-situ (in-situ) sensors 1020, a driver detector 1030, a driver detection model 1040, a behavior feature detector 1050, a feature behavior model 1060, a driver health estimator 1070, a functional status estimator 1055, a mental status estimator 1080, and a current user status generator 1090.

By one aspect of the present disclosure, the driver status analyzer 710 is configured to determine the status of the driver. Determining the status enables to predict in an accurate manner whether the driver is able to perform an activity related to steering the vehicle, whereby any potentially dangerous situation can be avoided.

By one embodiment, the sensor actuator 1010 actuates the home sensor 1020 to detect the status of the driver. The home sensor 1020 includes a plurality of sensors including a visual sensor, an auditory sensor, an olfactory sensor, a light sensor, and the like, which enable detection of the driver's state. For example, the vision sensor included in-situ sensor 1020 may include multiple spatially distributed camera devices within the vehicle that are capable of processing and fusing images of a scene from multiple perspectives into some form that is more useful than individual images. For example, the vision sensor may be configured to capture the posture of the driver (i.e., the posture/orientation of the driver in the driver's seat). The pose information may be used to determine additional information such as the position of the driver's head relative to the steering wheel, the direction the driver is looking (based on driver eye tracking), etc. Information captured by the home sensor 1020 is input to a driver detector 1030, which determines a driver pose based on a driver detection model 1040, for example.

Information detected by the driver detector unit 1030, such as the posture of the driver, may be input to the behavior feature detector 1050. The behavior feature detector 1050 may be configured to determine features associated with the detected driver gesture based on the feature behavior model 1060. For example, characteristics of whether the driver is asleep, whether the driver is yawned, whether the driver is drowsy and/or nausea, and so forth, may be determined by the characteristic behavior detector 1050.

By one embodiment, the detected driver behavior is input to a driver health estimator 1070, which is configured to determine the health of the driver. By way of an example, an in situ sensor may comprise: a plurality of sensors, such as wearable sensors, disposed on a vehicle driver; and/or sensors contained within a device worn by the driver (e.g., a watch) that collect information about the health of the driver. Such sensors may include: for example, a heart rate sensor configured to determine a heart rate of the driver; a temperature sensor configured to detect a temperature of a skin surface of a driver, and the like. Information about the health of the driver is input to the current user state generator 1090.

According to an embodiment of the present disclosure, information about the behavior of the driver may be input to the functional status estimator 1055 and the mental status estimator 1080. The functional state estimator 1055 also receives additional inputs from the characteristic behavior model 1060, the driver detector unit 1030, and the home sensor 1020 to determine the functional state of the driver. For example, the functional status estimator 1055 may be configured to determine whether the driver is able to operate the vehicle in a safe manner, i.e., whether the driver is functioning in an acceptable manner, based on the detected driver posture/orientation.

Based on the functional state of the driver estimated by the functional state estimator 1055, the mental state estimator 1080 may be configured to determine whether the mental state of the driver is normal. Such information may be used to determine whether the driver is able to maneuver the vehicle. Mental state information may include information about whether the driver is experiencing a life-threatening event (e.g., a sudden illness). It must be understood that information from sensors such as medical sensors (including heart rate monitors, etc.) may be used by the mental state estimator 1080 to determine the mental state of the driver. The information about the driver estimated by the modules such as the health estimator 1070, the mental state estimator 1080, and the functional state estimator 1055 is input to the current user state generator 1090, which combines the estimated health-related information to determine the current state of the user, i.e., the driver state information.

FIG. 11 shows an illustrative flow chart summarizing an exemplary process performed by driver status analyzer 710 according to one embodiment of the present teachings. The process begins in step 1110, where sensor actuators (1010 in FIG. 10) actuate a plurality of home sensors to respectively capture information about a vehicle driver. In step 1120, the driver detector detects the state of the driver using the driver detection model upon receiving information from the corresponding sensor (step 1130). Specifically, in step 1103, information regarding the vehicle driver's state, such as the position of the driver in the vehicle, the attitude of the driver, etc., may be determined based on the driver detection model.

The process then proceeds to step 1140, where the driver's behavioral characteristics are detected. Specifically, in step 1140, based on the information detected in step 1130, the behavioral characteristic model may be used to detect behavioral characteristics, such as whether the driver is asleep, drowsy, nausea, etc. The behavioral characteristics detected in step 1140 may be used in

steps

1150, 1160, and 1170 to estimate the functional status of the driver, the health of the driver, and the mental status of the driver, respectively, as described above with reference to fig. 10.

The detected functional state, health and mental state of the driver may be used in step 1180 to generate an overall state of the driver. For example, the current driver state generator (1090 in FIG. 10) may be used to determine an overall state of the driver based on detected mental, functional, and health information of the driver.

The process then proceeds to step 1190, where the generated driver state from step 1180 may be output for inclusion in the current mode risk report generator (740 in FIG. 7) and/or for use by the switching risk determiner and switching risk evaluator to determine, for example, a risk associated with the current mode in which the vehicle is operating.

FIG. 12 illustrates an exemplary block diagram of a handoff risk determiner 240 according to an embodiment of the present teachings. The switching risk determiner 240 includes a switching direction controller 1210, an automatic to human (a-H) mode switching risk determiner 1230, an a-H anomaly risk processor 1220, a human to automatic (H-a) mode switching risk determiner, and an H-a anomaly risk processor 1250.

As an input, the switch direction controller 1210 receives the current vehicle operating mode. Based on the current mode, handoff direction controller 1210 activates one of a-H handoff risk determiner 1230 and H-a handoff risk determiner 1240. As described below with reference to fig. 14 and 17, respectively, a-H handoff risk determiner 1230 determines a risk associated with transitioning from an automatic operating mode of the vehicle to a human driver operating mode, while H-a handoff risk determiner 1240 determines a risk associated with transitioning from a human driver mode of the vehicle to an automatic operating mode.

By way of an embodiment, a-H handoff risk determiner 1230 and H-a handoff risk determiner 1240, respectively, generate vehicle control signals based on a determination that a risk associated with switching the current mode of the vehicle is below a predetermined threshold level. Vehicle control signals may be sent by the a-H handoff risk determiner 1230 and the H-a handoff risk determiner 1240, respectively, to the handoff actuators (e.g., block 280 in fig. 2) to instruct the vehicle controls (260 in fig. 2) to perform the corresponding vehicle operating mode handoff.

In addition, a-H handoff risk determiner 1230 and H-a handoff risk determiner 1240 generate the following signals, respectively: an a-H switch alert command and an H-a switch alert command (described below with reference to fig. 14 and 17A, respectively). By one embodiment, both the A-H switch alert command and the H-A switch alert command are associated with a set of tasks to be performed by the vehicle driver, such that a successful switch in vehicle operating mode can occur.

However, if either the a-H handoff risk determiner 1230 or the H-a handoff risk determiner 1240 determines that the risk associated with the vehicle operating mode switch is unacceptable (i.e., high risk mode), the a-H handoff risk determiner 1230 and the H-a handoff risk determiner 1240 activate the a-H anomaly risk processor 1220 and the H-a anomaly risk processor 1250, respectively. As previously introduced, the a-H anomaly risk processor 1220 and the H-a anomaly risk processor 1250 may be configured by one embodiment to perform safety-related functions, such as stopping the vehicle immediately, placing a call to government authorities that requires assistance (e.g., auto-dialing 911 or calling roadside assistance, via a mobile phone associated with the vehicle), sending the GPS location of the vehicle to a server that monitors the operation of a particular vehicle and other similar vehicles. It must be understood that in order to perform the function of bringing the vehicle to a complete stop described above, it is possible to include the processing steps of: for example, the current speed of the vehicle, the current lane in which the vehicle is traveling, nearby traffic are determined, a shoulder portion of the road (e.g., a side parking area) is identified, and the vehicle is gradually brought to a complete stop in a safe manner.

Fig. 13 shows an illustrative flow chart outlining an exemplary process performed by the handover risk determiner 240 according to one embodiment of the present teachings. The process begins at step 1310, where real-time information (described above with reference to FIG. 4) is obtained by a handover risk determiner. In step 1315, the switch risk determiner receives information about the current state of the driver. For example, the switch risk determiner obtains the driver current state from the current driver state generator 1910 in fig. 10.

The process further proceeds to step 1320, where a query is performed to determine whether the vehicle intends to switch from an automatic mode of operating the vehicle to a human drive (A-H) mode, or whether the vehicle intends to switch from a human drive of operating the vehicle to an automatic (H-A) mode. If the response to the query is to switch from automatic mode to human driving mode (i.e., A-H switch), the process moves to step 1325. If the response to the query is to switch from human driving to automatic mode (i.e., H-A), the process moves to step 1350.

In step 1325, the switch risk determiner determines a risk associated with performing a switch from an automatic mode to a human driving mode of operating the vehicle. As will be described in detail later with reference to fig. 14 and 17B, when switching from the automatic mode to the human driver mode, the driver is presented with a set of presumed tasks to be performed in order to switch the operating mode of the vehicle. Based on whether the driver performs the set of presumed tasks, the a-H switch risk determiner 240 determines a risk associated with switching the vehicle to human driver mode.

Thereafter, the process in step 1330 performs a query to determine whether a high risk is associated with transitioning to human driving mode. If the response to this query is positive, the process moves to step 1345, where an exception handling process is performed. However, if the response to the query in step 1330 is negative, the process moves to step 1335.

In step 1335, the process generates an a-H switch command alert that is associated with a set of tasks to be performed by the vehicle driver before a successful switch to the human drive mode of vehicle operation can occur. Thereafter, the process in step 1340 generates (and outputs) an A-H vehicle control signal to instruct the control system of the vehicle to perform a switching operation while the driver is performing the set of presumed tasks.

Similar to steps 1325 through 1345 described above, the switch risk determiner in steps 1350 through 1370 performs functions related to switching from a human driving mode to an automatic mode of operating the vehicle. Specifically, in step 1350, the switch task determiner determines a risk associated with performing a switch from a human driving mode to an automatic mode of operating the vehicle.

The process then moves to step 1355, where a query is made to determine if the high risk is associated with a transition to an automatic mode of operating the vehicle. If the response to the query is positive, the process moves to step 1370, where an exception handling process is performed. However, if the response to the query in step 1335 is not qualitative, the process moves to step 1360.

In step 1360, the process generates an H-a switch order alert that is associated with a set of tasks to be performed by the vehicle driver when a successful switch to the automatic mode of operating the vehicle can occur. Thereafter, the process generates (and outputs) an H-A vehicle control signal in step 1365 to instruct the vehicle control system to perform the switch to the auto-run mode.

FIG. 14 illustrates an exemplary block diagram of an automatic mode-to-human driver/manual (A-H) mode switching risk determiner 1230, according to an embodiment of the present teachings. As previously described, the a-H switch risk determiner 1230 is configured to determine a risk associated with switching the operating mode of the vehicle from the automatic mode to the human driver operating mode. As shown in fig. 14, the a-H switching risk determiner 1230 includes a driver profile 1410, a general response time profile 1420, a response time estimator 1430, a switching task estimator 1460, a switching task model 1415, a switching risk estimator 1440, and an a-H warning instruction generator 1450.

With an embodiment of the present disclosure, the a-H switching task determiner 1230 determines the amount of time that the vehicle should transition to human driving mode to avoid a dangerous scene based on the degree/level of risk associated with the current mode of operating the vehicle (i.e., referring to the automatic mode of fig. 14). For example, consider the case where the vehicle is driving in an autonomous mode along a road covered with dense fog. In such a case, the sensors of the vehicle may not be able to accurately detect nearby vehicles, and thus, it may not be feasible to continue operating the vehicle in the automatic mode. Based on the risk associated with the automatic operating mode (e.g., determined based on real-time fog conditions, traffic volume in the area, etc.), a-H switch risk determiner 1230 may determine an amount of time that a transition to the human mode of operating the vehicle should be made for safety considerations.

Additionally, as inputs, switching task estimator 1460 receives the current occupant state, real-time data, and real-time vehicle data. As previously described with reference to fig. 4 and 5, the real-time data includes intrinsic and extrinsic performances of the vehicle, and the real-time vehicle data includes information on a location of the vehicle, an environment in which the vehicle is currently located, visibility, in-service information (in-service information) of the vehicle, and the like. By an embodiment, based on the received input, and in accordance with the switch task model 1415, the switch task estimator 1460 determines a set of tasks to be performed by the driver that enable a switch in vehicle operating mode from automatic to human driver mode. Additionally, with one embodiment, the switch task estimator 1460 receives the current driver state and determines whether the driver can perform any tasks. For example, if the driver status is completely out of alert, faint, etc., the switch task estimator may signal to the switch risk estimator a high risk associated with switching the vehicle operating mode to the human driving mode, and thus instruct the switch risk estimator 1440 to implement an exception handling process as previously described.

Before switching to the manual operating mode of the vehicle, a judgment may be required to verify whether the driver is able to operate the vehicle. Such a determination may include verifying the posture of the driver, determining the driver's level of vigilance, and the like. By way of an embodiment, the driver's level of vigilance may be determined by applying a series of tasks to the driver and determining whether the driver is performing the indicated tasks in a satisfactory manner (where, for example, the tasks may be displayed in the form of instructions on a display panel contained within the vehicle). These tasks may include instructing the driver to perform lane changes, displaying objects (e.g., flashing icons) on the vehicle display panel, and tracking the movement of the driver's eyes to determine the degree of vigilance, among other things. Additionally, a certain amount of time may be required to determine the health of the driver (e.g., via medical sensors such as blood pressure, heart rate monitors, etc. that measure key indicators of the driver, which are processed by sensors (described below with reference to FIG. 24) included in the vehicle system).

Additionally, based on the real-time information received by switching task estimator 1460, a set of tasks may be estimated for execution by the driver to successfully transition the vehicle to human driver mode. It must be understood that the set of tasks estimated by the switching task estimator 1460 according to the switching task model 1415 is not limited to the tasks described above.

As shown in fig. 14, the output of the switching task estimator 1460, i.e., the estimated task to be executed (e.g., by a human driver of the vehicle) to initiate switching to the manual mode of vehicle operation, is input to the response time estimator 1430. The response time estimator 1430 estimates the amount of time required for the vehicle driver to perform the set of estimated tasks. Specifically, the response time estimator 1430 estimates the amount of time required for the driver to complete each task estimated by the switching task estimator 1460. By one embodiment, the response time estimator 1430 estimates the amount of time required to perform the task based on the current occupant state, the driver profile 1410, and the universal response time profile 1420. It must be appreciated that the current occupant state may be obtained from a current state generator (1090, fig. 10) included in the driver state analyzer (i.e., block 710, fig. 7), as previously described.

The driver profile 1410 may contain information about the driver's driving history, such as the number of violations the driver has been involved in a predetermined preceding period of time, the model of vehicle the driver is running, and characteristics of the driver, such as whether the driver is disabled, myopic, etc. Additionally, driver profile 1410 may include a driver alertness score that may be calculated based on previous mode switching operations from automatic to manual mode and the corresponding time required by the driver to perform the set of presumed tasks in those switching operations. Additionally, information from the universal response time profile 1420 (e.g., which contains the amount of time spent by other drivers (having profiles similar to the current driver) in performing the set of presumed tasks) can be used by the response time estimator 1430 to estimate the amount of time needed (by the current driver) to perform the set of presumed tasks.

By way of an example, as shown in fig. 16, the response time required to perform the set of inferred tasks may be determined based on driver physiological information (e.g., the driver's current posture or attitude), user/driver information (e.g., the time required to previously switch the manual operating mode, the driver's status, etc.), extrinsic factors (e.g., road conditions, road visibility, traffic volume on the road), and intrinsic factors of the vehicle (e.g., the vehicle's allowable speed, the vehicle's allowable safety, the vehicle's condition (e.g., the number of vehicle defects)), and so forth.

The estimated set of tasks to be performed and the estimated corresponding response times to perform the tasks are input to the switch risk estimator 1440. By one embodiment, the handoff risk estimator 1440 will estimate the time (T) required to perform a task_E) Compared to the amount of time (T) that the vehicle should be switched to a different mode (human driver mode in the present embodiment). It must be understood that the estimated time to perform the set of tasks is determined by the universal response time profile 1420. If the estimated time (T)_E) Above (T), the switching risk estimator 1440 generates a high risk control signal indicating that the risk associated with switching to the manual operating mode is high, i.e., that the risk of switching to the human driver mode is high. In this case, the a-H switching task determiner 1230 may perform the exception handling operation described above. However, if the time (T) is estimated_E) Below (T), the switching risk estimator 1440 actuates the a-H switching command generator 1450. The a-H switch command generator 1450 generates a-H switch alert command signals associated with the set of tasks to be performed by the vehicle driver to successfully switch the vehicle operation mode to the human driving mode.

Turning to fig. 15, an illustrative flow chart outlining an exemplary process performed by the (a-H) mode switch risk determiner is shown in accordance with an embodiment of the present disclosure. The process begins in step 1510, where (a-H) the switch mode risk determiner receives real-time information including intrinsic and extrinsic properties of the vehicle and real-time vehicle data.

The process then moves to step 1520 where (a-H) the switch mode risk determiner presumes a set of tasks to be performed by the human driver to successfully switch the vehicle operating mode to the human driving mode. Note that with one embodiment, the set of tasks may be inferred based on the received real-time information according to a switching task model.

In step 1530, the response time, i.e., the amount of time required for the vehicle operator to complete each of the set of tasks, is estimated. It must be appreciated that the response time may be calculated based on at least one of the driver profile, the current driver state, and the universal response time profile as previously described with reference to FIG. 14.

The process further proceeds to step 1540, where the risk associated with switching the vehicle operation mode to the human driving mode is determined. As previously described, the risk of switching may be determined based on a comparison of the total estimated time to perform the set of tasks and the time at which the vehicle should transition to the different mode (e.g., human driver mode), which is determined based on a risk level associated with operating the vehicle in a current mode (e.g., automatic mode).

Next, the process in step 1550 executes a query to determine (in step 1540) whether the determined risk is high. For example, a determination may be made to verify whether the risk determined in step 1540 violates a particular criterion, e.g., T_EWhether greater than T. If the response to the query in step 1550 is positive, the process proceeds to step 1570, otherwise the process proceeds to step 1560.

In step 1560, the criterion (i.e., T) is not violated in response to the determined risk in step 1540_EBelow T), the process generates an a-H switching alert command signal that is further used in step 1580 to output (e.g., display on a panel) the set of presumed tasks to the driver of the vehicle. Details regarding the presentation of the set of tasks to the driver are described in more detail below with reference to fig. 19 and 21. However, if the risk in step 1550 is determined to be high, the process performs an exception handling step as previously described in step 1570.

FIG. 17A illustrates an exemplary block diagram of a human driver to automatic (H-A) mode switching risk determiner according to an embodiment of the present teachings. As previously described, the H-a switch risk determiner 1240 is configured to determine a risk of switching the operating mode of the vehicle from the manual mode to the automatic mode of the vehicle. As described below, based on the estimated risk of switching vehicle operating modes, (H-a) a switching mode risk determiner 1240 determines whether it is feasible to switch (from human driver mode) to an automatic mode of operating the vehicle.

As shown in fig. 17A, the H-a handover risk determiner 1240 includes a handover task determiner 1710, a handover time estimator 1720, an automatic takeover risk estimator 1750, an automatic takeover task model 1730, a vehicle model 1740, an H-a alert instruction generator 1760, and a control signal generator 1770. As inputs, the switch task determiner 1710 receives real-time data, real-time vehicle data, and a current driver state. As previously described with reference to fig. 4 and 5, the real-time data includes intrinsic and extrinsic properties of the vehicle, and the real-time vehicle data includes information about the location of the vehicle, the environment in which the vehicle is currently traveling, environmental visibility, vehicle in-service information, and the like. In addition, as an input, the switching task determiner 1710 receives the current occupant state, which is generated by the user state generator as previously described in fig. 10.

Based on the received input, the switch risk determiner 1710 uses the auto-takeover task model 1730 to generate a set of tasks to be performed in an auto-run mode (e.g., by the vehicle). For example, with one embodiment, the set of tasks may include controlling vehicle speed (i.e., decreasing or increasing vehicle speed based on nearby traffic), controlling vehicle maneuverability (i.e., controlling the functionality of the steering wheel of the vehicle), changing the lane in which the vehicle is currently traveling, determining (for safety purposes) whether the vehicle location is in a zone that limits autonomous driving (e.g., certain geographic locations such as school zones, particular areas on highways, like toll booths, etc.). It must be understood that each of the above-mentioned tasks to be performed in the automatic mode may be associated with a task that is: this task will be performed by the vehicle driver to ensure a smooth transition from the human driving mode to the automatic mode of running the vehicle. For example, the task to be performed by the driver may include moving the driver's foot away from the accelerator and/or brake, as the vehicle is to be controlled in an automatic manner in an automatic operating mode. Another task that may need to be performed by the driver may include the driver removing his hands from the steering wheel in order to give control of the vehicle's maneuverability to the automated system.

With one embodiment of the present disclosure, the H-A handoff risk determiner 1240 observes the current state of the driver to determine whether the driver is able to perform any of the above-mentioned tasks. For example, if it is determined that the driver of the vehicle is completely unconscious, the H-a switch risk determiner 1240 may initiate an immediate take-over of the vehicle autonomy function, i.e., the vehicle will operate in an immediate autonomy mode. It must be understood that the H-a handoff risk determiner 1240 may perform exception handling procedures as described below if complete immediate takeover of the vehicle is not feasible.

By an embodiment, the determined sequence of tasks, driver status, and current vehicle data are input to the switch time estimator 1720. Based on the received inputs, the switching time estimator 1720 uses the vehicle model 1740 to estimate the amount of time required by the vehicle and/or correspondingly by the vehicle driver to perform the respective determined task. By one embodiment, the vehicle model 1740 contains information about the previous operation of the current vehicle, such as the average amount of time required for the vehicle to change its speed from a first operating speed to a second operating speed, the amount of time required for the vehicle to change its operating lane (which may be based on nearby traffic), the amount of time required for the vehicle to control vehicle maneuverability, and so forth. Thus, in one embodiment, the switch time estimator 1720 estimates the time required for the vehicle to perform a particular task using real-time information based on the vehicle model 1740.

By one embodiment, the switch time estimator 1720 estimates the total time required to perform all respective estimated tasks by the vehicle and/or, correspondingly, by the vehicle driver. The total estimated time is input to the automatic takeover risk estimator 1750. As an input, the automatic takeover risk estimator 1750 receives an automatic takeover control signal, which may be generated by the switch direction controller 1210 of fig. 12. Specifically, when the switching direction controller (1210 in fig. 12) determines that a transition to the automatic operation mode is to be made, the automatic takeover control signal activates the automatic takeover risk estimator 1750. The automatic takeover risk estimator 1750 evaluates the risk associated with transitioning to an automatic operating mode of the vehicle.

By one embodiment, the risk associated with switching to the automatic mode may be determined based on a level/degree of risk associated with the current operating mode of the vehicle. For example, similar to the description provided above with reference to the a-H handoff risk determiner, the H-a handoff risk determiner determines a risk level associated with operating the vehicle in a human driver mode. Based on the risk level, the H-A switch risk determiner obtains an amount of time that the vehicle should switch to the automatic operating mode for safety considerations. Additionally, the automatic takeover risk determiner 1750 may calculate a risk associated with switching the vehicle to the automatic mode based on a comparison of an estimated time required to perform a task (e.g., by the vehicle) and an amount of time that the vehicle should switch operating modes.

Based on the assessed risk being within an acceptable range, the automatic takeover risk estimator 1750 actuates the H-a alert command generator 1760 and the control signal generator 1770. By one embodiment, H-A alert command generator 1760 generates an H-A switch alert command signal, thereby indicating to the driver that the vehicle is about to immediately transition to an autonomous mode of operation. In addition, user alert instruction generator 1760 may use an audio/visual mechanism (described below with reference to fig. 19 and 21) to indicate to the driver the tasks to be performed in order to successfully transition the vehicle into an automatic operating mode.

As described above, it must be understood that, in the present embodiment, the respective functions/tasks to be performed by the vehicle driver correspond to the tasks to be performed in the automatic operation mode. Additionally, if one of the tasks determined by the auto-takeover task model 1730 is to reduce vehicle speed, such a task may correspond to the driver removing his/her foot from the accelerator pedal. The execution of the H-a switch alert command signal is described in detail below with reference to fig. 19. In addition, the takeover task estimator 1750 triggers the control signal generator 1770 to generate a vehicle control signal to be sent to a switching actuator (e.g., unit 280 in fig. 2). The switch actuator in turn actuates the vehicle control system (element 260 in fig. 2), which indicates that a switch to the automatic operating mode is being performed. It must be appreciated that if risk estimator 1750 determines that the risk associated with transitioning to the automatic mode is unacceptable, risk estimator 1750 may signal control signal generator 1770 to instruct switch actuator 280 to perform exception handling.

Additionally, with one embodiment, when the risk associated with transitioning to the automatic mode is unacceptable, the risk estimator 1750 may perform a variety of additional functions before performing the exception handling function. For example, instead of switching from a human driver mode of operating the vehicle to a fully automatic mode as described above, the H-a switch risk determiner may determine a risk associated with switching the vehicle from a human driver mode of operating the vehicle to a low level automatic mode.

FIG. 17B provides a classification of various vehicle operating modes based on six different levels (ranging from fully manual to fully automatic systems) in accordance with an embodiment of the present teachings. The classification may be based on the amount of driver intervention and attention required. As shown in fig. 17B, there are six vehicle operating modes/levels:

mode 0, where the automated system issues an alert, but there is no vehicle control.

Mode 1 (also referred to as "hands-on" mode below) is a mode in which the driver of the vehicle and the automatic system share vehicle control. For example, Adaptive Cruise Control (ACC), where the driver controls the steering function and the automated system controls the vehicle speed, is an example of mode 1 operation. In addition, features such as parking assist (where steering is automatic and speed is manual) and Lane Keeping Assist (LKA) are examples of mode 1 operation of the vehicle. In mode 1 operation, the driver must be ready to regain full control of the vehicle at any time.

Mode 2 (also referred to herein as "hands-off" mode) is an operating mode in which the automatic system assumes full control of the vehicle (acceleration, braking, and steering). The driver monitors the driving and expects to be ready to intervene immediately in the event that the automated system fails to respond properly.

Mode 3 (also referred to as "out of eye" mode) is a mode in which the driver can safely divert his/her attention away from the driving task, e.g., the driver can text or watch a movie. The vehicle handles situations that require immediate response, such as emergency braking. In this mode, the driver must still be ready to intervene within some limited time frame specified by the manufacturer when the vehicle requires to do so. When actuated by a human driver, the vehicle achieves full control of all aspects of the driving of slow moving traffic at speeds up to 60 kilometers per hour. In some instances, the vehicle is programmed to operate in this mode only on highways with physical barriers separating oncoming traffic.

Mode 4 (also referred to herein as "centrifugal" mode) is similar to mode 3, but does not even require driver attention for safety purposes, i.e., the driver can safely fall asleep or leave the driver's seat. With an embodiment, the autopilot performance is supported only in limited areas or in certain situations, such as traffic jams. Outside of these areas or situations, the vehicle must be able to safely suspend the journey, i.e. stop the vehicle if the driver does not regain control.

In mode 5 (also referred to herein as "steering wheel selectable" mode), no human intervention is required. Thus, as previously described, with one embodiment, the driving mode switching unit 130 may be configured to switch the running mode of the vehicle from one of the modes described above to the other mode.

Thus, by one embodiment, the H-A switch risk determiner may determine the risk associated with switching from a human driver mode to all levels of the automatic mode shown in FIG. 17B. It must be appreciated that switching to a lower level of vehicle automation may require the vehicle automation system to perform a smaller number of tasks than switching to a fully automated level. However, a lower level of automatic control may require a greater level of driver vigilance.

Turning to fig. 18, an illustrative flow chart outlining an exemplary process performed by an (H-a) switching pattern risk determiner is shown in accordance with an embodiment of the present disclosure. The process begins at step 1810, where (H-a) the switching pattern risk determiner receives real-time information including current driver state, intrinsic and extrinsic properties of the vehicle, and real-time vehicle data.

The process then moves to step 1820, where (H-a) the switch mode risk determiner infers a set of tasks to be performed by the vehicle when switching the vehicle operation mode to the automatic mode, based on the automatic takeover task model. Additionally, in step 1820, a corresponding set of tasks to be performed by the vehicle driver may also be estimated.

In step 1830, the response time, i.e., the amount of time required to complete the presumed set of tasks, is estimated. It must be understood that the response time may be estimated based on a vehicle model. Additionally, in step 1840, the H-A switch risk determiner may determine the urgency of performing a switch to an automatic mode of operating the vehicle. Such a determination may be made based on the extremely high risk associated with operating the human mode of the vehicle, which may be determined based on the current state of the vehicle (e.g., the driver is unconscious). Additionally, in step 1840, the H-A switch risk determiner may determine a risk associated with switching the vehicle to the automatic operating mode.

Based on the determined risk associated with switching the vehicle to the autonomous mode (in step 1840) that the criteria are not violated, the process generates an H-a switch alert command signal and a control signal (in step 1860) in step 1850 to indicate to the vehicle system that the vehicle is to be switched to the autonomous mode.

Additionally, a switch alert command signal may be output (e.g., on a display panel) to display the set of presumed tasks to be performed by the vehicle driver. Details regarding the presentation of the set of tasks to the driver are described in more detail below with reference to fig. 19 and 21.

FIG. 19 illustrates an exemplary block diagram of a switching alert control unit 250 included in a driving mode switching unit (block 130 in FIG. 1) of a vehicle system, according to one embodiment of the present teachings. The switch alert control unit 250 includes an alert instruction analyzer 1910, a vehicle model 1920, a user profile 1930, an alert time determiner 1950, an alert content determiner 1940, an alert medium determiner 1970, a task/alert configuration model 1960, a plurality of multi-modal media models 1980, and a multi-modal alert instruction generator 1990.

By one embodiment, the alert directive analyzer 1910 receives an alert directive signal corresponding to a switch from automatic to human drive (A-H) mode or from human drive to automatic (H-A) mode of vehicle operation mode. Specifically, the alert instruction analyzer 1910 receives one of an A-H handoff alert instruction signal from an A-H handoff instruction generator (block 1450 of FIG. 14) and an H-A handoff alert instruction signal from an H-A handoff instruction generator (block 1760 of FIG. 17). It must be understood that the a-H switch alert command and the H-a switch alert command signal each correspond to a set of tasks to be performed in order to switch the operating mode of the vehicle. In addition, the alert instruction analyzer 1910 obtains an estimate of the time required to complete each task (referred to herein as the task duration).

The warning instruction analyzer 1910 inputs each analyzed task to a warning time determiner 1950. The alert time determiner 1950 determines the amount of time needed to perform an alert operation (referred to herein as an alert duration). In particular, the alert duration corresponds to a time period: a warning prompt is presented to the vehicle driver to ensure that the driver performs the corresponding task in due course. By one embodiment, the duration of the alert for each task is determined based on the current state of the driver. In addition, the alert duration for each task may be adjusted based on the current state of the driver. Details regarding adjusting the alert duration for each task are described below with reference to FIG. 20B.

For example, consider a task having a 30 second task duration, i.e., the vehicle driver expects to complete the task within 30 seconds. In this case, the warning time determiner 1950 may determine that a warning duration of 10 seconds is to be used to provide a warning prompt to the driver, thereby notifying the driver of the corresponding task to be performed. By one embodiment, the alert duration is a fraction of the task duration. Specifically, referring to the example above, a warning time of 10 seconds (within a time window of 30 seconds) is presented to the driver (e.g., at the beginning of the time window) to inform the driver to perform a task within the estimated window of 30 seconds. It must be understood that the warning time having an initial duration of 10 seconds may extend to, for example, 15 seconds when it is determined that the driver has not performed the assigned task.

Accordingly, as described above, the alert time determiner 1950 may determine alert durations for various tasks based on the current state of the driver. For example, consider a case where a driver is expected to perform a particular task, and the driver state analyzer (fig. 7) determines that the driver is not fully awake (e.g., yawned, drowsy, etc.). In such a case, the alert time determiner 1950 may determine that there is a longer alert duration than the driver state analyzer determines that the driver is fully alerted. It must be noted that the task duration for completing a specific task is estimated by the switching time estimator (e.g., 1720 in fig. 17) based on the driver state.

For each task, the alert time determiner 1950 inputs each task and an alert duration associated with the task to the alert medium determiner 1970 and the alert content determiner 1940. Additionally, as inputs, the alert medium determiner 1970 receives information about the vehicle model 1920, a user profile 1930 corresponding to the driver operating the vehicle, and the current status of the vehicle driver. By an embodiment, the alert medium determiner 1970 is configured to select (from a pool of the multi-medium models 1980) at least one medium to be used as a platform for providing alerts to the driver. Specifically, the warning medium determiner 1970 associates each task with the at least one medium 1980 for warning the driver to perform the corresponding task.

By one embodiment, the selected medium may be determined based on the driver's preferences. The driver's preferences may be predetermined and stored a priori within the driver profile 1930. Additionally or alternatively, the type of media selected may be determined based on the driver's condition. For example, if the driver is in a yawned state (as determined by the driver status analyzer block 710 of FIG. 7), visual media (i.e., a display) and/or audible media form (i.e., a speaker) may be used to alert the driver. Conversely, if the driver is sleeping or is extremely drowsy, the selected media form may include a combination of vibration mechanisms (i.e., tactile response) and/or other media forms. As another example, if the user/driver profile 1930 indicates that the vehicle driver is hearing impaired, the alert medium determiner 1970 may refrain from alerting the driver using an audible source. In such an arrangement, the alert medium determiner 1970 may alert the driver using a vibrating medium and/or a display medium. It must be understood that several media sources 1980 that may be used for warning media determiner 1970 are determined via the vehicle model 1920 currently being operated by the driver. It must be understood that the selectable sound source shown in fig. 19 may correspond to any non-voice sound selectable for warning purposes, such as a siren, while the audio may specifically correspond to any voice-related audible signal. With an embodiment of the present disclosure, selectable sound sources and audio sources can be independently selected and used to compose a multi-model enhanced alert.

By one embodiment, the alert content determiner 1940 determines alert content associated with each task. For example, for a display type medium, the content may correspond to a particular message to be displayed on the vehicle display panel. The displayed message serves as a warning to the driver of the vehicle. In a similar manner, the audio content may correspond to content to be played via an audio system included in the vehicle. The content related to the audio or video mode of the alert may also include decibel level (i.e., how loud the audio message should be), the frequency at which the alert is presented, the brightness of the displayed message on the panel, whether the message is displayed with an effect such as blinking, and the like.

With regard to the vibration-type media alert, the alert content determiner 1940 may determine a magnitude of the vibration. For example, if the vibration pattern alerting the driver includes providing a vibration to the steering wheel or the driver's seat, the alert content determiner may determine a magnitude of the corresponding vibration to be applied to the driver. As shown in fig. 19, alert content determiner 1940 may also use task/alert configuration model 1960 to associate various tasks with at least one media source and present alerts to the driver.

In addition, the type of media source used for alerting purposes (as determined by alert media determiner 1970) and alert content (as determined by alert content determiner 1940) are input to multimodal alert instructions generator 1990. Upon receiving input from the alert content determiner 1940 and the alert medium determiner 1970, the multimodal alert instruction generator 1990 generates a sequence of tasks to be performed by the driver.

By way of an embodiment, the multimodal alert instruction generator 1990 is also configured to generate an alert plan for the task for prompting the driver. In particular, the multimodal alert instruction generator 1990 determines a sequence in which tasks will be presented to the driver. For example, the multimodal alert instruction generator 1990 may generate a plan based on task durations of various tasks, wherein the task with the greatest task duration is presented to the driver first. Alternatively, the multimodal alert instruction generator 1990 may first present the driver with the task having the minimum task duration. In addition, the multimodal alert instructions generator 1990 may generate a task plan based on criticality scores (i.e., priorities) associated with the various tasks. The criticality score of a task corresponds to a level of urgency for performing the task. For example, when switching to an automatic mode of operating the vehicle, the task of, for example, removing the driver's foot from the accelerator pedal may be considered more important than, for example, changing lanes before switching the vehicle operating mode. In such a scenario, the multimodal alert instruction generator 1990 may generate a plan in which the task of removing the driver's foot from the pedal (i.e., the task with higher priority) is first presented to the driver. A detailed description of the execution of the multimedia alerting instruction is given below with reference to fig. 21.

FIG. 20A shows an illustrative flow chart outlining an exemplary process performed by the handover alert control unit 250 in accordance with one embodiment of the present teachings. The process begins at step 2010, where the handover alert control unit receives one of an a-H handover alert instruction and an H-a handover alert instruction.

In step 2020, the process analyzes the received switch alert command signal to obtain task durations for each task in a set of tasks (step 2030). Note that, as described earlier, each task has such an estimated time (i.e., task duration): during this time, the task must be executed. The process determines the alert duration for each task, i.e., the time required to perform the alert operation for that task, in step 2040. It must be appreciated that the duration of the alert for each task may be determined based on the current state of the driver. Additionally, the alert duration may be adjusted based on the driver state.

The process then moves to step 2050, where the switching alert control unit determines the alert content to be associated with each task. For example, as previously described with reference to fig. 19, the alert content determiner 1940 determines alert content to be associated with each task. In addition, in step 2060, the switching alert control unit selects at least one medium to be used for alerting the driver from the plurality of media sources.

Upon determining the contents and media to be associated with the respective tasks (steps 2050 and 2060), the switching alert control unit in step 2070 generates an alert plan for the task of prompting the driver. Specifically, as previously described with reference to fig. 19, the multimodal alert instruction generator 1990 of the switching alert control unit determines such a sequence: the tasks will be presented to the driver in this sequence.

Turning now to FIG. 20B, an exemplary mission plan 2800 is shown in which various media for alerting a vehicle driver to perform a mission in the plan are depicted. In accordance with an embodiment of the present teachings, for simplicity, FIG. 20B illustrates a plan that includes three tasks (task 1, task 2, and task 3, respectively). Mission plan 2800 includes a task with a magnitude of | t₁-t₀First task of task duration 2081a, having magnitude of | t₂-t₁Second task of task duration 2081b and having magnitude of | t₃-t₂A third task of task duration 2081 c. Note that with one embodiment of the present disclosure, the mission duration is estimated based on the current driver state of the vehicle.

Additionally, FIG. 20B shows at least one alert medium (from a set of four media sources 2085a, 2085B, 2085c, and 2085d) for prompting the vehicle driver to perform a corresponding task. For example, as shown in fig. 20B, medium 1(2085a) is used for warning drivingThe driver performs task 1, media 2(2085b) and 3(2085c) are used to alert the driver to perform task 2, and media 4(2085d) is used to alert the driver to perform task 4. The alert durations for the various tasks are as follows: first alert duration 2083a is associated with task 1 and has | w₀-t₀A time measurement of | where medium 1 is selected to alert the driver to perform task 1. Note that actuation of medium 1 continues for alert duration 2083a to alert the driver to perform task 1. Similarly, for task 2, the combination of media 2 and 3(2085 b and 2085c) is selected to alert the driver to perform task 2. Note that media 2 and 3 are actuated simultaneously for alert duration 2083b (i.e., with magnitude | w)₁-t₁|) to alert the driver to perform task 1. In a similar manner, a third alert duration 2083c is associated with task 3 and has a time magnitude | w₂-t₂Where medium 4 is selected to alert the driver to perform task 3.

It must be understood that alert durations 2083a to 2083c are plotted in time on the X-axis and intensity on the Y-axis. For example, the height of a particular alert duration corresponds to the intensity level of the media source used for alert purposes. For example, for audio type media, the intensity may correspond to a volume level, and for display type media, the intensity may correspond to a brightness level of a displayed message presented to the driver.

In addition, as shown with reference to task 2, a variety of media may be used to alert the driver to the corresponding task to be performed. Note that the various media used to alert the driver to perform the necessary tasks may be presented simultaneously, as shown in FIG. 20B, and/or may be presented in a sequential manner (i.e., one after the other). In addition, as previously described, the alert duration for each task may be configured based on the state of the driver. As shown in fig. 20B, arrows 2087a to 2087c and 2089a to 2089c indicate that the alert duration for each media type for alerting the driver to perform a task may be increased based on the detected driver state.

For example, if it is observed that the driver has not performed a task at the time the alert is presented, the duration of the alert may be adjusted. In a similar manner, the intensity of the type of media used to alert the driver to perform the corresponding task may also be adjusted as indicated by arrows 2089 a-2089 c. It must be appreciated that when using multiple media sources to alert the driver to perform a task (e.g., as shown in fig. 20B for task 2), the alert duration and intensity of each media type may be controlled in an independent manner. Additionally, with one embodiment, the alert duration for each task is preferably a fraction of the corresponding task duration. It must be noted that the alert duration for a particular task may have a maximum duration that is slightly less than the corresponding task duration (assuming that the vehicle driver takes a short period of time to perform the task). In addition, as previously described, if the driver of the vehicle does not perform a particular task within the estimated task duration, the vehicle control can terminate the operation of switching the vehicle mode and, instead, initiate the exception handling process.

FIG. 21 illustrates an exemplary block diagram of a multi-modal switching alert unit 140 in accordance with an embodiment of the present teachings. The multimodal switch alert unit 140 includes a multimodal alert instruction analyzer 2110, a time control model 2105, a plurality of media generators 2120, a plurality of multimodal media models 2130, a multimodal alert coordinator 2140, a multimodal alert delivery unit 2150, an alert effect monitor 2160, and an alert instruction modifier 2170.

The plurality of multimodal media models 2130 includes at least selectable sound media 2130-1, vibration media 2130-2, lights 2130-3, visual media 2130-4, enhanced visual media 2130-5, and audio media 2130-6. By one embodiment, each media contained in multimodal media model 2130 includes a corresponding media generator configured to select and actuate the corresponding media. For example, as shown in FIG. 21, the plurality of media generators 2120 includes at least a warning sound generator 2120-1, a vibration signal generator 2120-2, a warning light generator 2120-3, a visual warning signal generator 2120-4, an enhanced visual effect generator 2120-5, and an audio warning signal generator 2120-6.

As an input, the multi-model alert directive analyzer 2120 receives a multi-model alert directive, which is generated by the switching alert control unit 250 described above with reference to fig. 19. The multi-model alert instructions analyzer 2110 is configured to analyze the set of instructions, wherein each instruction may be associated with a plurality of tasks to be performed by a vehicle driver. Note that the instructions received by the multi-model alert instruction analyzer 2110 include a plan of tasks (determined by the switching alert control unit 250 of FIG. 19) that will be presented to the driver for execution.

By one embodiment, the multimodal alert instructions analyzer 2110 uses the time control model 2105 to actuate (at a particular time as determined by the schedule of the task group) at least one media generator 2120-1 to 2120-6 configured to actuate the corresponding media associated with the task. Note that a particular task may be associated with multiple media sources. For example, if the intended driver function to be performed is the driver removing his/her foot from the pedal, a corresponding alert may be presented to the driver using both visual and audio media sources.

Accordingly, for each task, the selected media is input to the multimodal alert coordinator 2140 along with the task plan. The multimodal alert coordinator 2140 coordinates the actuation of all media associated with a particular task and ensures that the respective media remains actuated for a duration corresponding to the alert duration associated with the task. In addition, the multimodal alert delivery unit 2150 executes the alert instructions by using the activated media source and presenting the alert to the driver.

By one embodiment, the multimodal switch alert unit 140 includes an alert effect monitor 2160 configured to determine whether the driver is performing (or has performed) a particular task associated with the alert when the particular alert is presented to the driver. For example, the warning effect monitor 2160 may use a plurality of sensors disposed in the vehicle to capture information about whether the driver has performed a task. For example, the warning effect monitor may use a sensor (e.g., a displacement sensor) to determine whether the driver has lifted his/her foot off the accelerator pedal, taking into account the task of the driver being alerted to remove his/her foot from the accelerator pedal.

By one embodiment, information captured by the plurality of sensors may be input to alert instruction modifier 2170. The alert instruction modifier 2170 is configured to modify/adjust parameters associated with alerts for tasks. For example, if the warning effect monitor 2160 determines that the driver has not removed his/her foot from the accelerator pedal, the instruction modifier 2170 may increase the decibel level of the audio source used to provide warning to the driver. Additionally, by one embodiment, the alert instruction modifier 2170 may use the multimodal media model 2130 to incorporate additional sources to be used to re-alert the driver to perform a task. Thus, the alert instruction modifier 2170 feeds back the update instruction to the multimodal alert instruction analyzer, which repeats the execution of the alert instruction. It must be understood that, when presenting a warning to the driver to perform the assigned task, the multimodal switch warning unit 140 monitors the driver to determine whether the task is being performed in due course. If it is observed that the driver does not perform the assigned task within the estimated task duration, the vehicle may provide feedback to the driving mode switching unit 130 (referring to fig. 1) to perform an exception handling process.

Turning to fig. 22, an illustrative flow chart is shown that outlines an exemplary process performed by the multimodal switch alert unit 140. The process starts in step 2210, wherein the multimodal switch alert unit 140 receives a multimodal alert instruction. In step 2220, the multimodal switch alert unit analyzes the multimodal alerts to obtain a plan of tasks associated with the respective instructions.

The process further proceeds to step 2230, where, based on the media selected for the respective task (i.e., selected by the alert media determiner 1970 in fig. 19), the multimodal switch alert unit 140 uses the time control model to select at least one media generator 2120-1 to 2120-6 configured to actuate the corresponding media associated with the task. In this way, an alert signal associated with the task is generated and sent to the alert coordinator of the multimodal switch alert unit 140.

In addition, the process coordinates actuation of all media associated with the particular task in step 2240 using the alert coordinator of the multimodal switch alert unit 140. Additionally, in step 2250, the generated alert is delivered to the driver.

The process then moves to step 2260, where the multimodal switching alert unit observes the effect of the alert. As previously described, the multimodal switch alert unit collects information from multiple sensors to observe whether alerts presented to a user are being followed. Additionally, the process determines in step 2270 whether the planned task is complete based on the observations of step 2260.

In response to the driver not completing the planned task, the multimodal switch alert unit 140 modifies the instructions in step 2280. In particular, the multimodal switch alert unit 140 modifies/adjusts parameters associated with task alerts. In addition, the multimodal switch alert unit 140 may modify the task by associating additional media to the alert instructions. In this way, the modified alert (of step 2280) is fed back to the multimodal switch alert unit 140 for re-presentation to the driver of the vehicle. Note that the exception handling process as described above may be initiated if a particular task is not completed within the corresponding task duration.

FIG. 23 illustrates an architecture of a mobile device 2300 that may be used to implement particular systems embodying the present teachings. Such a mobile device 2300 includes, but is not limited to, a smart phone, a tablet, a music player, a handheld game, a Global Positioning System (GPS) receiver, and a wearable computing device (e.g., glasses, a wristwatch, etc.), or in any other modality. The mobile device 2300 in this example includes: one or more Central Processing Units (CPUs) 2340; one or more Graphics Processing Units (GPUs) 2330; a memory 2360; a communication platform 2310, such as a wireless communication module; a memory 2390; one or more input/output (I/O) devices 2350; a display or projector 2320-a for vision-based presentation; and one or more multimodal interface channels 3020-b. The multi-modal channels may include auditory channels or other media channels for signaling or communication. Any other suitable components, including but not limited to a system bus or a controller (not shown), may also be included in mobile device 2300. As shown in fig. 23, a mobile operating system 2370 (e.g., iOS, Android, windows phone, etc.) and one or more applications 2380 may be loaded from memory 2390 into memory 2360 for execution by CPU 2340.

To implement the various modules, units, and functions thereof described in this disclosure, a computer hardware platform may be used as a hardware platform for one or more of the elements described herein. The hardware elements, operating system, and programming languages of such computers are conventional in nature, and it is assumed that those skilled in the art are sufficiently familiar with them to adapt these techniques to the present teachings presented herein. A computer with user interface elements may be used to implement a Personal Computer (PC) or other type of workstation or terminal device, but the computer may also operate as a server if suitably programmed. It is believed that one skilled in the art is familiar with the structure, programming, and general operation of such computer devices, and thus the drawings may be self-explanatory.

FIG. 24 illustrates an architecture of a computing device that can be used to implement particular systems that implement the present teachings. This particular system implementing the present teachings has a functional block diagram of a hardware platform that includes user interface elements. The computer may be a general purpose computer or a special purpose computer. Both of which can be used to implement a particular system for use with the present teachings. Such a computer 2400 may be used to implement any of the components or aspects of the present teachings as described herein. Although only one such computer is shown for convenience, the computer functions associated with the present teachings described herein may be implemented in a distributed fashion across several similar platforms to spread the processing load.

For example, the computer 2400 includes a COM port 2450 connected to a network connected thereto to facilitate data communications. The computer 2400 also includes a Central Processing Unit (CPU)2420 in the form of one or more processors for executing program instructions. An exemplary computer platform includes: an internal communication bus 2410; various forms of program storage and data memory, such as a disk 24170, Read Only Memory (ROM)2430 or Random Access Memory (RAM)2440, for various data files to be processed and/or communicated by a computer and possibly program instructions to be executed by a CPU. The computer 2400 also includes an I/O component 2460 that supports input/output flows in the form of different media between the computer and other components herein (e.g., interface elements 2480). An exemplary type of interface element may correspond to different types of sensors 2480-a configured on an autonomous vehicle. Another type of interface element may correspond to a display or projector 2480-b for vision-based communication. There may also be additional components for other multimodal interface channels, such as: an auditory device 2480c for audio-based communication; and/or a component 2480-d for signaling, e.g., a signal to cause a vehicle component (e.g., a vehicle seat) to vibrate, based on the communication. The computer 2400 may also receive programming and data via network communications.

Thus, embodiments of the methods of the present teachings as outlined above may be implemented in a program. Program aspects of the present technology may be viewed as an "article of manufacture" or "article of manufacture" typically in the form of executable code and/or associated data carried on or implemented in a machine-readable medium. Tangible, non-transitory "memory" type media include any or all of memory or other memory for a computer, processor, etc., or associated modules thereof, such as various semiconductor memories, tape drives, disk drives, etc., that may provide storage for software programming at any time.

All or a portion of the software may sometimes be transmitted over a network, such as the internet or various other telecommunications networks. For example, such a transfer may enable loading of software from one computer or processor into another (e.g., from a management server or search engine operator's host or other enhanced advertisement server onto a hardware platform of a computing environment or other system implementing the computing environment or similar functionality associated with the present teachings). Thus, another type of medium that can carry software elements includes optical, electrical, and electromagnetic waves, for example, used through physical interfaces between local devices, through wired and optical fixed networks, through various air links. The physical elements carrying such waves (e.g., wired or wireless links, optical links, etc.) are also considered to be media carrying software. As used herein, unless limited to a tangible "storage" medium, terms such as a computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

Thus, a machine-readable medium may take many forms, including but not limited to, tangible storage media, carrier wave media, or physical transmission media. Non-volatile storage media include any storage device, such as optical or magnetic disks, such as any computer, etc., which may be used to implement the system shown in the figures or any component thereof. Volatile storage media includes dynamic memory, such as the main memory of such computer platforms. Tangible transmission media include: coaxial cables, copper wire and fiber optics, including the wires that comprise a bus within a computer system. Carrier-wave transmission media can take the form of electrical or electromagnetic signals, or acoustic or light waves such as those generated during Radio Frequency (RF) and Infrared (IR) data communications. Common forms of computer-readable media therefore include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards, paper tape, any other physical storage medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, a link or cable carrying such a carrier wave, or any other medium from which a computer can read programming code and/or data. Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

It will be apparent to those skilled in the art that the present teachings are applicable to numerous modifications and/or enhancements. For example, although the implementation of the various components described above may be implemented in a hardware device, it may also be implemented as a software-only solution, for example installed on an existing server. Additionally, the teachings disclosed herein may be implemented as firmware, a firmware/software combination, a firmware/hardware combination, or a hardware/firmware/software combination.

While the present teachings and/or other examples have been described above, it will be appreciated that various modifications may be made thereto, and that the subject matter disclosed herein may be implemented in various forms and examples, and that the present teachings may be applied in numerous applications, only some of which have been described herein. The appended claims are intended to claim any and all such applications, modifications and variations that fall within the true scope of the present teachings.

Claims

1. A method implemented on a machine having at least one processor, memory, and a communication platform connectable to a network, for switching a vehicle from a current mode to a different mode, the method comprising:

receiving real-time data relating to a vehicle and a status of a driver in the vehicle;

determining, based on the real-time data, a set of tasks to be performed to switch the vehicle from the current mode to the different mode, according to a task model;

determining a first duration of time required to switch from the current mode to the different mode based on a first risk associated with the current mode;

for each of the set of tasks, estimating a task duration required to complete the task;

inferring a second risk of switching from the current mode to the different mode based on the required first duration and a total duration determined based on a task duration required to complete the set of tasks; and

switching from the current mode of the vehicle to the different mode in response to the second risk satisfying a criterion.

2. The method of claim 1, wherein the real-time data includes intrinsic performance parameters and extrinsic performance parameters, the intrinsic performance parameters specifying conditions inside the vehicle that limit vehicle operating performance, the extrinsic performance parameters specifying conditions outside the vehicle that limit vehicle operating performance.

3. The method of claim 1, wherein the task duration for each of the set of tasks is estimated based on a state of the driver and/or a driver profile.

4. The method of claim 3, wherein the task duration for each of the set of tasks is further inferred based on a response time profile, the response time profile including information corresponding to a time required for a human driver to complete the task, and the driver profile indicating a task duration required for the driver to complete the task.

5. The method of claim 1, wherein the criterion corresponds to a total duration of time required to complete the set of tasks being less than or equal to the first duration of time.

6. The method of claim 1, further comprising:

generating a switch alert instruction in response to the second risk meeting a criterion prior to switching based on the set of tasks to be performed and a task duration required to complete each of the set of tasks.

7. The method of claim 1, wherein the status of the driver:

presume based on information obtained from a plurality of sensors disposed within the vehicle; and is

Including at least one of the posture of the driver, the functional state of the driver, the health state of the driver, and the mental state of the driver.

8. A machine-readable medium having information stored thereon for switching a vehicle from a current mode to a different mode, wherein the information, when read by a machine, causes the machine to perform:

9. The medium of claim 8, wherein the real-time data includes intrinsic performance parameters that specify conditions inside the vehicle that limit vehicle operating performance and extrinsic performance parameters that specify conditions outside the vehicle that limit vehicle operating performance.

10. The medium of claim 8, wherein the task duration for each of the set of tasks is estimated based on a state of the driver and/or a driver profile.

11. The medium of claim 10, wherein the task duration for each of the set of tasks is further inferred based on a response time profile, the response time profile including information corresponding to a time required for a human driver to complete the task, and the driver profile indicating a task duration required for the driver to complete the task.

12. The medium of claim 8, wherein the criterion corresponds to a total duration of time required to complete the set of tasks being less than or equal to the first duration of time.

13. The medium of claim 8, wherein the information, when read by the machine, further causes the machine to perform:

14. The medium of claim 8, wherein the driver's status:

15. A system for switching a vehicle from a current mode to a different mode, the system comprising:

a handover risk estimator configured to:

receiving real-time data relating to a vehicle and a status of a driver in the vehicle; and

a response time estimator configured to:

determining a first duration of time required to switch from the current mode to the different mode based on a first risk associated with the current mode; and

a handover risk estimator configured to:

16. The system of claim 15, wherein the real-time data includes intrinsic performance parameters and extrinsic performance parameters, the intrinsic performance parameters specifying conditions inside the vehicle that limit vehicle operating performance, the extrinsic performance parameters specifying conditions outside the vehicle that limit vehicle operating performance.

17. The system of claim 15, wherein the task duration for each of the set of tasks is estimated based on a state of the driver and/or a driver profile.

18. The system of claim 17, wherein the task duration for each of the set of tasks is further inferred based on a response time profile, the response time profile including information corresponding to a time required for a human driver to complete the task, and the driver profile indicating a task duration required for the driver to complete the task.

19. The system of claim 15, wherein the criterion corresponds to a total duration of time required to complete the set of tasks being less than or equal to the first duration of time.

20. The system of claim 15, further comprising an alert instruction generator configured to: