EP3693943B1

EP3693943B1 - Method for assisting a person in acting in a dynamic environment and corresponding system

Info

Publication number: EP3693943B1
Application number: EP19155614.1A
Authority: EP
Inventors: Christiane WIEBEL-HERBOTH; Matti Krüger
Original assignee: Honda Research Institute Europe GmbH
Current assignee: Honda Research Institute Europe GmbH
Priority date: 2019-02-05
Filing date: 2019-02-05
Publication date: 2024-05-29
Anticipated expiration: 2039-02-05
Also published as: JP2020161117A; EP3693943A1; JP7071413B2

Description

The invention regards a method and a system for assisting a person in acting in a dynamic environment. The invention also concerns the area of human-machine interfaces, in particular for assistance systems operating in a dynamic environment, for example a traffic environment in the automotive, nautical or aviation domain.
Complexity of acting in a dynamic environment increases with the number of entities which are involved in the dynamic environment. A prominent example for a dynamic environment is a traffic environment. In many areas of the world, the traffic volume of automotive traffic is increasing. As a result, a driver of a vehicle has to cope with an increasing amount of information in order to make the best decision how to drive the vehicle. Different assistance systems that assist the driver in performing his driving task were developed. One aspect of an assistance system is that any information that is provided by a system that is capable of perceiving the environment of a person or a vehicle does not need to be perceived directly by the person or vehicle driver. Filtering information with respect to its importance is performed in many cases by such assistance systems.
A traffic situation is only one example where it is desirable to assist a person in perceiving relevant or important aspects in its environment and in filtering information. It is evident that an assistance system is also suitable for a person navigating a boat or a pilot operating an air vehicle, or, more generally, any other person that has to operate in a dynamic environment. Most assistance systems analyze a scene based on sensor data acquired by physical sensors. The assistance systems assist, for example, a vehicle driver based on their scene analysis by presenting warnings, making suggestions to the driver how to act in the current traffic situation or even by performing vehicle control actions for autonomous or semi-autonomous driving. These assistance systems often have the disadvantage that they require the driver to actively shift his or her attention from the traffic scene to a user interface of the system in order to achieve successful information transmission from the assistance system to the driver. Thus, the driver's concentration on other aspects of a traffic scene is distracted which may in turn result in a reduction of traffic safety. It would be advantageous if there was a way to provide information to a driver or any other person that extends the field of perception of the driver without necessarily requiring active shifts in attentional resources away from perception of the traffic environment. Thus, information would become available that the person otherwise could not use for deciding on how to act. Such human sensory enhancement for dynamic situations is particularly advantageous in situations where entities move relative to one another.
Known attempts communicate for example a distance to an entity and/or a direction of the entity to a person. As a consequence, the person who may not even have recognized the other entity, for example, because this entity was occluded by an object or out of the line of sight, can nevertheless react, because he is informed about the existence and proximity of such entity. This already improves environment perception of the person, in particular, when available senses of a human that are not actively needed by the person to perceive the dynamic environment are used, e.g. tactile perception. A known example for such assistance system is a blind spot surveillance system that observes an area of the environment of a vehicle, which is usually not observed actively or not observable at all by a vehicle driver who predominantly focusses on the area in front of the ego-vehicle. In case that another vehicle is close to the ego-vehicle in an unobserved area, a warning will be output to the driver. For example, vibration of a steering wheel of the ego-vehicle is used to stimulate the driver. This is a typical example how a sensory capability of a driver, which is not actively used to perceive the traffic environment can be used to provide additional information on the traffic environment, which is in turn then used by the driver for improved assessment of the entire traffic situation. In this example, the driver will be alerted of another vehicle, which is driving in the blind spot and thus, he can quickly have a look to get full knowledge of the situation of which he was previously unaware.
Unfortunately, such kind of communication is limited to informing the vehicle driver on the other vehicle's current state, for example its location, direction of travel and velocity. Having this information at hand, the vehicle driver still needs to consider the entire traffic situation, now being aware of the vehicle driving in his blind spot. The burden to judge the situation correctly and to estimate different possible developments of the actual traffic situation still lies with the driver of the ego-vehicle. US2016059855A1 discloses a method and system for post-collision manoeuvre planning. The method estimates the probability for at least two traffic objects to collide, and predicts a potential movement of the traffic objects presumably involved in the collision after the collision. This information about a potential movement of the traffic objects can be used to plan a suitable trajectory path of an evasion manoeuvre for a host-vehicle employing the method. A system detects the collision situation and applies crash situation models to predict the future dynamic properties of other traffic objects that collide or are about to collide with each other. The method and system applies specific crash models like collision models, models derived from crash simulations and tests or models of human reactions in emergency situations.
The object of the present invention is to assist a person in judging a situation in a dynamic environment by providing the person with easy to recognize information about potential future events relating to task-relevant entities.
This object is achieved by the method according to claim 1 in a first aspect of the invention and the corresponding system according to claim 10 in a second aspect of the invention.
The method according to the first aspect and the system according to the second aspect obtain information on actual states of at least two entities in a common environment of these two entities for assisting a person in assessing a dynamic environment. For obtaining such information, usually the environment of the entities is physically sensed by one or a plurality of sensors. Based on the sensor output, i.e. based on the obtained information, a first future state for each of the entities is calculated (predicted) by a processor. The first future state is a state that develops from the current, actual state under the assumption that no action is taken, i.e. that none of the entities involved changes its behavior. For a first event evolving from such steady behavior and involving the at least two entities, at least one of a time to event, a position of occurrence of the first event relative to the person or to a predetermined entity associated with the person, a direction to a current location of an entity involved in the first event and a probability of occurrence of the first event is calculated based on the predicted future states.
Additionally, in a similar way, a second future state for at least one of the entities is predicted. The second future state is predicted based on a hypothetical state of at least one of the entities, wherein a hypothetical state is defined as deviating from a current actual state of the respective entity in at least one parameter. This at least one parameter is suitable to cause another evolvement of the situation. Typical parameters that may be altered generate a hypothetical state are velocity, position, acceleration and direction. In particular at least one of a time to event for a corresponding second event involving the at least two entities, a position of occurrence of the corresponding second event relative to the person or to a predetermined entity associated with the person, a direction to a current location of an entity involved in the second event and a probability of occurrence of the second event is calculated based on the second future state. The processor then generates a signal for driving at least one actuator, wherein the signal is indicative of information which encodes at least one of the position of the first event, the time to event for the first event, the direction to a current location of an entity involved in the first event and the probability of occurrence of the first event, and further information which encodes at least one of the position of the second event, the time to event for the second event, the direction to a current location of an entity involved in the second event and the probability of occurrence of the second event. Based on the signal, the at least one actuator causes a stimulation by emitting stimuli being perceivable by the person by its perceptual capabilities.
The method and system according to the invention have the advantage, that the person who acts in the dynamic environment is informed not only about a first event that will occur if no change in the person's behavior starting from the current situation an actual states of the entities occurs , but also about a second event that might occur in case that at least one of the entities involved in the current situation will change its behavior. Such behavior change is reflected by at least one hypothetical state of an involved entity. Such hypothetical state can be generated by changing a parameter of the actual state. This approach corresponds to the consideration that are made by for example a vehicle driver when he recognizes that there is a chance that any one of the vehicles in his environment for example may change lane in order to overtake the preceding vehicle.
The term "actual" is hereinafter used to denote a predicted event or state, which is predicted under the assumption that task relevant parameters remain unchanged when calculating the prediction. The term "hypothetical" will hereinafter be used to denote the second state and second events and differs insofar, that a predicted trajectory of the predetermined entity or the person in the hypothetical future state differs from a predicted trajectory in the "actual" future state. Such possible changes may manifest itself in one or more changes in relevant parameters in the perceived environment. Such changed parameter can for example be a lane change of an entity, in particular another vehicle or the ego-vehicle in a traffic scene.
In the present invention the person operating in the dynamic environment is not only informed about existence and current state of an entity in his environment but actual and hypothetical future states of involved entities are predicted and thus, the analysis of the current state is performed to a large extent by the system.
The invention therefore supports the person in understanding potential consequences of sufficiently probable hypothetical situations in the environment. This may lead to an improved ability of the person to act appropriately in the currently perceived situation.
Moreover, since the system not only interpolates the current states of the involved entities but also predicts hypothetical events, the information provided by generating a stimulus for the person corresponds to an information from a situation estimation that otherwise would have to be made by the person itself.
The dependent claims define preferred embodiments of the present invention.
According to a preferred embodiment of the invention, the time to event is encoded such that the saliency of the stimuli is the higher the smaller the time to event is.
Additionally or alternatively, the position of occurrence is encoded such that a saliency of the stimuli is the higher the closer the position of occurrence is.
Additionally or alternatively, the probability of occurrence is encoded such that a saliency of the stimuli is the higher the higher the probability of occurrence is.
Additionally or alternatively, the position of occurrence is encoded such that a saliency of the stimuli is the higher the more distant the position of occurrence is.
For example such encoding the time to event into the information conveyed by the signal using the saliency of the stimulus provides the advantage that without own individual consideration of the situation, the person directly obtains information about an urgency of the predicted event but also the hypothetical event. This may advantageously improve his reaction to the current scene in the dynamic environment.
According to the invention, the saliency of the stimuli corresponding to hypothetical events are scaled such that their maximum saliency is equal to or smaller than the saliency of the stimuli corresponding to the actual event.
This ensures that stimuli communicating predicted actual events based on the predicted actual future state in the environment are always perceived with a higher saliency than the stimuli communicating hypothetical events predicted based on the hypothetical future state. Scaling may also be chosen according to event probability. When for instance a hypothetical event has a higher probability to occur than another relevant actual event, the saliency may also be scaled according to an event's probability. In such a case the saliency of the hypothetical event would be stronger than the saliency of the actual event. Scaling the maximum saliency corresponding to hypothetical events relative to the saliency of an actual event avoids that the high saliency of the hypothetical event masks the stimulus associated with the actual event.
According to another advantageous aspect, the signal drives a plurality of actuators, such that different stimuli can be identified to indicate different events.
This ensures that the person may differentiate between predicted actual events on the one hand and predicted hypothetical events on the other hand. This may for example be achieved by separate encodings for "actually" predicted information and hypothetical information. Different encodings may include different pitch in case of a sound output. Additionally or alternatively, a plurality of different actuators for providing different stimulations may be used. Using different actuators, each one associated with only one of the actual or hypothetical event, avoids that the person confuses information that is presented simultaneously to the person. Thus, the person who is informed about at least two events, namely one actual event and additionally a hypothetical event, will be aware which one is the predicted hypothetical or actual event. Thus, the invention enables the person to focus on the event, which is more likely to occur, because the actual event assumes that none of the involved entities changes its behavior.
According to another preferred embodiment, generating the stimulation corresponding to a hypothetical event is suppressed unless the respective hypothetical event is judged to be relevant. Modern assistance systems may generate a plurality of different signals and stimuli provided in order to assist a vehicle user. However, if all these signals and stimuli for informing the vehicle driver are output unconditioned, the driver might be overwhelmed by the amount of presented information. Filtering all the information for relevant information and simultaneously still concentrating on the traffic in the environment of the ego-vehicle is almost impossible. Consequently, an adverse effect is achieved, which is to be avoided. According to this embodiment, information is provided to the person only in case that such information is considered to be relevant.
Advantageously, a criterion for judging a hypothetical event to be relevant is set dependent on the person's preferences and/or the person's situation assessment capabilities. This ensures that only helpful information is provided to the person. This is not only an effective way to assist in assessing a dynamic situation, but also acceptance of the assistance system is improved. Assistance systems that are accepted by their users will finally contribute to improving overall traffic safety. This results from a tendency of humans to switch off a system rather than to be annoyed by unwanted or inappropriate presentation of information.
It is in particular advantageous, that a criterion for judging that the hypothetical event is relevant, is set dependent on the encountered situation. Thus, additionally or alternatively to setting the condition according to an individual preference or capability of the present person (driver), information particularly relevant for the person in the encountered situation can be provided. This allows to present information on particularly dangerous or important hypothetical events which can be identified from an analysis of the encountered situation. Similarly, the prediction of the hypothetical future states may be based on an analysis of the encountered situation. Thus, starting from a situation analysis of the currently encountered situation, the hypothetical event, which is most likely to occur can be determined and information only related to a subset of the most probable hypothetical events is given. Besides probabilities, also risks associated with different events may be used for weighting hypothetical events that are taken into consideration.
The prediction of the hypothetical future states can be based on the person's behavior and/or the person's entity operation. Taking into consideration for example a person's behavior and/or operation of the own entity by the person in the past provides an increased probability that a particular predicted future behavior will indeed occur. Thus, calculating the future states provides a high probability of occurrence for the calculated future states and accordingly the relevance of corresponding hypothetical event is very high.
Embodiments of the inventive system and method are discussed with reference to the figures, in which

Figs. 1: schematically illustrates a situation and stimuli generated for the respective situation for communicating to a person a direction of an event and simultaneously the time to event;
Fig. 2: shows a top view of a first traffic situation and stimuli for providing an ego-vehicle driver with information on an actual event and a hypothetical event;
Fig. 3: shows a top view of a second traffic situation and a plurality of modes having different stimuli for providing an ego-vehicle driver with information on an actual event and a hypothetical event;
Fig. 4: shows a top view of a third traffic situation and stimuli for providing an ego-vehicle driver with information on an actual event and a hypothetical event;
Figs. 5: schematically illustrates a further situation and stimuli generated for the respective situation for communicating to a person directions of actual and hypothetical events and simultaneously the times to event for illustrating the problem of masking;
Fig. 6: shows a top view of a plurality of similar situations for explaining different conditions for judging whether an event is relevant;
Fig. 7: shows a flowchart explaining the main method steps of the invention; and
Fig. 8: shows a block diagram of the inventive system.

Figure 1 shows a simplified two-dimensional example for a scenario with moving entities and visual representations of directional time to event (TTE) signals that have been determined for the scenario. Based on the signals a stimulation of human body is performed. In the scenario, the entity of interest relative to which directional TTEs are determined is represented by a dark circle. White circles represent other relevant entities in the environment, for example other vehicles in a traffic scenario. The entity of interest may be any entity for which a relative position of an event, in particular a collision between entities is estimated. According to a preferred embodiment one of the entities that are involved in the event is the entity of interest. The event is a collision between an ego-vehicle and one of the other traffic objects. Events not directly involving the ego-vehicle also may be regarded, for example a collision between two vehicles in front of the ego-vehicle. This may be highly relevant for the ego-vehicle driver as well, because the crashed cars may block his lane and other vehicles may break sharply. For the following description however, it is always assumed that the event involves the ego-vehicle and at least one other entity or other traffic participant, and the event is referred to as collision.
The direction of an outgoing arrow represents the moving direction of the attached entity and the arrow-length represents the velocity of movement into that direction. In the representation of the produced stimuli, the orientation of an incoming arrow represents the direction of the predicted future contact and the magnitude of the arrow represents the signal component for encoding the TTE. It should be understood in a reciprocal manner such that a long arrow represents a short TTE and therefore a signal with high saliency, a short arrow a long TTE and no arrow an infinite or above a threshold TTE.
It is to be noted, that the basis for a stimulation of a person is information included in the signal which is generated by a processor and that, after being appropriately adapted to the actuators input characteristic by a driver, is converted by the actuator into a stimulus for the person. Actuators can operate using different modalities for generating stimulation perceivable by the person. In order to avoid that only one specific stimulation is encompassed, explanation of the figures refers to the signals, but not to a specific type of actuator.
The two-dimensional example illustration of a scenario shown in figure 1 shows that a signal causing multiple stimulations may be created when collisions with multiple entities are predicted. As the signal is indicative of the times to event (or TTE), the person to which such event (here: collision) is communicated will nevertheless be aware what direction is more urgent. The representation shows five entities that move in a common environment. The dark circle moves along the same path as two white circles on the left side but their velocities differ in such a way that a collision with the preceding and the succeeding entities would occur at approximately the same time. In addition, one white circle (upper right) moves with a relatively high velocity towards a future location of the dark one, which creates another possible future collision. No collision is likely to occur between the dark and a white circle (lower right) moving on different paths and in opposite directions. From the perspective of the dark circle, collisions with multiple entities from different directions are predicted. Due to differences in relative speed, the collisions on the top and the bottom are predicted to occur at the same time. The collision with the top-right circle is predicted to occur at an earlier point in time and the corresponding information in the signal is therefore represented by a longer arrow.
It is to be noted, that figure 1 only serves for explaining the basic principle of the present invention. Thus, it is limited to explaining only the stimulus that is generated for an actual event. An actual event is an event that assumes that certain dimensions, aspects or parameters of a current state of the involved entities do not change. For example, a vehicle's trajectory or speed remain unchanged, while other parameters of the vehicle such as an absolute location (position) in the environment do change.
Given the situation depicted on the left side of figure 1 it is thus assumed that all involved vehicles will move with constant speed and without change of the direction. This can also be mentioned as continuous behavior of the respective vehicle. It is to be noted with regard to the time to event TTE, that there also exist versions of the parameter time to collision as examples for a time to event, which take an acceleration of the vehicles and therefore varying vehicle velocities into account. In contrast thereto, a lane change performed by one of the vehicles is caused by a change in behavior and could potentially lead to an entirely different traffic situation. With the present invention, the driver is even informed about such changing traffic situation. Thus, not only an estimation for actual events is performed but also for hypothetical events. This will become clearer considering the following examples.
The left side of figure 2 for example shows a top view of a road having two lanes on which vehicles drive in the same direction. Again, the arrows connected to the different vehicles illustrate direction and velocity of movement of the vehicle. The ego-vehicle E is driving on the right lane. It follows currently on the right lane a vehicle A that drives slower than the ego-vehicle E. On the left lane, still behind the ego-vehicle E, a further vehicle B drives, but with a velocity which is significantly higher than the velocity of the ego-vehicle E. Having in mind the estimation of only actual events, no event involving the vehicle B on the left lane and the ego-vehicle E will be predicted, estimated and signaled to the ego-vehicle operator. The only actual event that would evolve if anyone of the vehicles shown in the traffic situation of figure 2 maintained its direction and speed regards the ego-vehicle E and its leading vehicle A. Thus, the system as explained with reference to figure 2 would predict the collision between the ego-vehicle E and its leading vehicle A.
On the right side of figure 2, the corresponding signal is shown as a solid arrow pointing to the front of the ego-vehicle E. As can be seen in the portion explaining the signals an additional signal is generated which is depicted as a dashed arrow that is directed to the rear left of the ego-vehicle E. The signal corresponds to a hypothetical event which can be predicted by adapting the parameter settings of the scenario thereby creating a hypothetical scenario. Such adaptation of the parameter settings to hypothetical parameter settings of the scenario is made in addition to predicting actual events.
The hypothetical scenario which is underlying the predicted hypothetical event, which results in the dashed arrow, would for example occur if the ego-vehicle E would move to the left lane. Thus, the hypothetical situation is a consequence of a change in behavior of the ego-vehicle E. But such a change of behavior may also be predicted, because it is evident that the ego-vehicle E is faster than its predecessor. There are systems in the market that can predict such change in behavior and thus will predict with certain probability that the ego-vehicle E will change its lane. Starting from such a potential hypothetical lane change, the system will then estimate for this hypothetical behavior respective time to event and/or position of the event. Thus, the dashed arrow shows the point of collision with respect to the ego-vehicle E but also encodes by its length with the time to hypothetical event.
Figure 2 discusses the embodiment with reference to a point of collision as example for an event. Another example would be a predicted event, wherein the event involves another entity, for example another vehicle. The other vehicle then represents a source for the predicted event. The other vehicle or source is located in a certain direction from the ego-vehicle E. In this embodiment, a signal provided to the person operating the ego-vehicle E may also include information encoding a direction to the source or vehicle, with which the ego-vehicle E would collide if the hypothetical event would occur.
In order to distinguish between the different events it is preferred that the stimuli for the actual event and for the hypothetical event are output using different actuators. For example, one actuator or set of actuators could be arranged in the seat belt for outputting the stimulation encoding direction and urgency of an actual event. Additionally, a further actuator or set of actuators could be provided in the steering wheel in order to provide information on a hypothetical event. Using the steering wheel for providing information on a hypothetical event is particularly preferred as the steering wheel is more closely associated with an action. The driver will directly identify a hypothetical event and his/her driving action, i.e., the driver will understand the message: "If you turn the wheel left, this is going to happen". Using the steering wheel and the seat belt are only examples and other objects like, for example, a seat, arm rests or the like may be considered. It is to be noted, that there are a plurality of different ways for providing information using a person's capability of sensing its environment. Consequently, the term "actuator" includes every means for outputting information in such a way that a person may perceive it. Thus, output stimulations may be for instance visual, tactile, auditory, olfactory, vestibular or the like. In the example, which is directed to a traffic situation, the time to event, in particular the time to collision, shall be presented to the ego-vehicle driver. The saliency of the stimulation is inversely proportional to the time to event TTE. It is now easier for the ego-vehicle's driver to notice the more urgent predicted events and respond thereto. Thus, it is facilitated that the ego-vehicle's driver shows a natural response to a more urgent event. As will be explained later, such saliency of the stimulus, sometimes also referred to as saliency of the signal, may be adapted for the hypothetical event, so that the actual event, which has the higher probability to occur can always be recognized by the ego-vehicle driver.
It is noted in this context, that the saliency of the stimulus may be influenced by an intensity of the stimulus perceivable by the driver of the ego-vehicle E. Additionally or alternatively, the saliency of the stimulus may also be influenced by other modalities of the stimulus, for example a frequency of the stimulus.
Another example for a traffic scenario illustrated in top view is shown in figure 3. The ego-vehicle E drives with a velocity higher than the velocity of its predecessor and will consequently collide with the predecessor A at a certain point in time in the future. On the other side, a further vehicle B drives on the left lane with the speed similar to the predecessor's speed. The actual event would thus be a collision between the ego-vehicle E and its predecessor A on the right lane. The hypothetical event, assuming a lane change of the ego-vehicle E from the right lane to the left lane, would be a collision between the ego-vehicle E and, in the actual scenario, the vehicle B driving on the left lane. Dashed lines in figure 3 represent the outlines of the ego-vehicle E with an altered hypothetical geometry, such that it extends to the left lane as well. Since the vehicle B on the left lane drives at the same speed but a small distance ahead of the ego-vehicle's predecessor A, this hypothetical event is predicted to occur slightly later than the predicted actual event.
Figure 3 shows four different possibilities for informing the driver. All four opportunities have in common that the actual event, which corresponds to the signal indicated as solid arrow, is output with a certain saliency indicating the time to the actual event and at a position, which corresponds to the position of collision of the ego-vehicle E and its predecessor A. Alternatively, the arrow may point towards a current direction of the source or origin of the actual event. Thus, the solid arrow points towards the front of the ego-vehicle E in all four cases, which are marked as a), b), c) and d).
Figures a) and b) show the point of collision of the hypothetical event at the front left of the ego-vehicle E. This is one possible way to indicate that, having in mind the current position of the ego-vehicle E, the vehicle B that is also involved in the hypothetical collision currently is left of the ego-vehicle E. Alternatively, the dashed arrow's could also point to the front of the ego-vehicle E and thus the output stimulus would indicate the same position as for the actual event, The reason is, that the hypothetical event only occurs, if the ego-vehicle E would change its lane to the left lane, as it is indicated by the dashed line silhouette of the ego-vehicle E in the scenario at the left side of figure 3.
Further, as shown in a) and d) the dashed arrow and consequently the saliency of the output stimulus is only slightly shorter than the solid arrow indicating the time to collision of the actual event. The time to collision of the actual event and the hypothetical event thus are encoded using the same scale for the signal saliency. Alternatively, the signal saliency of the hypothetical event is scaled so that the resulting saliency is reduced. This avoids an unwanted masking of the information on the actual event, which may be considered as having a higher occurrence probability. Figure 3 illustrates this by reducing the length of the dashed arrow. In some scenarios or embodiments, the hypothetical event may even have a higher occurrence probability than the predicted actual event. This may apply in cases where a particular traffic situation and possibly some knowledge available in the assistance system about the driver's situation awareness. These aspects may indeed create a situation in which the driver of the ego-vehicle E carries out an action that would generate the hypothetical situation with a high occurrence probability.
According to a preferred embodiment the saliency of the signal corresponding to a hypothetical event will always be scaled in such a way that the maximum saliency of the hypothetical event is always smaller than the saliency for an actual event. The saliency for an actual event may be defined when setting up the system or can be adjusted by the individual driver of the ego-vehicle E. If referring to a time to collision as an example for a time to event TTE, a threshold can be defined for the time interval such that a collision that might occur at a point in time lying further in the future will not be notified to the ego-vehicle driver at all.
Various options for encoding and communicating the direction to such hypothetical collision objects are possible.
As options a) and b) in figure 3 show, the ego-vehicle driver may be informed about a hypothetical event by stimulating the driver corresponding to the position that indicates the direction of the collision relative to the actual location of the ego-vehicle E. Contrary, options c) and d) show stimulating the driver at positions corresponding to the direction of the collision relative to the hypothetical location of the ego-vehicle E or relative to the hypothetical vehicle geometry in figure 3. Additionally or alternatively, the position of the stimulation and direction of the event could be indicated through a map that may contain coordinates for locations outside the actual vehicle boundaries. The map of one embodiment is egocentric with respect to the ego-vehicle E or the driver of the ego-vehicle E. It is to be noted that the term map can here also refer to non-visual maps such as a tactile or auditory maps where certain areas could be mapped to outside coordinates as well as multimodal maps which may be read through multiple senses.
Figure 4 shows another example for the road having two lanes on which three vehicles, including the ego-vehicle E, currently drive. Again, there are two vehicles A, B in front of the ego-vehicle, both driving slower than the ego-vehicle E but at similar speed. The actual situation corresponds to the situation of figure 3. But in figure 3 the hypothetical situation is generated by extending the geometry of the ego-vehicle E such that it extends to the left lane as well. Contrary, figure 4 adds its possible lane changing behavior of the ego-vehicle E, indicated by the dashed vehicle outlines showing the change in direction. The signal is similar as in figure 3a), which means that it indicates the collision as originating from the top left and thus relative to the actual current ego-vehicle location. The signal may be interpreted as, "if you would enter the left lane, you would collide with the vehicle located on the front left relative to your current position in x seconds given current estimates of situation dynamics".
Figure 5 shows a further situation involving the ego-vehicle, or the entity of interest to be more general, and five other entities that surround the ego entity. Again, movement direction and velocity are indicated by the solid arrows to represent the current states of the entities. Further, the dashed arrows in the scenario shown at the left side of figure 5 show the hypothetical movement directions of the ego entity. It is obvious, that the actual situation would lead to a collision between the black circle and the white one directly above it in the drawing. But also the hypothetical development of the situation would lead to collisions with either the white circle at the right side above the ego circle but also with the white circle to the lower left of the ego circle. Due to respective velocities of the entities, the time to collision will vary to a large extent. This is illustrated in option a). Here it can be seen, that the most urgent collision would happen if the ego entity moved to the right side, because the white circle at the right side of the ego circle is rather close and significantly slower than the ego entity. Consequently, the signal indicating this hypothetical collision has a strong saliency as indicated in option a). On the other hand, the collision between the ego entity and the white circle on the lower left side will happen next with respect to a timely order of all the events, actual and hypothetical. Thus, the saliency of the respective signal communicating the second hypothetical event to the lower left of the ego-vehicle would be weaker than for the first hypothetical event but still stronger than the signal which refers to the predicted actual event which is indicated by the solid arrow. In such a case there is a risk that the person who is assisted by the inventive system will in fact recognize the more salient stimulation indicated by the upper right arrow. But this stimulation corresponds to a hypothetical event. Unfortunately, it might happen, that the person well perceives the indication of the hypothetical event but due to the strong saliency of the stimulation, he might be unaware of the indication of the actual event. The stimulation corresponding to the actual event might be masked.
In order to avoid such masking and as indicated above already, the saliency that corresponds to hypothetical events may be scaled such that no masking with indications of actual events can occur. The saliencies might for example be scaled relative to the solid arrow or following a different absolute (possibly even binary) scale which allows the evidence-based and for some embodiments arguably more relevant signal component (solid arrows) to be most clearly perceivable. It is to be noted, that the simultaneous evaluation and communication of information for different hypothetical settings may in some scenarios increase the need for having multiple separately identifiable encodings (e.g. moving left vs. moving right).
The topic of relevance itself may further be specifically addressed on broader scope for some embodiments. When for instance continuously deriving and communicating information from a hypothetical presence on neighboring lanes while driving, any vehicle which passes the ego-vehicle or is passed by the ego-vehicle would elicit the signal. Some drivers may appreciate this input but in environments in which vehicle passing occurs regularly and without an immediate associated danger, it may often be considered irrelevant and also annoy or even distract a user. Thus, there is a need to avoid such distraction of the user in order to ensure that the user will not switch off the entire system. It is thus preferred to consider relevance of an event and to output stimuli only in case that the corresponding event is considered to be relevant for the assisted person, in most cases the driver of the ego-vehicle.
One way to avoid that the ego-vehicle driver is annoyed by output information is to output such information only in case that the ego-vehicle driver requests for such information. Thus, outputting information to the driver is suppressed unless it is requested by the driver. Suppressing output of information may be well done after generating the corresponding signal so that simply the corresponding actuator, which generates the stimulus for the ego-vehicle driver is switched off. Such a driver request may for instance occur in the form of speech interaction, bodily or ocular gestures or even the use of any other user input device. In vehicles for instance the handle used for activation of the indicator lights could co-serve as a trigger for providing hypothetical time to collision information from the respective neighboring Lane.
But reducing the output information to only relevant events may not only be caused by the use of the system but could be automatically achieved by analyzing the current situation. This means, that for situations in which the information about hypothetical situations could be especially useful the communication of the information is automatically activated.
One indicator that providing information on hypothetical events is reasonable is when an actual event is determined which fulfills the condition for being output to the ego-vehicle driver. In case that such event is the collision, this occurs when the time to collision falls below a certain threshold. To improve understanding of this approach, figure 6 shows three different scenarios. In the situation depicted in scenario a) on the left, the ego-vehicle E is not on a collision path with its predecessor A, because both vehicles are driving with the same velocity. Consequently, a lane change of the ego-vehicle E does not need to be assumed and the vehicle B coming from behind on the left lane but at a higher velocity may simply pass. Here indicating the hypothetical event of the collision between the vehicle B on the left lane and the ego-vehicle E would rather annoy the ego-vehicle driver than add any benefit.
The situation is different in the scenario b) shown in the middle of figure 6. Here, the ego-vehicle E drives with a velocity that is higher than the velocity of its predecessor A. Again, on the left lane there is a vehicle B coming from behind driving even faster than the ego-vehicle E. In this situation, a time to event is estimated for a collision between the ego-vehicle E and its predecessor A. If this time to collision falls below a certain threshold, the condition for informing the driver of the upcoming actual event is fulfilled and the signal will cause an output stimulating the driver indicating its collision at the front side of the ego-vehicle E and the time to collision. In this case, the driver could consider a lane change and thus the hypothetical situation in which the ego-vehicle E changed lane may lead to a hypothetical collision between the ego-vehicle E and the vehicle B on the left lane. Since this hypothetical development of the situation has a certain probability, the respective collision may be indicated to the ego-vehicle driver as indicated in the lower part of figure 6.
A third example is illustrated on the right side of figure 6, where the ego-vehicle E has a successor C driving at a higher velocity than the ego-vehicle E and again another vehicle B is driving on the left lane with an even higher velocity. Thus, here again, an actual event is estimated, which is a collision between the ego-vehicle E and its successor C and since the time to collision falls below the threshold for outputting the information to the ego-vehicle driver, also the hypothetical event of a collision between the vehicle B on the left lane of the ego-vehicle E is output for the driver.
The discussed scenarios are only simple examples showing how the relevance for a certain event may be determined. Generally, a personalized model of user behavior, experience or preferences could tune decisions about the relevance of the situation. An experienced driver may for instance be more likely to initiate an overtaking maneuver in the kind of the situation described in the middle of figure 6, than a beginner or very cautious drivers which would make the presentation of signals - based on hypothetical lane change - more relevant for the experienced driver. For a novice driver on the other hand, information about such hypothetical situations could be overwhelming especially when presented in addition to the information provided about the actual scene and can thus be imagined to not only be momentarily irrelevant but even distracting in certain embodiments.
The opposite effect to the described case may also be feasible. For example, a novice in driving may benefit from information about hypothetical scenarios involving the hypothetical events and therefore experience a learning effect. The experienced driver may appreciate receiving a momentary feedback on the actual situation.
Besides a rather persistent preference- or skill-based system modulation, also more momentary intention estimations may be used to influence relevance estimations in some situations. The intent to perform a lane change might for instance be identified based on certain behavioral patterns such as prolonged or more frequent glances between the lanes and rearview mirrors. The occurrence of such behavior patterns which indicate or strongly correlate (possibly in a personalized manner) with a lane change intent could serve as a strong relevance indicator
Now and with reference to figure 7 the main method steps shall be explained that lead to outputting stimulations for the driver indicating actual events as well as hypothetical events. At first, dynamic environment data is obtained as indicated in step S1. Such data collection is performed for each time step. A step size may differ between embodiments and is influenced, in particular limited, by hardware capabilities and requirements.
The data may for example be collected using one or a plurality of sensors that are mounted on the vehicle for which assistance shall be provided. The signal measurement may be supplemented or substituted by other ways of information reception, for example transferring information on states of the other traffic participants by a car to car communication system or other external measurement and communication resources. Signal measurement or reception is performed in step S2. Based on the received actual environment data, a spatiotemporal state estimation is performed in step S3. The spatiotemporal state estimation defines the current state of the respective entity, for example a current location, current trajectory, current velocity of the entity.
In step S4 the situation parameters which are derived from information on the environment of the ego-vehicle are altered in order to generate one or multiple hypothetical situation(s). As it was already explained with reference to the drawings, this may be achieved by simulating geometry changes of the involved entities but also by assuming moving direction and/or velocity variations for the involved entities. Then, in steps S5 and S5'it is determined whether an event or a hypothetical event occurs. Such an event is determined to happen if a target state deviation can be recognized. Such target state deviation occurs when the predicted state of an involved entity deviates from states that are considered to be desirable. This is the case for example if the time to collision falls below a certain threshold. Thus, by defining conditions for which developments of the situations can be accepted, a deviation from a target state can be identified.
If such deviation is recognized for the actual situation and/or the hypothetical situation, it is checked in steps S6 and S6'whether the relevance criteria are met. If such relevance criteria are met, a signal is generated by the system's processor based on which an actuator generates a stimulus which is perceivable by the user. The signal generation is performed in step S7 and output of the respective stimulation is done in step S8.
On the other hand, if either no deviation from the target state can be recognized in step S5 or S5', the respective evidence-based or hypothetical signal iteration is terminated. This is also true in case that the relevance criteria, which are analyzed in step S6 and S6', are not met.
Figure 8 illustrates the system with its main components. Sensors 1 - 4 physically sense entities in a dynamic scenario in which entities may move relative to one another, repeatedly. In this example, TTE estimation may be achieved by incorporating information from a variety of resources.

Data from radar, cameras and/or laser scanners as examples for sensors 1-4 and built in- or onto a car are filtered for features that identify relevant entities and used to infer locations and distances.
Integration of distances and locations of entities over multiple samples may be used to infer current relative velocities.
In combination with information about the velocity and acceleration and geometry of the ego-vehicle, as well as topographic information such as about road curvature and slope obtained from an available map or based on online measurements, predictions about future collisions of the ego vehicle with other entities may be made.

This sensing is the basis for determining a relative position and velocity, relative to the person who is assisted by the system and from the sensed values information on states of the entities (direction, velocity) is derived for every repetition. This information is stored in a memory 6 and is the basis for behavior prediction and trajectory estimation. For each measurement-iteration, individual trajectories and relative velocities of the involved entities are estimated in a processor 5. The estimates provide the basis for inferences or predictions about possible future contact between the entity or entities of interest and other relevant entities in the environment. Also, additional information that may be available and relevant for a scenario may be incorporated when making such estimates. The time to event (TTE) estimation is performed also by processor 5. The algorithms for predicting a future behavior (estimating a future trajectory) of the entities (including the ego-vehicle) is known in the art and thus details thereof are omitted. For the prediction procedure, probability distributions over different TTEs could be generated for each direction in which potentially relevant entities are identified. Such distributions would be advantageous in that they preserve the uncertainties associated with the available information and may be suitable for the application of signal selection criteria.
The signals are adapted by a driver 7 that is configured suitably to drive the actuator 8. Signal generation encodes a direction of a predicted event and TTE such that one or a plurality of actuator elements of actuator 8 are driven to stimulate the person at a location of its body indicative of the direction where the event will occur and with an perceived saliency indicative of the TTE. Alternatively, the one or the plurality of actuator elements of the actuator 8 are driven to stimulate the person at a location of its body indicative of the direction towards the momentary location of the event related entity of interest and with an perceived saliency indicative of the TTE.
For the assistance system preferably the tactile sense of the driver or assisted person is used as a channel for signal transmission. Communication is realized in the form of an array of tactile actuators 8 (e.g. vibrotactile) which is arranged for example around the driver's torso and which is capable of simultaneously varying the perceived stimulus locations, frequencies and amplitudes. Using this interface, the direction towards a relevant entity with a TTE below a certain threshold corresponds to the location of the driver's torso, which is oriented towards this direction. For each such direction, additionally the TTE is encoded in the vibration frequency such that the frequency approaches the assumed optimal excitation frequency for human lamellar corpuscles with shortening of the TTE, which has the advantage of coupling stimulus detectability with situation urgency. Furthermore, the encoding in frequency has high potential for personalization because stimulus amplitude and frequency range could be adapted to the driver's preferences and sensitivity, which lowers the risk of creating annoying or undetectable signals.
To form the actuator 8 as mentioned in figure 8, the actuator 8 comprises a plurality of actuator elements that are attached to the user's seat belt and embedded in the area of the user's seat that is in contact with his or her lower back. This setup would have the advantage that a user would not need to be bothered with putting on additional equipment which increases the probability of actual usage in places where seat belts are common or even a legal requirement. Alternatively, the actuators could also be embedded in a belt, jacket or another piece of clothing that can be extended with an arrangement of tactile actuator elements around the waist of the wearer. To account for different body shapes, the placement and/or the control of the actuators would have to be adapted such that the perceived signal location always corresponds to the correct direction with respect to the spatial frame of reference of the body. In the case of using a seat belt, the mapping of actuator directions could for instance be a function of the belt's length around the waist. In addition the use of an actuator-array with sufficient spatial resolution, the exploitation of vibrotactile illusions or a combination of both could aid in achieving this personalization.
It is to be noted that for the sake of simplicity of the description and for an easy understanding of the invention, the above given explanations of the system in figure 8, for example the actuator 8, do not distinguish between predicted actual events and predicted hypothetical events. Of course, the actuator 8 is used for indicating actual events as well as hypothetical events and it is a design measure which of the actuator elements is dedicated to which of the event types in an embodiment.

Claims

Method for assisting a person in assessing a dynamic environment, comprising the steps of:
- obtaining information on actual states of at least two entities in a common environment (S2),

- predicting a first future state for each of the entities based on the obtained information (S3),

- estimating at least one of a time to event for a first event involving the at least two entities, a position of occurrence of the first event relative to the person or to a predetermined entity associated with the person, a direction to a current location of an entity involved in the first event, and a probability of occurrence of the first event based on the predicted first future states (S5), and

- additionally predicting a second future state for the person and/or the predetermined entity (S4, S3),

- estimating at least one of a time to event for a second event involving the at least two entities, a position of occurrence of the second event relative to the person or to the predetermined entity, a direction to a current location of an entity involved in the second event, and a probability of occurrence of the second event (S5'), and

- generating a signal (S7) for driving at least one actuator,

wherein the signal is indicative of an information, which encodes at least one of the position of the first event, the time to event for the first event, the probability of occurrence of the first event, and a further information, which encodes at least one of the position of the second event, the time to event for the second event, the direction to the current location of the entity involved in the second event and the probability of occurrence of the second event,

the signal causing a stimulation by the at least one actuator (8) emitting stimuli being perceivable by the person by its perceptual capabilities (S8), and

the method comprises altering (S4) at least one parameter that describes an actual state of the respective entity for generating a hypothetical state of at least one of the entities,

wherein predicting the second future state is based on the generated hypothetical state of the at least one of the entities, and

the step of estimating at least one of a time to event for the second event involving the at least two entities, a position of occurrence of the second event relative to the person or to the predetermined entity, a direction to a current location of an entity involved in the second event, and a probability of occurrence of the second event, is performed based on the predicted second future state that is generated based on the hypothetical state of the at least one of the entities (S5'),
characterized in that

saliency of the stimuli corresponding to the second event are scaled such that its maximum saliency is equal to or below the saliency of the stimuli corresponding to the first event.
Method according to claim 1,
characterized in that
the time to event is encoded such that the saliency of the stimuli is the higher the smaller the time to event is, and/or

the position of occurrence is encoded such that the saliency of the stimuli is the higher the closer the position of occurrence is, and/or

the probability of occurrence is encoded such that the saliency of the stimuli is the higher the higher the probability of occurrence is.
Method according to claim 1,
characterized in that
the position of occurrence is encoded such that the saliency of the stimuli is the higher the more distant the position of occurrence is.
Method according to any one of claims 1 to 3,
characterized in that
the signal drives a plurality of actuators (8) such that different stimuli can be identified to indicate different events.
Method according to any one of the preceding claims,
characterized in that
the method comprises steps of determining a relevance of the second event based on analyzing relevance criteria, and

terminating emitting the stimuli corresponding to the second event in case the second event is determined to be not relevant (S6, S6').
Method according to claim 5,
characterized in that
the method comprises setting the relevance criterion dependent on preferences of the person and/or situation assessment capabilities of the person.
Method according to claim 5 or 6,
characterized in that
setting the relevance criterion includes setting the relevance criterion dependent on a determined situation.
Method according to claims 7,
characterized in that
the method comprises analyzing the determined situation, and

the step of predicting the second future state is based on the analysis of the determined situation.
Method according to any one of the preceding claims,
characterized in that
the method comprises a step of determining a behavior of the person and/or an operation of an own entity by the person, and

the step of predicting the second future state is based on the determined behavior of the person and/or the determined operation of an own entity by the person.
System for assisting a person in assessing a dynamic environment, the system comprising:
at least one sensor (1-4) for obtaining information on actual states of at least two entities in a common environment,

a processor (5) for

calculating a prediction for a first future state for each of the entities based on the obtained information, and at least one of

an estimation of a time to event for a first event involving the at least two entities, and/or

a position of occurrence of the first event relative to the person or to a predetermined entity associated with the person based on the predicted first future states,

a direction to a current location of an entity involved in the first event, and

a probability of occurrence of the first event, and for

additionally calculating a prediction for a second future state for at least one of the entities, and at least one of an estimation of at least one of a time to event for the second event involving the at least two entities, a position of occurrence of the second event relative to the person or to a predetermined entity associated with the person, a direction to a current location of an entity involved in the second event and a probability of occurrence of the second event, based on the second future state, and for

generating a signal for driving at least one actuator (8), wherein the signal is indicative of an information which encodes at least one of the position of the first event, the time to event for the first event, the direction to the current location of the entity involved in the first event and the probability of occurrence of the first event, and indicative of further information, which encodes at least one of the position of the second event, the time to event for the second event, the direction to the current location of the entity involved in the second event and the probability of occurrence of the second event,

the least one actuator (8) is configured to output, based on the signal, a stimulation by the at least one actuator emitting stimuli being perceivable by the person by its perceptual capabilities,
and

the processor (5) is configured to alter at least one parameter describing an actual state of the respective entity for generating a hypothetical state of at least one of the entities,

to predict the second future state based on the generated hypothetical state of the at least one of the entities, and

to estimate at least one of a time to event for the second event involving the at least two entities, a position of occurrence of the second event relative to the person or to the predetermined entity, a direction to a current location of an entity involved in the second event, and a probability of occurrence of the second event, based on the predicted second future state based on the hypothetical state of the at least one of the entities,
characterized in that

saliency of the stimuli corresponding to the second event are scaled such that its maximum saliency is equal to or below the saliency of the stimuli corresponding to the first event.