KR20240021695A - Virtual fields driving related operations - Google Patents

Abstract

A method of communicating with a driver of a vehicle, the method comprising: (i) obtaining object information regarding one or more objects located within the environment of the vehicle; (ii) analyzing the object information; (iii) using one or more neural networks (NN), based on the object information, determining virtual forces that are associated with a physical model and that represent the influence of one or more objects on the behavior of the vehicle; and (iv) determining, based at least on the virtual forces, force feedback for use in providing a physical restraint to be applied in connection with driver movements made by the driver using the vehicle.

Description

VIRTUAL FIELDS DRIVING RELATED OPERATIONS}

This application is a continuation-in-part of U.S. Patent Application No. 17/823,069, filed Aug. 29, 2022, which claims priority from U.S. Provisional Application No. 63/260,839, which is incorporated herein by reference. This application claims priority from U.S. Provisional Application No. 63/368,155, filed July 11, 2022, the entire contents of which are hereby incorporated by reference.

The present invention relates to a method for communicating with a driver of a vehicle.

Autonomous vehicles (AVs) can help significantly reduce the number of traffic accidents and _CO2 emissions, as well as contribute to more efficient transport systems. However, today's candidate AV technologies cannot scale in three ways:

Limited field of view, lighting and weather issues, and occlusion all lead to detection errors and noisy localization/kinematics. To deal with this poor real-world perceptual output, one approach for AV technology is to invest in expensive sensors and/or integrate specialized infrastructure into road networks. However, these efforts are expensive and geographically limited in terms of infrastructure, so they cannot lead to generally accessible AV technology.

AV technology, which is not based on expensive hardware and infrastructure, relies entirely on machine learning and thus relies on data to process real-world situations. In order to handle detection errors and learn good and sufficient driving policies for complex driving tasks, a huge amount of data and computational resources are required, but there are still edge cases that are not handled accurately. The common denominator in these extreme cases is that machine learning models do not generalize well to invisible or confusing situations, and the black box nature of deep neural networks makes it difficult to analyze incorrect behavior.

Currently, road-ready autonomous driving is implemented in the form of separate ADAS functions such as ACC, AEB, and LCA. Reaching fully autonomous driving requires seamlessly combining existing ADAS functions and adding more of these functions (e.g. lane changes, intersection handling, etc.) to fill the gaps that are currently not automated. In short, current autonomous driving is not based on a holistic approach that can be easily expanded to implement fully autonomous driving.

Accordingly, the present invention provides a method for communicating with a driver of a vehicle, comprising: obtaining object information regarding one or more objects located within the environment of the vehicle; analyzing the object information; determining, by using one or more neural networks (NN) and based on the object information, virtual forces that are associated with a physical model and that represent the influence of one or more objects on the operation of the vehicle; and determining, based at least on the virtual forces, force feedback for use in providing a physical resistance force to be applied in connection with driver actions made by a driver using the vehicle.

The present invention also provides a non-transitory computer-readable medium for communicating with a driver of a vehicle, the non-transitory computer-readable medium storing instructions for: being located within the environment of the vehicle; Obtain object information about one or more objects that have been identified; Analyzing the object information; determine, by using one or more neural networks (NN) and based on the object information, virtual forces that are associated with a physical model and represent the influence of one or more objects on the motion of the vehicle; and determining, at least based on the virtual forces, force feedback for use in providing a physical resistance force to be applied in connection with driver actions made by the driver using the vehicle.

Embodiments of the disclosure will be more fully understood and appreciated from the following detailed description taken in conjunction with the drawings:
Figure 1 shows one embodiment of the method.
Figure 2 shows one embodiment of the method.
Figure 3 shows one embodiment of the method.
Figure 4 shows one embodiment of the method.
5 shows an embodiment of a vehicle.
Figures 6-9 illustrate embodiments of context and perception fields.
Figure 10 shows one embodiment of the method.
Figure 11 shows one embodiment of a scene.

In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures and components are not described in detail so as not to obscure the invention.

The subject matter considered to be the invention is specifically pointed out and clearly asserted in the conclusion of the detailed description. However, the objects, features and advantages of the present invention, together with its construction and method of operation, may be best understood by reference to the following detailed description when read in conjunction with the accompanying drawings.

For simplicity and clarity of explanation, elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to others for clarity. Additionally, where deemed appropriate, reference numerals may be repeated among the drawings to indicate corresponding or similar elements.

Since the illustrated embodiments of the present invention can be implemented using mostly electrical components and circuits known to those skilled in the art, the details are provided to facilitate understanding and appreciation of the underlying concepts of the present invention and to avoid confusing the teachings of the present invention. In order not to be confusing or distracting, no more will be explained than is deemed necessary as aforesaid.

Any reference in the detailed description of the method shall apply mutatis mutandis to a device or system capable of executing the method and/or a non-transitory computer-readable medium storing instructions for executing the method.

Any reference in the detailed description to a system or device should apply mutatis mutandis to a method executable by the system and/or may apply mutatis mutandis to a non-transitory computer-readable medium storing instructions executable by the system.

Any reference in the detailed description to a non-transitory computer-readable medium shall apply mutatis mutandis to a device or system capable of executing instructions stored on the non-transitory computer-readable medium and/or may apply mutatis mutandis to a method for executing the instructions. there is.

Any combination of modules or units listed in any of the drawings, detailed description, and/or claims may be provided.

Any one of the units and/or modules illustrated in the application may be implemented in hardware and/or code, and instructions and/or instructions stored in a non-transitory computer-readable medium may be stored in a vehicle, outside the vehicle, mobile device, server, etc. may be included.

The vehicle may be any type of vehicle, including ground transport vehicles, air vehicles, and watercraft.

This detailed description and/or drawings may refer to images. An image is an example of a media unit. Any reference to an image may apply mutatis mutandis to a media unit. A media unit may be an example of a Sensed Information Unit (SIU). Any reference to a media unit may include any type of natural signal, such as a signal generated by nature, a signal representing human behavior, a signal representing operations related to the stock market, a medical signal, a financial series, or a geodetic signal. , may be applied mutatis mutandis to signals representing operations related to geophysics, chemistry, molecules, text and numeric signals, time series, etc., but is not limited thereto. Any reference to a media unit may apply mutatis mutandis to a sensed information unit (SIU). SIUs can be of any type and include visible light cameras, audio sensors, sensors capable of detecting infrared, radar images, sensors capable of detecting ultrasound, electro-optics, radiography, LIDAR (light detection and ranging), and thermal sensors. , can be sensed by any type, such as passive sensors, active sensors, etc. Sensing may involve generating samples (e.g., pixels, audio signals) that represent the transmitted signal or otherwise reaching the sensor. An SIU may be one or more images, one or more video clips, text information about one or more images, text describing kinematic information about an object, etc.

Object information may include, but is not limited to, any type of information related to an object, such as the location of the object, the motion of the object, the speed of the object, the acceleration of the object, the direction of propagation of the object, the type of object, one or more dimensions of the object, etc. It doesn't work. Object information may be raw SIU, processed SIU, text information, information derived from SIU, etc.

Obtaining object information includes receiving object information, generating object information, participating in processing of object information, processing only part of object information, and/or receiving only other parts of object information. It can be included.

Obtaining object information may include object detection or may be performed without performing object detection.

Processing of object information may include at least one of object detection, noise reduction, signal-to-noise ratio improvement, and definition of a bounding box.

Object information may be received from one or more sources, such as one or more sensors, one or more communication units, one or more memory units, one or more image processors, etc.

Object information can be shared in one or more ways - for example in an absolute way (for example - giving coordinates of the object's position) or in a relative way - for example in relation to a vehicle (for example - an object is stored at a certain distance in relation to the vehicle). (at a certain angle) can be provided.

The vehicle is also called an ego-vehicle.

The detailed description and/or drawings may represent a processor or processing circuitry. A processor may be a processing circuit. The processing circuitry may be implemented as a central processing unit (CPU) and/or one or more other integrated circuits, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, etc., or a combination of these integrated circuits. there is.

Any combination of any steps of any method shown in the description and/or drawings may be provided.

Any combination of any of the subject matter of the claims may be provided.

Any combination of systems, units, components, processors, and sensors illustrated in the description and/or drawings may be provided.

Any reference to an object can be applied to the pattern. Therefore, any reference to object detection can be applied mutatis mutandis to pattern detection.

Successful driving depends on circumventing surrounding road objects based on their positions and movements, but humans are not good at estimating their kinematics. The inventor suspects that humans circumvent the need for kinematic estimation by using internal representations of surrounding objects in the form of virtual force fields that immediately imply action. Consider a scenario where your vehicle is driving in one lane and a vehicle diagonally ahead in an adjacent lane begins to swerve into your lane. The human response to braking or turning is immediate and instinctive and can be experienced as a virtual force pushing the ego away from the turning vehicle. These virtual force representations are learned and associated with specific road objects.

Inspired by the above considerations, the present inventor proposes a novel concept of perception field. The perceptual field is a learned representation of road objects in the form of a virtual force field that is “sensed” through the ego vehicle's control system in the form of ADAS and/or AV software. In the present invention, a field is defined as a mathematical function that varies depending on spatial location (or similar quantity).

One embodiment of inference method 100 is shown in Figure 1 and includes:

Method 100 may be executed for each one or more frames of the vehicle environment.

Step 110 of method 100 may include detecting and/or tracking one or more objects (eg, including one or more road users). Detection and/or tracking may be performed in any manner. The one or more objects may be any object that can affect the operation of the vehicle. For example - road users (pedestrians, other vehicles), the road and/or path on which the vehicle is traveling (e.g. the condition of the road or path, the shape of the road - e.g. curved, straight road segments), traffic signs. , traffic lights, road crossings, schools, kindergartens, etc. Step 110 may include obtaining additional information, such as kinematic and contextual variables related to one or more objects. Acquiring can include receiving or generating. Acquisition may include processing one or more frames to generate kinematic and contextual variables.

It should be noted that step 110 may include acquiring kinematic variables (without acquiring one or more frames).

Method 100 may also include step 120 of obtaining respective perceptual fields associated with one or more objects. Step 120 may include determining which mapping between objects should be retrieved and/or used, etc.

Step 110 (and even step 120) includes step 130 determining one or more virtual forces associated with one or more objects by passing relevant input variables such as perceptual fields (and one or more pseudo-physical model functions), kinematic and contextual variables. It can follow.

Step 130 may be followed by step 140 of determining a total virtual force applied to the vehicle based on one or more virtual forces associated with one or more objects. For example, step 140 may include performing a vector weighted sum (or other function) on one or more virtual forces associated with one or more objects.

Step 140 may be followed by step 150 of determining, based on the total virtual force, a desired (or target) virtual acceleration - for example, based on an equivalent of Newton's second law. The desired virtual acceleration may be vector - or otherwise have a direction.

Step 150 may be followed by step 160 of converting the desired virtual acceleration into one or more vehicle driving operations that will cause the vehicle to propagate along the desired virtual acceleration.

For example, step 160 may include translating a desired acceleration to acceleration or deceleration or changing the direction of travel of the vehicle using gas pedal movement, brake pedal movement, and/or steering wheel angle. The translation may be based on a dynamic model of the vehicle with a specific control scheme.

Advantages of perceptual fields include, for example, explainability, generalizability, and robustness to noisy input.

Explainability. Representing ego movements as configurations of individual perceptual fields means decomposing behavior into its more fundamental components and is itself an important step towards explainability. The possibility to visualize these fields and apply intuitions from physics to predict ego movements represents additional explanatory potential as compared to typical end-to-end black-box deep learning approaches. This increased transparency also allows passengers and drivers to trust AV or ADAS technology more.

Generalizability. Representing one's reaction to an unknown road object as an aversive virtual force field constitutes an inductive bias in an unseen situation. There is a potential advantage to this representation in that it allows edge cases to be handled in a safe manner with little training. Additionally, perceptual field models are holistic in the sense that the same approach can be used for all aspects of driving policy. It can also be divided into narrow driving functions used in ADAS such as ACC, AEB, LCA, etc. Finally, the complex nature of the perceptual field allows the model to be trained on atomic scenarios and still be able to adequately handle more complex scenarios.

Robustness to noisy inputs: Physical constraints on the time evolution of the perceptual field in combination with potential filtering of the input can better handle noise in the input data compared to pure filtering of localization and kinematic data.

Physical or virtual forces allow for mathematical formulation, for example in terms of second-order ordinary differential equations involving so-called dynamical systems. The advantage of representing the control policy in this way is that it is sensitive to the intuitions of dynamical systems theory and that it is simple to integrate external modules such as prediction, exploration and filtering of inputs/outputs.

Another advantage of the perceptual field approach is that it does not rely on any specific hardware and is not computationally more expensive than existing methods.

training process

The process of learning a perceptual field can be one or a combination of two types: behavioral replication (BC) and reinforcement learning (RL). BC approximates a control policy by fitting a neural network to observed human state-action pairs, whereas RL involves learning through trial and error in a simulated environment without reference to expert demonstration.

We can combine the learning algorithms of these two classes by first learning a policy through BC to use it as an initial policy to be fine-tuned using RL. Another way to combine the two approaches is to first learn a so-called reward function (used in RL) through behavioral replication to infer what constitutes desirable behavior for humans, and later use regular RL to Training is done through trial and error. This latter approach goes by the name of inverse RL (IRL).

Figure 2 is an example of a training method 200 used for learning through BC.

Method 200 may begin at step 210 by collecting human data that is considered an expert demonstration of how to handle a scenario.

Step 210 may be followed by step 220 of constructing a loss function that finds the difference between the kinematic variables resulting from the perceptual field model and the corresponding kinematic variables of the human demonstration.

Step 220 is followed by step 230 of updating the parameters of the perceptual field and the auxiliary function (which may be a cyber-physical model function different from the perceptual field) to minimize the loss function by some optimization algorithm such as gradient descent. You can.

Figure 3 is an example of a training method 250 used for reinforcement learning.

Method 250 may begin at step 260 of building a realistic simulation environment.

Step 260 may be followed by step 270, where the reward function is constructed by learning from an expert demonstration or manual design.

Step 270 may be followed by step 280 of executing the episode in a simulation environment and continuing to update the parameters of the perceptual field and auxiliary function to maximize the expected cumulative reward by some algorithm such as proximity policy optimization.

Figure 4 shows one embodiment of method 400.

Method 400 may be for perceptual field navigation related operations.

Method 400 may begin by initializing step 410.

Initialization step 410 may include receiving a group of NNs trained to execute step 440 of method 400.

Alternatively, step 410 may include training a group of NNs to perform step 440 of method 400.

Various embodiments of training NN groups are provided below.

● A group of NNs can be trained to map object information to one or more virtual forces using behavioral cloning.

● A group of NNs can be trained to map object information to one or more virtual forces using reinforcement learning.

● A group of NNs can be trained to map object information to one or more virtual forces using a combination of reinforcement learning and action replication.

● A group of NNs can be trained to map object information to one or more virtual forces using reinforcement learning with a reward function defined using behavioral replication.

● A group of NNs can be trained to map object information to one or more virtual forces using reinforcement learning with an initial policy defined using behavioral replication.

● A group of NNs may be trained to map object information to one or more virtual force and perceptual fields and one or more different virtual physical model functions.

● The NN group may include a first NN and a second NN, where the first NN is trained to map object information to one or more perceptual fields and the second NN is trained to map object information to one or more virtual physical model functions. trained.

The initialization step 410 may lead to a step 420 of obtaining object information about one or more objects located within the vehicle's environment. Step 410 may be repeated multiple times - and the following steps may be repeated multiple times. Object information may include video, images, audio, or other sensed information.

Step 420 may be followed by step 440, which determines one or more virtual forces applied to the vehicle, using one or more neural networks (NNs).

One or more NNs may be an entire NN group (from initialization step 410) or may be part of a NN group, leaving one or more unselected NNs in the group.

One or more virtual forces represent one or more influences of one or more objects on the motion of the vehicle. The impact may be a future impact or a present impact. Impacts can change the vehicle's progress.

One or more virtual forces belong to a virtual physical model. A virtual physical model is a virtual model that can virtually apply physical rules (eg, mechanical rules, electromagnetic rules, optical rules) for vehicles and/or objects.

Step 440 may include at least one of the following steps:

● Calculating the total virtual force acting on the vehicle, based on one or more virtual forces acting on the vehicle.

● Determining the desired virtual acceleration of the vehicle based on the total virtual acceleration exerted on the vehicle by the total virtual force. The desired virtual acceleration may be the same as or different from the total virtual acceleration.

Method 400 may also include at least one of steps 431, 432, 433, 434, 435, and 436.

Step 431 may include determining the status of the vehicle based on the object information.

Step 431 may be followed by step 432, which selects one or more NNs based on the situation.

Additionally or alternatively, step 431 may be followed by step 433 of supplying context metadata to one or more NNs.

Step 434 may include detecting the class of each of one or more objects based on the object information.

Step 434 may be followed by step 435 of selecting one or more NNs based on the class of at least one of the one or more objects.

Additionally or alternatively, step 434 may be followed by step 436 of supplying one or more NNs with class metadata representing a class of at least one of the one or more objects.

Step 440 may be followed by step 450, which performs one or more driving-related operations of the vehicle based on one or more virtual forces.

Step 450 may be executed without human driver intervention and may include changing the speed and/or acceleration and/or direction of travel of the vehicle. This may include performing autonomous driving or advanced driver assistance systems (ADAS), which may include temporarily controlling the vehicle and/or one or more drive-related units of the vehicle. This may include setting the vehicle's acceleration to a desired virtual acceleration, without human driver intervention.

Step 440 may include suggesting to the driver to set the acceleration of the vehicle to a desired virtual acceleration.

5 shows an embodiment of a vehicle. The vehicle includes one or more sensing units 501, one or more driving-related units (e.g., autonomous driving units, ADAS units, etc.), a processor 560 configured to execute any method, and storing instructions and/or method results, functions, etc. It may include a memory unit 508 and a communication unit 504 to do this.

Figure 6 shows embodiments of a method 600 for suboptimal centering RL with suboptimal sample points as input. Lane sample points are located within the vehicle's environment.

RL assumes a simulation environment in which an agent (self-vehicle) generates input data that can implement its learned policies (perceptual fields).

Method 600 may begin by step 610 detecting the nearest lane or side of the road for road sample points (X _L,i ,Y _L,i ) and (X _R,i ,Y _R,i ), where L is the left, R is the right, and index i is the sample point. The speed of the ego vehicle (formerly called vehicle) is denoted by V _ego .

After step ₆₁₀ , the left lane input vectors (X _{L,i ,} Y _{L,i )} and _{V ego are centered} in A focusing step 620 may follow.

Step 620 may be followed by step 630 that calculates the suboptimal perceptual fields f _θ (X _L ) and f _θ (X _R ). This is performed by one or more NNs.

Step 630 may be followed by step 640 of constructing a differential equation describing the ego acceleration applied to the ego vehicle: a = f _θ (X _L ) + f _θ (X _R ).

This may be the output of an inference process. Step 640 may be followed by step 450 (not shown).

The method may include updating one or more NNs. In these cases, RL can assume a reward function that is learned based on expert demonstrations or created by hand. In the embodiment of Figure 6 the reward function may be incremented for every timestamp that the subject vehicle remains in its lane.

The update may include step 670 of implementing in a simulation environment and the RL learning algorithm records what happens at the next time step, including the earned rewards.

Step 670 may include using a particular RL algorithm (e.g., PPO, SAC, TTD3) to sequentially update the network parameter θ to maximize the average reward.

Figure 7 shows a method 700 for multi-object RL with visual input.

Step 710 of method 700 includes receiving a sequence of panoramicly segmented images over a short time window from the ego perspective (image acquired by the ego _vehicle ), the relative distances to individual objects can do.

Step 710 may be followed by step 720, which captures high-level spatio-temporal features (X _i ) by applying a spatio-temporal CNN to individual instances (objects).

Step 720 may be followed by step 730 of calculating the individual perceptual fields f _θ (X _i ,i) and the sum Σf _θ (X _rel,I ,X _i ,i).

Step 730 may be followed by step 740 of constructing a differential equation describing the ego acceleration applied to the ego vehicle: a = Σf _θ (X _rel,I ,X _i ,i).

This may be the output of an inference process. Step 740 may be followed by step 450 (not shown).

The method may include updating one or more network parameters (θ) using some RL process.

The method may include step 760 of implementing in a simulation environment the RL learning algorithm recording what happens at the next time step, including the rewards obtained.

RL can assume reward functions that are learned based on expert demonstrations or created by hand.

Step 760 may be followed by step 770, which sequentially updates the network parameters θ to maximize the average reward using a specific RL algorithm such as PPO, SAC, or TTD3.

Figure 8 shows a method 800 for multi-object BC with kinematic input.

Step 810 of method 800 may include receiving a list of detected object relative kinematics (X _rel,i ,V _rel,i ), where X _rel,i is the detected object i relative to the ego vehicle. is the relative position of and V _rel,i is the relative velocity of the detected object i with respect to the host vehicle. It also receives ego vehicle speed V _ego .

Step 810 may be followed by step 820 of calculating the perceptual field f _θ (X _rel,i ,V _rel,i ,V _ego ,i) for each object.

Step 820 may be followed by step 830, which sums contributions from individual perceptual fields. Step 830 may also include normalizing the magnitude of the resulting 2d vector so that it is equal to the highest magnitude of the individual terms: N*Σf _θ (X _rel,i ,V _rel,i ,V _ego ,i).

Step 830 may be followed by step 840 of constructing a differential equation describing the ego acceleration applied to the ego vehicle: a = N*Σf _θ (X _rel,i ,V _rel,i ,V _ego ,i).

This may be the result of an inference process. Step 840 may be followed by step 450 (not shown).

The method may include updating one or more network parameters.

The method initial conditions If (t;x ₀ ,v ₀ ) is provided, a step 860 of calculating the ego trajectory may be included.

Following step 860, loss function = Σ( Step 870 may follow to calculate (t;x ₀ ,v ₀ ))-x(t;x ₀ ,v ₀ )) ² . This propagates the loss accordingly.

Figure 9 shows an inference method 900 that adds a loss function to an adaptive cruise control model implemented with kinematic variables as input.

Step 910 of method 900 determines the position of the ego vehicle ₍ X _ego ), the velocity of the ego vehicle (V _ego ), the position of the nearest vehicle in front of the ego vehicle ( It may include receiving Vd _CIPV ).

Step 910 may be followed by step 920 of calculating the relative position X _rel = X _ego - X _CIPV and the relative velocity V _rel = V _ego - V _CIPV .

Step 920 may be followed by step 930:

● Calculating the perceptual field function g _θ (X _rel ,V _CIPV ) by the first NN.

● Calculating the auxiliary function h _ø (V _rel ) by the second NN.

● Multiplying g _θ (X _rel ,V _CIPV ) by h _ø (V _rel ) to give the target acceleration (equal to the target force).

This may be the output of an inference process. Step 930 may be followed by step 450 (not shown).

The method may include updating one or more NN parameters.

The method initial conditions If (t;x ₀ ,v ₀ ) is provided, a step 960 of calculating the ego trajectory may be included.

Following step 960, loss function = Σ( Step 970 may follow to calculate (t;x ₀ ,v ₀ ))-x(t;x ₀ ,v ₀ )) ² . This propagates the loss accordingly.

force feedback

Human drivers (hereafter referred to as drivers) can steer the vehicle in problematic ways. The driver may drive the vehicle in violation of one or more traffic rules and/or drive the vehicle in a reckless manner and/or drive the vehicle in a dangerous manner and/or drive the vehicle in any other inappropriate manner.

Examples of problematic driving include failing to maintain a safe distance from other road users, ignoring one or more road users, driving in an aggressive manner, bypassing other road users when a right-of-way is prohibited, and This may include introducing unnecessary changes in direction and/or speed of propagation.

A method of communicating with a driver of a vehicle is provided, and in particular a method of providing a restraining force to cause the driver to drive in a correct and appropriate manner.

Figure 10 shows one embodiment of a method 3000 for communicating with a driver.

According to one embodiment, method 3000 begins at step 3010 with obtaining object information regarding one or more objects located within the environment of the vehicle.

According to one embodiment, step 3010 also includes analyzing object information. Analysis may include determining location information and/or motion information of one or more objects. The location information and motion information may include the relative position of one or more objects (relative to the vehicle) and/or the relative movement of one or more objects (relative to the vehicle).

According to one embodiment, step 3010 is followed by step 3020, which uses one or more neural networks (NNs) and, based on object information, determines virtual forces associated with a physical model and represents the influence of the one or more objects on the motion of the vehicle. Follow. Virtual forces are applied virtually to the vehicle.

According to one embodiment, step 3020 includes step 3020-1 of determining one or more virtual fields based on object information, and step 3020-2 of determining a virtual force based on the one or more virtual fields.

Step 3020 may be driven from a cyber-physical model. For example, assuming that the virtual physical model represents an object as an electromagnetic charge, one or more virtual fields are virtual electromagnetic fields and a virtual force represents the electromagnetic force produced by the virtual charges. For example, assuming the virtual physical model is a mechanical model, the virtual force field is driven from the acceleration of the object.

According to one embodiment, step 3020 is followed by step 3030 of determining, based at least on virtual forces, force feedback for use in providing a physical resistance force to be applied in connection with driver actions made by the driver using the vehicle.

Step 3030 may be executed based on assumptions regarding the relationship between the vehicle's virtual force and the desired virtual acceleration. For example, the virtual force may have a virtual acceleration (virtually applied to the vehicle) and the desired virtual acceleration of the vehicle may correspond to the virtual acceleration (virtually applied to the vehicle).

According to one embodiment, step 3020 is followed by step 3025 of determining a desired virtual acceleration of the vehicle based on the virtual acceleration. Step 3025 is followed by step 3030.

According to one embodiment, the desired virtual acceleration may have the same magnitude as the virtually applied acceleration but may be directed in the opposite direction.

According to one embodiment, the desired virtual acceleration has a magnitude different from the magnitude of the virtually applied acceleration.

According to one embodiment, the desired virtual acceleration has a direction other than the direction of the virtually applied acceleration.

According to one embodiment, step 3030 includes at least one of the following:

● Calculation of the difference between the desired virtual acceleration of the vehicle and the acceleration to be set under the driver's driving operation control.

● Determine force feedback by setting a physical resistance force to reduce the difference.

● Determination of one or more parameters of force feedback. One or more parameters may include magnitude and/or duration and/or manner of providing force feedback. For example, the resistive force may be continuous or discontinuous, and may be of a fixed magnitude or may vary in magnitude over time.

● Determine the effect of force feedback on the effort required by the driver to control the vehicle's driving. If the resistance force is strong, the driver needs to put in more effort to overcome the resistance force.

● Determine the effect of force feedback on the effort required by the driver to turn the vehicle's steering wheel.

● Determine the effect of force feedback on the effort required by the driver to change the vehicle's speed or acceleration.

● Determining the resistance to limit the impact of driver actions on the car once the resistance is applied. For example, the driver may need to rotate the steering wheel more times for the vehicle to make the same directional change.

● Determining the resistance to prevent the effects of driver actions on the vehicle once the resistance is applied.

Determining force feedback by setting a physical resistance force to reduce the difference may include applying an arbitrary mapping between the value of the difference and one or more parameters of the resistance force.

● For example, a larger difference can trigger the application of a stronger resistance force.

● The relationship between the size of the difference and the size of the resistance may be a linear relationship, a non-linear relationship, etc.

● In another embodiment the duration and/or duty cycle associated with the application of the resistive force may be varied as a function of the difference.

According to one embodiment, step 3030 is followed by step 3040 in response to the determination of force feedback.

According to one embodiment, step 3040 includes at least one of the following:

● Providing force feedback in response to driver actions made by the driver using the vehicle. Providing may include applying a resistive force. Applying a resistive force can include mechanical movement, changes in the friction of mechanical elements, changes in the rate of mechanical transmission, etc.

● Directing the vehicle's control unit to provide force feedback in response to driver actions performed by the driver using the vehicle. The control unit may control one or more units of the vehicle, such as a steering wheel, accelerator pedal, brake pedal, throttle linkage, etc.

● Instructing the vehicle's control unit to provide force feedback in response to driver actions made by the driver using the vehicle.

● Triggers provision of force feedback upon identification of driver actions.

● Vehicle Control – Transferring control from the driver to the autonomous driving unit.

Figure 11 shows one embodiment of a scene.

11 shows one embodiment of a vehicle 1031 located within a segment 1011-1 of a first roadway.

Pedestrian 1022 begins to cross the segment in front of vehicle 1301. The pedestrian is represented by a virtual field of the pedestrian pushing the vehicle 1031 (shown as virtual equipotential field line 1022').

Figure 11 also shows the desired acceleration 1042 of the vehicle.

Assume that the vehicle moves parallel to the edge of the road before receiving the hypothetical force.

Under this assumption, the current steering position 1050 must be changed and the steering wheel rotated (by the necessary rotation of the steering wheel 1051) to steer the vehicle in the desired direction 1043 parallel to the desired direction of acceleration.

Figure 11 also shows three examples of unwanted differences between the vehicle's desired direction 1043 set by the driver and the vehicle's unwanted propagation direction. The undesired direction is indicated by a dashed line marked “Driver Set Direction” (1045).

In these three embodiments a resistive force 1044 is provided to reduce the difference and is set to cause the vehicle to propagate according to the desired direction 1043.

In the foregoing detailed description of the invention, the invention has been described with reference to specific examples of embodiments of the invention. However, it will be apparent that various modifications and changes may be made without departing from the scope and broader spirit of the invention as asserted in the appended claims. Additionally, in the description of the invention and the patent claims, terms such as “front,” “rear,” “top,” “bottom,” “top,” and “bottom” are used for descriptive purposes and necessarily describe permanent relative positions. It's not meant to be done. It is to be understood that the terms so used are interchangeable in appropriate contexts so that the embodiments of the invention described herein may operate in different directions, for example, than those illustrated or otherwise described herein. Additionally, the terms "assert" or "set" and "negation" (or "deassert" or "clear") are used herein to refer to a signal, status bit, or similar device as logically true or logically true. It is used to refer to rendering each in a false state. A logically true state is logic level 1, and a logically false state is logic level 0. And if a logically true state is logic level 0, then a logically false state is logic level 1. Those skilled in the art will recognize that the boundaries between logical blocks are merely examples and that alternative embodiments may merge logical blocks or circuit elements or impose alternative decompositions of functionality on various logical blocks or circuit elements. Accordingly, it will be appreciated that the architecture depicted herein is merely illustrative and that many other architectures may be implemented in practice that achieve the same functionality. Any arrangement of components to achieve the same function is effectively “related” so that the desired function is achieved. Accordingly, any two components combined in the present invention to achieve a particular function may be viewed as “related” to each other such that the desired function is achieved, regardless of architecture or intermediate components. Likewise, any two such related components may be viewed as being “operably connected” or “operably coupled” to each other to achieve the desired functionality. Additionally, those skilled in the art will recognize that the boundaries between the foregoing operations are illustrative only. Multiple operations may be combined into a single operation, a single operation may be distributed into additional operations and the operations may be executed at least partially temporally overlapping. Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may vary in various other embodiments. Also for example, in one embodiment, the illustrated embodiments may be implemented on a single integrated circuit or as a circuit located within the same device. Alternatively, embodiments may be implemented as any number of separate integrated circuits or individual devices interconnected with each other in any suitable manner. However, other modifications, variations and alternatives are also possible. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. In the claims, any reference signs between parentheses should not be construed as limiting the scope of the claims. The word 'comprising' does not exclude the presence of elements or steps other than those listed in the claims. Additionally, the term “one” or “one” used in this specification is defined as one or more than one. Additionally, the use of introductory phrases such as "at least one" and "one or more" in a claim should not be construed as limiting a particular claim, nor should the introduction of other claim elements by the indefinite article "one" or "an". , for inventions that contain only one such element in a claim containing such introduced claim element, if the same claim contains the preface "one or more" or "at least one" and an indefinite article such as "an" or "one" The same is true in this case. The same goes for the use of the definite article. Unless otherwise specified, terms such as “first” and “second” are used to optionally distinguish between the elements described by such terms. Accordingly, these terms are not necessarily intended to indicate the temporal or other priority of these elements. The mere fact that certain measures are recited in different claims does not indicate that a combination of these measures cannot be used to advantage. Although certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Accordingly, the appended claims are to be construed as including all such modifications and changes as fall within the true spirit of the invention. It is understood that various features of embodiments of the disclosure that are described in the context of separate embodiments for the sake of clarity may also be provided in combination in a single embodiment. Conversely, various features of embodiments of the disclosure that are described in the context of a single embodiment for the sake of brevity may also be provided individually or in any suitable sub-combination. Those skilled in the art will understand that the embodiments of the invention are not limited by what has been specifically shown and described above. Rather, the scope of embodiments of the invention is defined by the appended claims and their equivalents.

Claims (19)

As a method for communicating with the driver of a vehicle,
Obtaining object information regarding one or more objects located within the environment of the vehicle;
analyzing the object information;
determining, by using one or more neural networks (NN) and based on the object information, virtual forces that are associated with a physical model and that represent the influence of one or more objects on the operation of the vehicle; and
A method comprising determining, based at least on virtual forces, force feedback for use in providing a physical resistance force to be applied in connection with driver actions made by a driver using the vehicle. The method of claim 1, comprising determining one or more virtual fields based on the analysis of the object information, wherein determining the virtual force is based on the one or more virtual fields. The method of claim 1 including providing force feedback in response to driver actions made by a driver using the vehicle. The method of claim 1 including instructing a control unit of the vehicle to provide force feedback in response to driver actions made by a driver using the vehicle. The method of claim 1 including triggering provision of force feedback upon identifying driver action. The method of claim 1, wherein determining the force feedback includes determining the effect of the force feedback on the effort required by the driver to control the operation of the vehicle. The method of claim 1, wherein determining the force feedback includes determining the effect of the force feedback on the effort required by the driver to turn the steering wheel of the vehicle. The method of claim 1, wherein determining the force feedback includes determining the effect of the force feedback on the effort required by the driver to change the speed or acceleration of the vehicle. The method of claim 1, including calculating a desired virtual acceleration of the vehicle based on the virtual force. 10. A method according to claim 9, comprising calculating the difference between the acceleration and the desired virtual acceleration of the vehicle to be set under control of the driver's driving operation. 11. The method of claim 10, wherein determining the force feedback includes setting the physical resistance force to reduce the difference. 2. The method of claim 1, comprising determining a desired path of the vehicle based on one or more iterations of determination of virtual forces. 13. The method of claim 12, wherein determining force feedback includes setting a physical resistance force to reduce the difference between the path indicated by the driver and the desired path. 2. The method of claim 1, wherein once said resistance force is applied, the effect of driver actions on the vehicle is limited. 2. The method of claim 1, wherein the resistance force, once applied, prevents the effects of driver actions on the vehicle. 2. The method of claim 1, comprising performing the travel-related operation. 1. A non-transitory computer-readable medium for communicating with a driver of a vehicle, the non-transitory computer-readable medium storing instructions for:
Obtain object information regarding one or more objects located within the vehicle's environment;
Analyzing the object information;
determine, by using one or more neural networks (NN) and based on the object information, virtual forces that are associated with a physical model and represent the influence of one or more objects on the motion of the vehicle; and
Determining, based at least on the virtual forces, force feedback for use in providing a physical resistance force to be applied in connection with driver actions made by the driver using the vehicle. 18. The non-transitory computer-readable medium of claim 17, storing instructions comprising determining one or more virtual fields based on analysis of the object information, and wherein the determination of the virtual force is based on the one or more virtual fields. 19. The method of claim 18, based on the virtual force, calculating a desired virtual acceleration of the vehicle, calculating a difference between the acceleration and a desired virtual acceleration of the vehicle to be set under control of the driver's driving operation, and A non-transitory computer-readable medium in which the determination includes establishing a physical resistance force to reduce the difference.