JP2024009794A - Virtual fields driving related operations - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method for bringing full autonomous driving.

SOLUTION: A method for communicating with a driver of a vehicle, includes: i. obtaining object information regarding one or more objects located within an environment of the vehicle; ii. analyzing the object information; iii. determining, by using one or more neural network NNs, and based on the object information, a virtual force associated with a physical model and representing an impact of the one or more objects on a behavior of the vehicle; and iv. determining, based on at least the virtual force, a force feedback for use in providing a physical restraining force to be applied in association with a driver action made by the driver using the vehicle.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

This application is part of U.S. Patent Application No. 17/823,069, filed Aug. 29, 2022, which claims priority from U.S. Provisional Patent Application No. 63/260,839, which is incorporated herein by reference. This is a continuation application. This application claims priority from U.S. Provisional Patent No. 63/368,156, filed July 11, 2022, which is incorporated herein in its entirety.

Autonomous vehicles (AV) can help significantly reduce the number of traffic accidents and _CO2 emissions and contribute to a more efficient transportation system. However, today's candidate AV technologies are not scalable in three respects:

Limited field of view, lighting and weather challenges, and occlusion all lead to detection errors and noisy localization/kinematics. To address such insufficient real-world perceptual output, one approach to AV technology is to invest in expensive sensors and/or integrate specialized infrastructure into the road network. However, such efforts are very costly and, in the case of infrastructure, geographically limited and therefore cannot lead to a generally accessible AV technology.

AV technology, which is not based on expensive hardware and infrastructure, relies entirely on machine learning and thus data to process real-world situations. Dealing with detection errors and learning driving policies sufficient for complex driving tasks requires vast amounts of data and computational resources, and there are still edge cases that are not properly handled. A common feature in these edge cases is that machine learning models do not generalize well to unseen or confusing situations, and due to the black-box nature of deep neural networks, it is difficult to analyze incorrect behavior. It is difficult to do so.

Current road-enabled autonomous driving is implemented in the form of separate ADAS functions such as ACC, AEB, and LCA. To reach full autonomy, we need to seamlessly combine existing ADAS functions together and bridge any gaps that are not currently automated by adding more such functions (e.g. lane changes, intersection handling, etc.). also needs to be covered. In short, current autonomous driving is not based on a holistic approach that can be easily expanded to bring about full autonomy.

Embodiments of the present disclosure will be more fully understood and appreciated from the following detailed description, taken in conjunction with the drawings.

FIG. 2 is a diagram illustrating an example of the method. FIG. 2 is a diagram illustrating an example of the method. FIG. 2 is a diagram illustrating an example of the method. FIG. 2 is a diagram illustrating an example of the method. It is a figure which is an example of a vehicle. FIG. 3 is a diagram showing an example of a situation and a perceptual field. FIG. 3 is a diagram showing an example of a situation and a perceptual field. FIG. 3 is a diagram showing an example of a situation and a perceptual field. FIG. 3 is a diagram showing an example of a situation and a perceptual field. FIG. 2 is a diagram illustrating an example of the method. FIG. 3 is a diagram showing an example of a scene.

In the detailed description that follows, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to objects, features, and advantages, as well as arrangements and methods of operation, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings.

It will be appreciated that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Furthermore, where considered appropriate, reference signs may be repeated between the figures to indicate corresponding or similar elements.

Since the illustrative embodiments of the invention may be implemented to a large extent using electronic components and circuits known to those skilled in the art, the details are provided for the purpose of understanding and appreciating the basic concepts of the invention and In order not to obscure or depart from the teachings of the present invention, they have not been described to more extent than is deemed necessary as set forth above.

Any reference herein to a method applies mutatis mutandis to a device or system capable of performing the method and/or a non-transitory computer-readable medium storing instructions for performing the method. shall be carried out.

Any reference herein to a system or device shall apply mutatis mutandis to the manner in which it may be performed by the system and/or to a non-transitory device storing instructions executable by the system. It may be applied mutatis mutandis to any computer-readable medium.

Any reference herein to a non-transitory computer-readable medium applies mutatis mutandis to a device or system capable of executing instructions stored on the non-transitory computer-readable medium. and/or the method of executing the instructions may be applied mutatis mutandis.

Any combination of any modules or units recited in any of the figures, any part of the specification, and/or any claims may be provided.

Any one of the units and/or modules illustrated in this application may be implemented in hardware and/or code, instructions and/or commands stored on a non-transitory computer-readable medium, in a vehicle, in a vehicle, etc. It may be included externally, within a mobile device, within a server, etc.

The vehicle may be any type of vehicle, such as land transport vehicles, air transport vehicles, and water vessels.

The specification and/or drawings may refer to images. An image is an example of a media unit. Any reference to an image may apply mutatis mutandis to the media unit. A media unit may be an example of a sensed information unit (SIU). Any reference to the media unit shall not include signals generated by nature, signals representing human behavior, signals representing operations related to the stock market, medical signals, financial series, geodetic signals, geophysical, chemical, molecular It may be applied mutatis mutandis to any kind of natural signals such as, but not limited to, targets, text and numerical signals, time series, etc. Any reference to a media unit may apply mutatis mutandis to a sensed information unit (SIU). The SIU can be of any type, including visible light cameras, audio sensors, infrared, radar imaging, ultrasound, electro-optical, radiographic, and LIDAR (light detection and ranging) sensing. The sensor may be sensed by any type of sensor, such as a magnetic sensor, a thermal sensor, a passive sensor, an active sensor, and the like. Sensing may include generating samples (eg, pixels, audio signals) representative of the transmitted signal or may otherwise reach the sensor. The SIU can be one or more images, one or more video clips, textual information about one or more images, text describing kinematic information about an object, etc.

Object information may include, but is not limited to, object position, object behavior, object velocity, object acceleration, object propagation direction, object type, object dimension or dimensions, etc. may contain information related to the object. Object information can be raw SIU, processed SIU, textual information, information obtained from SIU, etc.

Obtaining object information may include receiving object information, generating object information, participating in processing object information, processing only a portion of object information, and/or processing only another portion of object information. may include receiving.

Obtaining object information may include object detection or may be performed without object detection.

Processing the object information may include at least one of object detection, noise reduction, signal-to-noise ratio improvement, bounding box definition, and the like.

Object information may be received from one or more sources, such as one or more sensors, one or more communication units, one or more memory units, one or more image processors.

Object information may be provided in one or more ways, e.g., in an absolute manner (e.g., providing the coordinates of the object's location) or in a relative manner, e.g., with respect to the vehicle (e.g., if the object is relative to the vehicle). located at a certain distance and at a certain angle).

The vehicle is also called the own vehicle.

The specification and/or drawings may refer to processors or processing circuits. A processor may be a processing circuit. The processing circuit may be used as a central processing unit (CPU) and/or as an application-specific integrated circuit (ASIC) or a field programmable gate array (FPGA). (mable gate arrays) , a full custom integrated circuit, or a combination of such integrated circuits.

Any combination of any of the method steps shown herein and/or in the drawings may be provided.

Any combination of the subject matter of any of the claims may be provided.

Any combination of systems, units, components, processors, sensors shown herein and/or in the figures may be provided.

Any reference to an object may be applicable to a pattern. Therefore, any reference to object detection is applicable mutatis mutandis to pattern detection.

Unimpeded driving relies on circumventing surrounding road objects based on their position and movement, but humans are well known to be poor at evaluating kinematics. Humans appear to use internal representations of surrounding objects in the form of virtual force fields that immediately suggest actions, thus precluding the need for kinematic evaluation. Consider a scenario in which your vehicle is driving in one lane, and a vehicle diagonally ahead of you in the next lane begins to swerve into your lane. The human reaction to braking or derailing is immediate and instinctive, and can be experienced as a virtual force pushing the vehicle away from the veering vehicle. This virtual force representation is learned and associated with specific road objects.

Inspired by the above considerations, we propose a new concept of perceptual fields. The perceptual field is a learned road object representation in the form of a virtual force field that is "sensed" through the own vehicle's control system in the form of ADAS and/or AV software. A field is defined here as a mathematical function that depends on spatial location (or similar quantity).

An example of an inference method 100 is shown in FIG. 1 and includes the following.

Method 100 may be performed for each frame or frames of the vehicle's environment.

Step 110 of method 100 may include detecting and/or tracking one or more objects (eg, including one or more road users). Detection and/or tracking may be performed in any manner. The object or objects may be any object that can affect the behavior of the vehicle. For example, road users (pedestrians, other vehicles), the road and/or route the vehicle is traveling on (e.g. road or route conditions, road geometry, e.g. curves, straight road segments), traffic signs, traffic signals. , road intersections, schools, kindergartens, etc. Step 110 may include obtaining additional information such as kinematic and contextual variables associated with the one or more objects. Obtaining may include receiving or generating. Obtaining may include processing one or more frames to generate kinematic variables and contextual variables.

Note that step 110 may include acquiring kinematic variables (without acquiring one or more frames).

Method 100 may also include obtaining 120 respective perceptual fields associated with one or more objects. Step 120 may include determining which mappings between objects to search and/or seek, and the like.

After step 110 (and even step 120), one or more objects can be modified by passing relevant input variables, such as kinematic and context variables, to the perceptual field (and one or more virtual physics model functions). A step 130 may follow of determining one or more virtual forces associated with the .

Step 130 may be followed by step 140 of determining a total virtual force applied to the vehicle based on one or more virtual forces associated with the one or more objects. For example, step 140 may include performing a vector weighted sum (or other function) on one or more virtual forces associated with one or more objects.

Step 140 may be followed by step 150 of determining a desired (or target) virtual acceleration based on the total virtual force, eg, based on the equivalent of Newton's second law. The desired acceleration may be vector, or in other cases may have a direction.

Step 150 may be followed by step 160 of converting the desired virtual acceleration into one or more vehicle maneuvers that propagate the vehicle in response to the desired virtual acceleration.

For example, step 160 may include converting the desired acceleration into acceleration or deceleration or changing the direction of travel of the vehicle using gas pedal movement, brake pedal movement, and/or steering wheel angle. The transformation may be based on a dynamic model of the vehicle with a particular control scheme.

Advantages of perceptual fields include, for example, explainability, generalizability, and robustness to noisy inputs.

Explainability. Representing the vehicle's movements as a configuration of individual perceptual fields suggests a decomposition of actions into more fundamental components, and as such is an important step toward explainability. The possibility of visualizing these fields and applying intuition from physics to predict the movement of your vehicle offers additional explainability compared to typical end-to-end black-box deep learning approaches. represents. This increased transparency also allows passengers and drivers to have more trust in the AV or ADAS technology.

Generalizability. Representing the vehicle's reaction to unknown road objects as a repelling virtual force field constitutes a guided bias in invisible situations. This representation has a potential advantage in that the representation can handle edge cases in a safe manner with less training. Furthermore, the perceptual field model is holistic in the sense that the same approach can be used for all aspects of driving policy. Perceptual field models can also be divided into narrow driving functions used in ADAS such as ACC, AEB, LCA, etc. Finally, the complexity of the perceptual field allows the model to be trained on atomic scenarios and still be able to adequately handle more complex scenarios.

Robustness to noisy inputs. Physical constraints on the temporal evolution of the perceptual field combined with potential filtering of the input may lead to better treatment of noise in the input data compared to pure filtering of localization and kinematic data.

Physical or virtual forces allow a mathematical formulation, for example in second-order ordinary differential equations involving so-called dynamical systems. The advantage of representing the control policy as such is that it is amenable to intuition from dynamic systems theory and easy to incorporate external modules such as input/output prediction, navigation, and filtering.

A further advantage of the perceptual field approach is that it does not depend on any specific hardware and the computational cost is no higher than existing methods.

training process

The process of learning perceptual fields can be one of two types, behavioral cloning (BC) and reinforcement learning (RL), or a combination thereof. BC approximates a control policy by adapting a neural network to observed human state-action pairs, whereas RL learns by trial and error in a simulated environment without reference to expert demonstrations. impose.

These two classes of learning algorithms can be combined by first learning a policy with BC and using it as an initial policy that is fine-tuned using RL. Another way to combine the two approaches is to first learn the so-called reward function (used in RL) by behavioral cloning to infer what constitutes desirable behavior for humans, and then use regular RL. training through trial and error. This latter approach is known as inverse RL (IRL).

FIG. 2 is an illustration of a training method 200 used for learning with BC.

Method 200 may begin by collecting 210 human data that is considered a demonstration by an expert on how to process a scenario.

Step 210 may be followed by step 220 of constructing a loss function that accounts for the difference between the kinematic variables resulting from the perceptual field model and the corresponding kinematic variables of the human demonstration.

Step 220 is followed by step 230 of updating the parameters of the perceptual field and the auxiliary function (which may be a virtual physical model function different from the perceptual field) to minimize the loss function by some optimization algorithm such as gradient descent. obtain.

FIG. 3 is an example of a training method 250 used for reinforcement learning.

Method 250 may begin by building 260 a realistic simulation environment.

Step 260 may be followed by step 270 of constructing a reward function, either by learning the reward function from expert demonstration or by manual design.

Step 270 is followed by step 280 of running the episode in the simulation environment and continuously updating the parameters of the perceptual field and auxiliary function by some algorithm, such as proximal policy optimization, to maximize the expected cumulative reward. obtain.

FIG. 4 shows an example of a method 400.

Method 400 may be for sensory field driving related operations.

Method 400 may begin with an initialization step 410.

Initialization step 410 may include receiving a group of NNs that are trained to perform step 440 of method 400.

Alternatively, step 410 may include training a group of NNs to perform step 440 of method 400.

Various examples of training NN groups are presented below.
- A group of NNs can be trained to map object information to one or more virtual forces using behavioral cloning.
- A group of NNs can be trained to map object information to one or more virtual forces using reinforcement learning.
- A group of NNs can be trained to map object information to one or more virtual forces using a combination of reinforcement learning and behavioral cloning.
- A group of NNs can be trained to map object information to one or more virtual forces using reinforcement learning with a reward function defined using behavioral cloning.
- A group of NNs can be trained to map object information to one or more virtual forces using reinforcement learning with an initial policy defined using behavioral cloning.
- The NNs may be trained to map object information to one or more virtual physical model functions that are different from one or more virtual forces and perceptual fields.
- The group of NNs may include a first NN and a second NN, where the first NN is trained to map object information to one or more perceptual fields, and the second NN is trained to map object information to one or more perceptual fields. to one or more virtual physical model functions.

The initialization step 410 may be followed by a step 420 of obtaining object information regarding one or more objects located within the environment of the vehicle. Step 410 may be repeated multiple times, and the following steps may also be repeated multiple times. Object information may include video, images, audio, or any other sensed information.

Step 420 may be followed by step 440 of determining one or more virtual forces applied to the vehicle using one or more neural networks (NN).

The one or more NNs may be the entire group of NNs (from initialization step 410), or only a portion of the group of NNs, leaving one or more unselected NNs of the group. There may be.

The one or more virtual forces represent one or more influences of one or more objects on the behavior of the vehicle. An impact can be a future impact or a current impact. The impact may cause the vehicle to change its course.

One or more virtual forces belong to a virtual physical model. A virtual physical model is a virtual model that can substantially apply physical laws (eg, mechanical, electromagnetic, optical) to vehicles and/or objects.

Step 440 includes the following steps:
- calculating a total virtual force applied to the vehicle based on the one or more virtual forces applied to the vehicle;
- determining a desired virtual acceleration of the vehicle based on the total virtual acceleration applied to the vehicle by the total virtual force. The desired virtual acceleration may be equal to the total virtual acceleration or may be different from the total virtual acceleration.

Method 400 may also include at least one of steps 431, 432, 433, 434, 435, and 436.

Step 431 may include determining the condition of the vehicle based on the object information.

Step 431 may be followed by step 432 of selecting one or more NNs based on the situation.

Additionally or alternatively, step 431 may be followed by step 433 of providing context metadata to one or more NNs.

Step 434 may include detecting a class of each of the one or more objects based on the object information.

Step 434 may be followed by step 435 of selecting one or more NNs based on the class of at least one of the one or more objects.

Additionally or alternatively, step 434 may be followed by step 436 of providing class metadata to one or more NNs indicating the class of at least one of the one or more objects.

Step 440 may be followed by step 450 of performing one or more driving-related maneuvers of the vehicle based on the one or more virtual forces.

Step 450 may be performed without human driver intervention and may include changing the speed and/or acceleration and/or heading of the vehicle. This includes performing autonomous driving or performing advanced driver assistance system (ADAS) driving maneuvers, which may include temporarily taking control of the vehicle and/or one or more driving-related units of the vehicle. obtain. This may include setting the vehicle acceleration to a desired virtual acceleration with or without human driver involvement.

Step 440 may include suggesting to the driver to set the vehicle's acceleration to the desired virtual acceleration.

FIG. 5 is an example of a vehicle. The vehicle includes one or more sensing units 501, one or more driving-related units 510 (such as an autonomous driving unit, an ADAS unit, etc.), and a processor 506 configured to perform any of the methods. a memory unit 508 for storing instructions and/or method results, functions, etc., and a communication unit 504.

FIG. 6 shows an example of a method 600 for lane centering RL using lane sample points as input. Lane sample points are located within the vehicle's environment.

RL assumes a simulation environment in which an agent (self-vehicle) has generated input data that allows it to implement its learned policies (perceptual fields).

The method 600 uses the nearest lane of road sample points (X _L,i , Y _L,i ) and (X _R,i , Y R, _i ) where L is left, R is right, and index i points to the sample point. Alternatively, it may begin with step 610 of detecting the side. The speed of the own vehicle (heretofore referred to as vehicle) is indicated by _Vego .

After step 610, the left lane input vector (X _L,i , Y _L,i ) and _Vego are concatenated to X _L and the right lane input vector (X _R,i , Y _R,i ) and _Vego are connected to X A step 620 of connecting to _R may follow.

Step 620 may be followed by step 630 of calculating lane perception fields f _θ (X _L ) and f _θ (X _R ). This is done by one or more NNs.

Step 630 may be followed by step 640 of constructing a differential equation describing the own vehicle acceleration applied to the own vehicle, ie, a=f _θ (X _L )+f _θ (X _R ).

This can be the output of an inference process. Step 640 may be followed by step 450 (not shown).

The method may include updating one or more NNs. In this case, the RL may assume a reward function that is either learned based on expert demonstrations or handcrafted, and in the example of FIG. may be incremented for every timestamp that maintains.

Updating may include performing 670 in a simulated environment, where the RL learning algorithm records what happens in the next step, including obtaining a reward.

Step 670 may include using a particular RL algorithm (eg, PPO, SAC, TTD3) to sequentially update network parameters θ to maximize average reward.

FIG. 7 shows a method 700 for multi-object RL using visual input.

Step 710 of the method 700 includes receiving a series of panoramic segmented images (images acquired by the own vehicle) over a short time window from the own vehicle perspective, relative distances to individual objects X _rel,i . may be included.

Step 710 may be followed by step 720 of applying a spatio-temporal CNN to individual instances (objects) to capture high-level spatio-temporal features X _i .

Step 720 may be followed by step 730 of calculating the individual perceptual fields f _θ (X _i ,i) and the sum Σf _θ (X _rel,I ,X _i ,i).

Step 730 may be followed by step 740 of constructing a differential equation describing the own vehicle acceleration applied to the own vehicle, ie, a=Σf _θ (X _rel,I , X _i ,i).

This can be the output of an inference process. Step 740 may be followed by step 450 (not shown).

The method may include updating one or more network parameters θ using some RL process.

The method may include performing 760 in a simulated environment, where the RL learning algorithm records what happens in the next step, including the reward obtained.

The RL may assume a reward function that is either learned or handcrafted based on demonstrations by experts.

Step 760 may be followed by step 770 of using a particular RL algorithm, such as PPO, SAC, TTD3, to sequentially update network parameters θ to maximize the average reward.

FIG. 8 shows a method 800 for multi-object BC using kinematic input.

Step 810 of method 800 may include receiving a list of detected object relative kinematics (X _rel,i , V _rel,i ), where X _rel,i is the relative kinematics of detected object i relative to the host vehicle. is the relative position, and V _rel,i is the relative velocity of the detected object i with respect to the own vehicle. The host vehicle speed V _ego is also received.

Step 810 may be followed by step 820 of calculating a perceptual field f _θ (X _rel,i , V _rel,i , V _ego, i) for each object.

Step 820 may be followed by step 830 of summing the contributions from the individual perceptual fields. Step 830 may also include normalizing so that the magnitude of the resulting 2d vector is equal to the maximum magnitude of the individual terms, i.e., N ^* Σf _θ (X _rel,i , V _{rel, i} , V _ego, i).

Step 830 may be followed by step 840 of constructing a differential equation describing the own vehicle acceleration applied to the own vehicle, i.e., a=N ^* _Σfθ (X _rel,i , V _rel,i , Vego _, i). .

This can be the output of an inference process. Step 840 may be followed by step 450 (not shown).

The method may include updating one or more network parameters.

The method is initial conditions

may include step 860 of calculating the own vehicle trajectory given .

After step 860, the loss function

A step 870 of calculating . Also, losses are propagated depending on the situation.

FIG. 9 shows the inference of a method 900 with the addition of a loss function of an adaptive cruise control model implemented using kinematic variables as inputs.

Step 910 of the method 900 comprises determining the position of the host vehicle X _ego , the speed of the host vehicle V _ego , the position of the nearest vehicle in front of the host vehicle X _CIPV , and the speed of the nearest vehicle in front of the host vehicle V _CIPV may include receiving.

Step 910 may be followed by step 920 of calculating relative position X _rel =X _ego -X _CIPV and relative velocity V _rel =V _ego -V _CIPV .

Step 920 may be followed by step 930 below.
- Calculate the perceptual field function g _θ (X _rel , V _CIPV ) by the first NN.
- Calculate the auxiliary function h _Ψ (V _rel ) by the second NN.
- Multiply g _θ (X _rel , V _CIPV ) by h _Ψ (V _rel ) to provide the target acceleration (equal to the target force).

This can be the output of an inference process. Step 930 may be followed by step 450 (not shown).

The method may include updating one or more NN parameters.

The method is initial conditions

may include step 960 of calculating the own vehicle trajectory given .

After step 960, the loss function

A step 970 of calculating . Also, losses are propagated depending on the situation.

force feedback

Human drivers (hereinafter referred to as drivers) may operate vehicles in problematic ways. The driver may drive the vehicle in violation of one or more traffic rules and/or drive the vehicle in an reckless manner and/or drive the vehicle in a dangerous manner and/or drive the vehicle in any other manner. may operate a vehicle in an illegal manner.

Examples of problematic driving include not maintaining a safe distance from other road users, ignoring one or more road users, driving in an aggressive manner, and where detours are prohibited. This may include bypassing other road users, causing unnecessary changes in the direction and/or speed of the vehicle's propagation, etc.

A method is provided for communicating with a driver of a vehicle, and in particular for providing restraint to the driver to drive in an appropriate manner.

FIG. 10 shows an example of a method 3000 for communicating with a driver.

According to one embodiment, method 3000 begins with step 3010 of obtaining object information about one or more objects located within a vehicle's environment.

According to one embodiment, step 3010 also includes analyzing object information. The analysis may include determining location and/or motion information of one or more objects. The position and motion information may include the relative position of the one or more objects (with respect to the vehicle) and/or the relative movement of the one or more objects (with respect to the vehicle).

According to one embodiment, after step 3010, by using one or more neural networks (NNs) and based on the object information, one associated with the physical model and one for the vehicle behavior. Alternatively, a step 3020 of determining virtual forces representing the influence of a plurality of objects follows. Virtual forces are essentially applied to the vehicle.

According to one embodiment, step 3020 includes determining 3020-1 one or more virtual fields based on the object information and determining 3020 a virtual force based on the one or more virtual fields. -2.

Step 3020 may be driven from a virtual physical model. For example, if the virtual physical model represents the object as an electromagnetic charge, the one or more virtual fields are virtual electromagnetic fields and the virtual force represents the electromagnetic force generated due to the virtual charges. For example, suppose the virtual physical model is a mechanical model, in which case the virtual force field is driven from the acceleration of the object.

According to one embodiment, after step 3020, force feedback for use in providing an applied physical restraint force in connection with a driver action performed by a driver using the vehicle is determined based at least on the virtual force. A step 3030 follows.

Step 3030 may be performed based on assumptions regarding the relationship between the virtual force and the desired virtual acceleration of the vehicle. For example, the virtual force may have a virtual acceleration (substantially applied to the vehicle), and a desired virtual acceleration of the vehicle may offset the virtual acceleration substantially applied to the vehicle.

According to one embodiment, step 3020 is followed by step 3025 of determining a desired virtual acceleration of the vehicle based on the virtual acceleration. Step 3025 is followed by step 3030.

According to one embodiment, the desired virtual acceleration may have substantially the same magnitude as the applied acceleration, but be directed in the opposite direction.

According to one embodiment, the desired virtual acceleration has a magnitude that is different from the magnitude of the substantially applied acceleration.

According to one embodiment, the desired virtual acceleration has a direction that is not substantially opposite to the direction of the applied acceleration.

According to one embodiment, step 3030 includes at least one of the following.
- Calculate the difference between the desired virtual acceleration and the vehicle acceleration set under the control of the driver's driving operation.
- Determine the force feedback by setting the physical restraint to reduce the difference.
- Determine one or more parameters of force feedback. The one or more parameters may include the magnitude and/or duration and/or manner in which force feedback is provided. For example, the restraining force may be continuous or discontinuous, have a constant magnitude, or have a magnitude that changes over time.
- Determine the impact of force feedback on the effort required by the driver to control vehicle operation. The stronger the restraint force, the more the driver may need to increase his effort to overcome the restraint force.
- Determine the effect of force feedback on the effort required by the driver to turn the vehicle steering wheel.
- Determine the effect of force feedback on the effort required by the driver to change the speed or acceleration of the vehicle.
Determine the restraint force such that when the restraint force is applied, the restraint force limits the effect of driver actions on the vehicle. For example, the driver may need to make larger turns of the steering wheel to cause the vehicle to make the same change of direction.
- Determine the restraint force so that when the restraint force is applied, the restraint force prevents the influence of driver actions on the vehicle.

Determining the force feedback by setting the physical restraint force to reduce the difference includes applying an arbitrary mapping between the value of the difference and one or more parameters of the restraint force. obtain.
- For example, a larger difference may trigger the application of a stronger restraining force.
- The relationship between the magnitude of the difference and the magnitude of the restraining force can be a linear relationship, a non-linear relationship, etc.
- For yet another example, the duration and/or duty cycle associated with applying a restraining force may vary as a function of the difference.

According to one embodiment, step 3030 is followed by step 3040 of responding to the force feedback determination.

According to one embodiment, step 3040 includes at least one of the following.
・Provide force feedback in response to driver actions performed by the driver using the vehicle. Providing may include applying a binding force. Applying a restraining force may include making a mechanical movement, changing the friction of a mechanical element, changing a mechanical transmission ratio, etc.
- instructing a control unit of the vehicle to provide force feedback in response to driver actions performed by a driver using the vehicle; The control unit may control one or more units of the vehicle, such as the steering wheel, gas pedal, brake pedal, throttle linkage, etc.
- instructing a control unit of the vehicle to provide force feedback in response to driver actions performed by a driver using the vehicle;
- Triggers the provision of force feedback when a driver action is confirmed.
・Perform control over the vehicle - Transfer control from the driver to the automatic driving unit.

FIG. 11 shows an example of a scene.

FIG. 11 shows an example of a vehicle 1031 located within a first road segment 1011-1.

Pedestrian 1022 begins to cross the segment in front of vehicle 1301. Pedestrians are represented by a pedestrian virtual field (indicated by virtual equipotential field lines 1022') that displaces vehicles 1031.

FIG. 11 also shows the desired acceleration 1042 of the vehicle.

Before receiving the virtual force, the vehicle moves parallel to the road boundary.

Under this assumption, the current steering position 1050 needs to be changed and the steering wheel is rotated (by the required rotation 1051 of the steering wheel) to drive the vehicle in the desired direction 1043 that is parallel to the direction of the desired acceleration. ) needs to be rotated.

FIG. 11 also shows three examples of undesirable differences between the desired direction (1043) and the undesired direction of vehicle propagation set by the driver. Undesirable directions are represented by dashed lines indicating "driver set directions" 1045.

In these three examples, a restraining force 1044 is provided to reduce the difference and to set the vehicle to propagate according to a desired direction 1043.

In the foregoing specification, the invention has been described with reference to specific illustrations of embodiments of the invention. It will be apparent, however, that various modifications and changes may be made herein without departing from the broader spirit and scope of the invention as set forth in the appended claims. Furthermore, in the description and claims, "front", "back", "top", "bottom", "over", and "under" are used. Terms such as, where indicated, are used for descriptive purposes and not necessarily to describe permanent relative positions. The terms so used are appropriate, such that the embodiments of the invention described herein are capable of operation in, for example, orientations other than those illustrated or otherwise described herein. It is understood that they can be replaced under certain circumstances. Additionally, the terms "assert" or "set" and "negate" (or "deassert" or "clear") refer to signals, status bits, or similar is used herein to refer to placing a device in its logically true or logically false state, respectively. If a logically true state is a logic level 1, a logically false state is a logic level 0. Also, if a logically true state is a logic level 0, a logically false state is a logic level 1. Those skilled in the art will appreciate that boundaries between logic blocks are merely exemplary and that alternative embodiments may merge logic blocks or circuit elements or provide alternative functionality to various logic blocks or circuit elements. will recognize that it is possible to impose a decomposition. Accordingly, it should be understood that the architecture shown herein is merely exemplary, and that, in fact, many other architectures may be implemented that accomplish the same functionality. Any arrangement of components to accomplish the same function is effectively "associated" so that the desired function is achieved. Thus, any two components herein that are combined to achieve a particular function, regardless of architecture or intermediate components, are "associated" with each other such that the desired function is achieved. ” can be considered as a thing. Similarly, any two components so associated are also considered to be "operably connected" or "operably coupled" to each other so as to accomplish a desired function. I can do it. Additionally, those skilled in the art will recognize that the boundaries between the operations described above are merely exemplary. Multiple operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed at least partially overlapping in time. Furthermore, alternative embodiments may include multiple instances of particular operations, and the order of the operations may be changed in various other embodiments. Also, for example, in one embodiment, the illustrated examples may be implemented on a single integrated circuit or as circuits located within the same device. Alternatively, the instances may be implemented as any number of separate integrated circuits or separate devices interconnected with each other in any suitable manner. However, other modifications, variations and substitutions are also possible. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word ``comprising'' does not exclude the presence of elements or steps other than those listed in a claim. Additionally, the terms "a" or "an" as used herein are defined as one or more than one. Additionally, the use of prefix phrases such as "at least one" and "one or more" in a claim may also mean that the same claim includes the prefix phrases "one or more" or "at least one" and " The definition of another claim element by the indefinite article “a” or “an” even if it includes an indefinite article such as “a” or “an” The preamble should not be construed as suggesting that any particular claim containing such preambled claim element is limited to the invention containing only one such element. The same applies to the use of definite articles. Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements such terms describe. Accordingly, these terms are not necessarily intended to indicate temporal or other prioritization of such elements. The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage. While certain features of the invention have been shown and described herein, many modifications, substitutions, changes and equivalents will immediately occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention. It will be understood that various features of embodiments of the disclosure that are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of embodiments of the disclosure that are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination. It will be understood by those skilled in the art that the embodiments of this disclosure are not limited to what has been particularly shown and described above. Rather, the scope of the embodiments of this disclosure is defined by the following claims and their equivalents.

Claims (19)

A method of communicating with a driver of a vehicle, the method comprising:
obtaining object information regarding one or more objects located within the environment of the vehicle;
analyzing the object information;
By using one or more neural networks (NN) and based on said object information, a virtual model is created that is associated with a physical model and represents the influence of said one or more objects on the behavior of said vehicle. determining power and
determining, based at least on the virtual force, force feedback for use in providing an applied physical restraint force in connection with a driver action performed by the driver using the vehicle. 2. The method includes determining one or more virtual fields based on the analysis of the object information, and the determination of the virtual force is based on the one or more virtual fields. The method described in. 2. The method of claim 1, comprising providing the force feedback in response to the driver actions performed by the driver using the vehicle. 2. The method of claim 1, comprising instructing a control unit of the vehicle to provide the force feedback in response to the driver actions performed by the driver using the vehicle. 2. The method of claim 1, comprising triggering the provision of the force feedback upon confirming the driver action. 2. The method of claim 1, wherein the determining the force feedback includes determining the effect of the force feedback on effort required by the driver to control operation of the vehicle. 2. The method of claim 1, wherein the determining the force feedback includes determining the effect of the force feedback on effort required by the driver to rotate a steering wheel of the vehicle. 2. The method of claim 1, wherein the determining the force feedback includes determining the effect of the force feedback on effort required by the driver to change speed or acceleration of the vehicle. 2. The method of claim 1, comprising calculating a desired virtual acceleration of the vehicle based on the virtual force. 10. The method of claim 9, comprising calculating a difference between the desired virtual acceleration and an acceleration of the vehicle that is set under control of the driver's driving maneuvers. 11. The method of claim 10, wherein the determining the force feedback includes setting the physical restraint to reduce the difference. 2. The method of claim 1, comprising determining a desired path for the vehicle based on one or more iterations of the virtual force determination. 13. The method of claim 12, wherein the determining the force feedback includes setting the physical restraint force to reduce a difference between the driver-indicated route and the desired route. 2. The method of claim 1, wherein the restraining force, when applied, limits the effect of the driver action on the vehicle. 2. The method of claim 1, wherein the restraining force, when applied, prevents the effect of the driver action on the vehicle. 2. The method of claim 1, comprising performing a driving-related operation. A non-transitory computer-readable medium for communicating with a driver of a vehicle, the medium comprising:
instructions for obtaining object information regarding one or more objects located within the environment of the vehicle;
instructions for analyzing the object information;
By using one or more neural networks (NN) and based on said object information, a virtual model is created that is associated with a physical model and represents the influence of said one or more objects on the behavior of said vehicle. commands that determine power,
instructions for determining, based at least on the virtual force, force feedback for use in providing an applied physical restraint force in connection with a driver action performed by the driver using the vehicle;
A non-transitory computer-readable medium that stores information. The non-transitory computer-readable medium stores instructions that include determining one or more virtual fields based on the analysis of the object information, and the determination of the virtual force comprises determining one or more virtual fields based on the analysis of the object information; 18. The non-transitory computer-readable medium of claim 17, wherein the non-transitory computer-readable medium is based on a virtual field of . The non-transitory computer readable medium includes instructions for calculating a desired virtual acceleration of the vehicle based on the virtual force, the desired virtual acceleration, and the vehicle being set under the control of the driver's driving operations. 19. The non-transitory computer of claim 18, wherein the non-transitory computer stores instructions for calculating a difference between an acceleration of readable medium.