DE102020127049B4 - Optics-based hands-on/off determination and hand-gesture-based function selection for a human driver - Google Patents

Abstract

Product comprising:• an autonomous steering system which has a steering interface and which is constructed and set up to simultaneously accept and interpret steering instructions from a human driver, which are entered by a human driver via at least one steering interface, and autonomous steering instructions from an autonomous steering system, wherein • the autonomous steering system is capable of switching at least between a first steering mode and a second steering mode,• the first steering mode is a steering mode in which steering is exclusively based on autonomous steering instructions generated by the autonomous steering system,• the second steering mode is a steering mode in which steering takes place exclusively on the basis of steering instructions from a human driver, which are generated by a human driver via the operation of the at least one steering interface,• the product also has at least one camera or one optical sensor , which is configured to track a position and/or posture of a human limb,• the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human limb,• the at least an optical sensor or the camera for tracking the position or posture of the at least one human limb has at least one camera positioned to capture images of at least one of the human driver's hands on or near a steering wheel, and• the The autonomous steering system is configured to use an image analysis system to determine that each finger of the at least one human driver's hand has been moved from a grasping or clasping posture to a fully extended position and to change steering modes based at least on this determination.

Description

technical field

The field to which the disclosure generally relates includes products and methods for autonomous steering systems, which are intended in particular for switching between different steering modes.

background

Vehicles typically include steering systems. The proliferation of camera-based hand recognition along with other sensors in a steering system or other monitoring system to interpret a driver's intention is known. Capacitive strips that can be installed on a steering wheel are known, which can provide improved performance, but these can also depend on physical touch and interpretation of a driver's intention is difficult using them. Interpreting a driver's intent purely based on a moment of touch does not provide a necessary input to initialize or prepare autonomous driving assistance systems to function in a reliable manner.

"https://www.valeo.com/en/valeo-lays-the-ground-for-the-autonomous-commercial-vehiclesrevolution" discloses camera-based hands-on/off monitoring for a vehicle's steering wheel.

"https://www.elektroniknet.de/automotive/haende-am-lenker. 110699.html” describes hand recognition systems for a steering wheel based on capacitive or resistive sensors.

DE 10 2016 210 452 A1 discloses a method and apparatus for a vehicle capable of autonomous and manual operation, wherein a transition between autonomous and manual driving operation occurs when a driver releases and resumes a means for steering the vehicle, respectively, and when also a driver's intention to change driving operations is checked for plausibility. For this purpose, for example, the position of the driver's hands is monitored by means of an interior camera.

DE 11 2017 005 199 T5 discloses a driving mode switching controller, a driving mode switching control method and a driving mode control program for switching the driving mode of a vehicle between a manual driving mode and an automatic driving mode, and in particular for appropriately controlling the switching of the driving mode despite a failure of monitoring a driver's condition.

DE 10 2014 107 194 A1 discloses, among other things, a method for operating a steering system of a motor vehicle, wherein a steering means is connected to a steering gear via a superimposed steering system, characterized in that during autonomous driving operation of the motor vehicle, driver intervention in the steering means is determined, and that the superimposed steering is operated in this way that a coupling between a steering center angle and an introduced into the steering gear angle is set depending on the determined driver intervention.

DE 10 2011 111 897 A1 discloses, among other things, a motor vehicle with at least one driver assistance system and a navigation device, which is provided with at least one sensor for detecting compressive forces, tensile forces or torsional forces, which is connected to an evaluation unit for evaluating its signals, which, when the signals exceed or fall below a threshold value, Driver assistance system puts into operation or puts out of operation.

U.S. 2017/0344838 A1 discloses, among other things, object recognition based on a computer-implemented method, in which image data are received from one or more sensors and one or more objects are detected using the image data. The object potentially represents extremities of a user of a moving platform. The method processes the objects based on processors, with at least two confidence values being determined for each of the objects.

Proceeding from the prior art mentioned above, it is an object of the present invention to achieve a particularly reliable change between a manual and an autonomous driving mode for a vehicle.

The object is achieved by a product with the features of claim 1, by a method with the features of claim 10, by a product with the features of claim 16 and by a method with the features of claim 18. Advantageous developments result from the subclaims.

Overview of illustrative variations

Some illustrative variations may include a method or product for detecting driver intervention in an autonomous steering system.

Further illustrative variations within the scope of the invention will become more apparent from the detailed description provided below. It should be understood that the detailed description and specific examples, while disclosing variations of the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.

character list

• 1 eine veranschaulichende Variation darstellt, in der ein Bild, welches ein Bild einer Kamera oder Daten eines optischen Sensors repräsentiert, welches einen menschlichen Fahrer darstellt, der mit einer Lenkschnittstelle eines autonomen Lenksystems interagiert, mittels eines Verfolgungsmoduls für eine menschliche Gliedmaße analysiert und markiert wird. In dieser veranschaulichenden Variation verwendet das Verfolgungsmodul für eine menschliche Gliedmaße Daten von wenigstens einem optischen Sensor oder einem Bild von wenigstens einer Kamera, um zu ermitteln, dass ein menschlicher Fahrer wünscht, die Kontrolle einer Lenkschnittstelle zu übernehmen, basierend wenigstens auf wenigstens einer Handposition oder Handhaltungscharakteristik des menschlichen Fahrers, gemäß einigen Variationen.
• 2 eine veranschaulichende Variation darstellt, in der ein Bild, welches ein Bild einer Kamera oder Daten eines optischen Sensors repräsentiert, welches einen menschlichen Fahrer darstellt, der mit einer Lenkschnittstelle eines autonomen Lenksystems interagiert, mittels eines Verfolgungsmoduls für eine menschliche Gliedmaße analysiert und markiert wird. In dieser veranschaulichenden Variation verwendet das Verfolgungsmodul für eine menschliche Gliedmaße Daten von wenigstens einem optischen Sensor oder ein Bild von wenigstens einer Kamera um zu ermitteln, dass ein menschlicher Fahrer wünscht, die Kontrolle über eine Lenkschnittstelle abzugeben, basierend wenigstens auf einer Handposition oder einer Handhaltungscharakteristik des menschlichen Fahrers, gemäß einigen Variationen.
• 3A eine Veranschaulichung einer Hand des Fahrers an einem Lenkrad ist, während eine Mehrzahl von Fingern um das Lenkrad gelegt ist und/oder diese greift, gemäß einigen Variationen.
• 3B die Hand des Fahrers veranschaulicht, wie zum Beispiel in 3A veranschaulicht, im Sichtfeld eines optischen Sensors, welcher derart positioniert ist, dass er eine vordere Fläche 200 des Lenkrades sieht, gemäß einigen Variationen.
• 3C die Hand des Fahrers veranschaulicht, wie zum Beispiel in 3A veranschaulicht, im Sichtfeld eines optischen Sensors, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, gemäß einigen Variationen.
• 4A eine Veranschaulichung einer Hand eines Fahrers im Sichtfeld eines optischen Sensors ist, welcher derart positioniert ist, dass er eine vordere Fläche des Lenkrades sieht, wobei der Fahrer eine Handgeste ausführt, welche die Absicht des Fahrers anzeigt, die Lenkkontrolle an das autonome Fahrsystem des Fahrzeugs zu übergeben, indem eine Mehrzahl von Fingern der Hand des Fahrers bewegt wird, wobei die Mehrzahl von Fingern gestreckt ist und nicht um das Lenkrad gelegt ist oder das Lenkrad greift, gemäß einigen Variationen.
• 4B die Hand des Fahrers veranschaulicht, wie zum Beispiel in 4A veranschaulicht, im Sichtfeld eines optischen Sensors, welcher derart positioniert ist, dass er eine vordere Fläche des Lenkrades sieht, gemäß einigen Variationen.
• 4C eine Veranschaulichung einer Hand des Fahrers im Sichtfeld eines optischen Sensors ist, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, wobei der Fahrer eine Handgeste ausführt, welche die Absicht des Fahrers anzeigt, die Lenkkontrolle an das autonome Fahrsystem des Fahrzeugs abzugeben, indem eine Mehrzahl von Fingern des Fahrers bewegt wird, wobei die Mehrzahl von Fingern gestreckt ist und nicht um das Lenkrad gelegt ist oder das Lenkrad greift, gemäß einigen Variationen.
• 4D Daten veranschaulicht, die ein optischer Sensor bezüglich einer Hand eines Fahrers im Sichtfeld eines optischen Sensors erzeugen kann, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, wobei der Fahrer eine Handgeste ausführt, welche die Absicht des Fahrers anzeigt, die Lenkkontrolle an das autonome Fahrsystem des Fahrzeugs zu übergeben, indem eine Mehrzahl von Fingern des Fahrers bewegt wird, wobei die Mehrzahl von Fingern gestreckt ist und nicht um das Lenkrad gelegt ist oder das Lenkrad greift, gemäß einigen Variationen.
• 4E eine Veranschaulichung eines Fahrers ist, welche die Absicht des Fahrers anzeigt, die Lenkkontrolle an das autonome Fahrsystem des Fahrzeugs zu übergeben, indem die Hände des Fahrers vollständig vom Lenkrad entfernt werden, gemäß einigen Variationen.
• 5A eine Hand eines Fahrers im Sichtfeld eines optischen Sensors veranschaulicht, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, wobei die Mehrzahl von Fingern einer Hand des Fahrers um das Lenkrad gelegt ist, was anzeigt, dass es die Absicht des Fahrers ist, die Kontrolle über das Lenksystem des Fahrzeugs zu übernehmen, gemäß einigen Variationen.
• 5B Daten veranschaulicht, die ein optischer Sensor bezüglich einer Hand des Fahrers im Sichtfeld eines optischen Sensors erzeugen kann, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, wobei eine Mehrzahl von Fingern einer Hand des Fahrers um das Lenkrad gelegt ist, was die Absicht des Fahrers anzeigt, die Kontrolle über das Lenksystem des Fahrzeugs zu übernehmen.
• 6 eine Hand eines Fahrers im Sichtfeld eines optischen Sensors veranschaulicht, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, wobei eine Mehrzahl von Fingern der Hände des Fahrers um das Lenkrad gelegt ist, was die Absicht des Fahrers anzeigt, die Kontrolle über das Lenksystem des Fahrzeugs zu übernehmen, gemäß einigen Variationen.
• 6B Daten veranschaulicht, die ein optischer Sensor bezüglich einer Hand des Fahrers im Sichtfeld eines optischen Sensors erzeugen kann, welcher derart positioniert ist, dass er einen seitlichen Rand des Lenkrades sieht, wobei eine Mehrzahl von Fingern jeder der Hände des Fahrers um das Lenkrad gelegt ist, was die Absicht des Fahrers anzeigt, die Kontrolle über das Lenksystem des Fahrzeugs zu übernehmen, gemäß einigen Variationen.
• 7 das Sichtfeld eines optischen Sensors veranschaulicht, welcher derart positioniert ist, dass er eine vordere Fläche des Lenkrades sieht und eine Detektierungszone zeigt, in welcher der Fahrer seine Hände platziert hat, was die Absicht des Fahrers anzeigt, die Kontrolle über das Lenksystem des Fahrzeugs zu übernehmen.
• 8 das Sichtfeld eines optischen Sensors gemäß einigen Variationen veranschaulicht.

Variations within the scope of the invention will be more fully understood from the detailed description and accompanying figures, wherein:

• 1 Figure 12 shows an illustrative variation in which an image representing a camera image or optical sensor data depicting a human driver interacting with a steering interface of an autonomous steering system is analyzed and tagged using a human limb tracking module. In this illustrative variation, the human limb tracking module uses data from at least one optical sensor or an image from at least one camera to determine that a human driver desires to take control of a steering interface based on at least one hand position or hand posture characteristic of the human driver, according to some variations.
• 2 Figure 12 shows an illustrative variation in which an image representing a camera image or optical sensor data depicting a human driver interacting with a steering interface of an autonomous steering system is analyzed and tagged using a human limb tracking module. In this illustrative variation, the human limb tracking module uses data from at least one optical sensor or an image from at least one camera to determine that a human driver desires to relinquish control of a steering interface based at least on a hand position or hand posture characteristic of the driver human driver, according to some variations.
• 3A Figure 14 is an illustration of a driver's hand on a steering wheel while a plurality of fingers wrap around and/or grasp the steering wheel, according to some variations.
• 3B the driver's hand illustrated, as for example in 3A 12, in the field of view of an optical sensor positioned to view a front face 200 of the steering wheel, according to some variations.
• 3C the driver's hand illustrated, as for example in 3A 12, in the field of view of an optical sensor positioned to see a lateral edge of the steering wheel, according to some variations.
• 4A Figure 13 is an illustration of a driver's hand in the field of view of an optical sensor positioned to view a front surface of the steering wheel, the driver performing a hand gesture indicating the driver's intention to turn steering control to the vehicle's autonomous driving system handed over by moving a plurality of fingers of the driver's hand, wherein the plurality of fingers are extended and not wrapped around or grasping the steering wheel, according to some variations.
• 4B the driver's hand illustrated, as for example in 4A 1, in the field of view of an optical sensor positioned to view a front face of the steering wheel, according to some variations.
• 4C Figure 14 is an illustration of a driver's hand in the field of view of an optical sensor positioned to view a lateral edge of the steering wheel, with the driver performing a hand gesture indicating the driver's intent to relinquish steering control to the vehicle's autonomous driving system by moving a plurality of fingers of the driver, wherein the plurality of fingers are stretched and not wrapped around or gripping the steering wheel, according to some variations.
• 4D Illustrates data that an optical sensor can generate regarding a driver's hand in the field of view of an optical sensor positioned to see a lateral edge of the steering wheel, the driver performing a hand gesture indicating the driver's intention to control the steering to hand over to the vehicle's autonomous driving system by moving a plurality of fingers of the driver, wherein the plurality of fingers are extended and not wrapped around or grasping the steering wheel, according to some variations.
• 4E is an illustration of a driver indicating the driver's intention that Passing steering control to the vehicle's autonomous driving system by completely removing the driver's hands from the steering wheel, according to some variations.
• 5A Figure 13 illustrates a driver's hand in the field of view of an optical sensor positioned to see a lateral edge of the steering wheel, with the plurality of fingers of a driver's hand wrapped around the steering wheel indicating that it is the driver's intention to take control of the vehicle's steering system, according to some variations.
• 5B Figure 12 illustrates data that an optical sensor can generate regarding a driver's hand in the field of view of an optical sensor positioned to see a lateral edge of the steering wheel with a plurality of fingers of a driver's hand wrapped around the steering wheel, which indicates the driver's intention to take control of the vehicle's steering system.
• 6 Figure 12 illustrates a driver's hand in the field of view of an optical sensor positioned to view a lateral edge of the steering wheel, with a plurality of fingers of the driver's hands wrapped around the steering wheel indicating the driver's intent to gain control take over the steering system of the vehicle, according to some variations.
• 6B Figure 12 illustrates data that an optical sensor can generate regarding a driver's hand in the field of view of an optical sensor positioned to see a lateral edge of the steering wheel with a plurality of fingers of each of the driver's hands wrapped around the steering wheel, indicating the driver's intent to take control of the vehicle's steering system, according to some variations.
• 7 Figure 12 illustrates the field of view of an optical sensor positioned to see a front surface of the steering wheel and showing a detection zone in which the driver has placed his hands, indicating the driver's intent to take control of the vehicle's steering system .
• 8th Figure 12 illustrates the field of view of an optical sensor according to some variations.

Detailed Description of Illustrative Variations

The following description of the variations is merely illustrative in nature and is in no way intended to limit the scope of the invention, its application, or uses.

In some illustrative variations, a vehicle for cargo or passengers may be propelled by automotive power obtained from a motor that converts a source of stored energy into motive power for the vehicle, such as, but not limited to, an internal combustion engine, a battery powered one engine, a fuel cell powered engine, or any other known engine for providing automotive motive power to a passenger or cargo vehicle. The motive power resulting from the conversion of the stored energy by the engine may be transferred from the engine to a tractive medium along which the vehicle travels, such as but not limited to a terrain, road, waterway, airway, or any other medium known for vehicles to move along through space. The transmission of motive power from the engine to the traction medium may be via any means of automotive propulsion such as, but not limited to, road wheels, treads, casters, rollers, propellers, gas engines, liquid engines, ion propulsion engines, or any other means of automotive propulsion.

As used herein, "wheels" or "wheel" can refer to a traditional road wheel or tire assembly even when modified by a descriptive adjective such as, but not limited to, naming "steerable road wheels," "steerable wheels." , or "drive wheels", but may also refer to any modification of traditional road wheels or tire assemblies such as, but not limited to, rimless maglev tires, spherical tires, or any other means of automotive movement such as, but not limited to, treads, casters , rollers, propellers, gas engines, liquid engines, or ion-powered engines.

As used herein, even when modified by a descriptive adjective, "road" can refer to a traditional road surface such as, but not limited to, a concrete or asphalt road, but can also refer to any surface or medium along which a vehicle for cargo or passengers travels, such as but not limited to water, ice, snow, dirt, air or other gases, or space in general.

As used herein, "limb," even when modified by a descriptive adjective, can mean a hand, finger, forearm, elbow, arm, shoulder, neck, head, thigh, knee, lower leg , a foot, a toe, or any other body part.

As used herein, the “observation zone” may refer to a point, area, or zone from which at least one sensor or camera is configured to acquire data. As a non-limiting example, if a camera is generally designed to capture images of a hand wheel, "observation zone" may refer to the hand wheel, but it may also refer to any location captured in the image. As a further non-limiting example, if a sensor is configured to collect data at the location of the sensor or data at a location remote from the sensor, the location about which the data is collected may be referred to as an "observation zone." As used herein, “observation zone” may also refer to the area of an image that at least one module is configured to analyze, or to the data set captured by sensors that at least one module is configured to analyze analyze. As used herein, “observe” may refer to the use of a camera or a sensor's ability to collect data at the sensor's location or at a location remote from the sensor.

As used herein, even when modified by a descriptive adjective, "gesture" may refer to the pose or position of at least one limb of a human driver.

In some illustrative variations, a vehicle may include a steering system that allows a driver to change a direction of the vehicle or steer it off a path in which it is traveling. This steering system may operate in conjunction with a source of powered vehicle motion, such as a pair of powered road wheels. As a non-limiting example, a vehicle may be equipped with an internal combustion engine that mechanically drives a pair of rear road wheels to propel the vehicle forward along a road. In such an example, the vehicle may be additionally equipped with a set of steerable front road wheels which may be actuated by the steering system via a steering interface such as, but not limited to, a handwheel to steer the vehicle left and right while the Vehicle moving along the road. In such an example, the powered rear road wheels serve as the means for moving a powered motor vehicle, and the steerable pair of front road wheels, operated via the steering interface, serve as the steering system. Importantly, this is not the only means by which a vehicle is considered to be propelled or steered in this disclosure. Indeed, in some illustrative variations, the front road wheels can be both the driven road wheels and the steerable road wheels. Likewise, the means of powered vehicle movement need not be of the same type as the means of steering. That is, if the means for powered vehicle movement includes road wheels, the steering means need not also include road wheels. To this end, as a non-limiting example, it is contemplated that a snowmobile is propelled toward the rear of the vehicle by a set of treads and may be steered toward the front of the vehicle by a set of steerable skis. Additionally, it can be contemplated that the means of powered vehicle movement, such as but not limited to powered road wheels, and the steering means, such as but not limited to steerable road wheels, may change function or change function during operation. As a non-limiting example, a vehicle that has a pair of powered road wheels near the rear of the vehicle and also has a pair of steerable road wheels near the front of the vehicle can change driving modes and begin using every available road wheel, including the front steerable road wheels as driven road wheels while maintaining the steerability property and steering function of the front steerable road wheels. Similarly, in some cases it can be considered that driven road wheels are used intermittently or optionally as steerable road wheels.

In some illustrative variations, a vehicle's steering system may also be autonomous in that the vehicle steers itself toward a predefined location communicated to it without driver assistance or intervention. The vehicle may include an obstacle avoidance system that allows the vehicle to detect and avoid objects in its path. In In some cases, a human driver of the vehicle may wish to assist the vehicle in avoiding an object or to change the vehicle's direction of travel. In such a case, the driver can take control of at least one driver-side steering system controller or at least one steering interface and support the autonomous steering system. When the driver moves to place at least one limb of the human driver on the steering interface, an optical capture system or an imaging system, which includes at least one optical sensor or camera, can capture images or optical data that are processed by a human tracking module Limbs are analyzed to determine whether the human driver intends to take over the task of steering or driving from the autonomous steering system. If the human limb tracking module alone is not able to determine the driver's intention, the human limb tracking module can correlate the analyzed images or optical data with sensor data from sensors embedded in or connected to any steering interface are which the driver is obviously trying to take over, based on the analysis of the optical sensor or image data. In some such cases, the sensor data from sensors embedded in or connected to any steering interface that the human driver is obviously attempting to adopt may include sensor data such as, but not limited to, pressure data from a pressure sensor embedded in a steering interface , touch data from a touch sensor embedded in a steering interface, torque data from any torque sensor connected to a steering interface such as, but not limited to, a torque sensor embedded or connected to a steering column for a hand wheel, a position sensor, which connected to a steering interface such as, but not limited to, a position sensor embedded or connected to a steering column for a handwheel, or any other sensor known to be used to provide appropriate data regarding the operation of a steering interface provided by a human user.

In some illustrative variations, an optical sensing system or imaging system, including at least one optical sensor or camera, may be positioned above or behind a human operator within the vehicle cabin. In some cases, a sensing optical system or imaging system may be positioned on one of the human driver's sides or in front of the human driver on the ceiling of the vehicle cabin, as long as the zone of observation for the sensing optical system or imaging system is still properly established, gestures of a human limb through observing a human driver or interactions with a steering interface by a human driver. In some cases, the detection optical system or imaging system need not be directly focused on an associated observation zone, but may be indirectly focused through the use of at least one mirror. As a non-limiting example, an optical detection system or an imaging system may be positioned near the rear of the vehicle cabin and aligned with a mirror positioned on the ceiling of the vehicle. In such a case, the mirror may be positioned to allow the detection optical system or the imaging system to observe a connected observation zone.

To support a cooperative steering system, in some illustrative variations, the sensors of an autonomous steering system may be configured to detect and identify a steering input as a driver input. Accurate identification of driver input may involve the use of an image analysis neural network to identify and track movement, positions, postures, and gestures, as described herein, of a human driver's limb, in combination with the use of simpler detection methods such as that Detecting a human driver's grip on a steering interface or a human driver-originated force, actuation, or touch on a steering interface through the use of dedicated sensors embedded in the steering interface or indirectly connected to the steering interface. Correlating the image analysis with data from the aforementioned simpler acquisition methods yields an advantageously improved method for determining human driver intent for use in accurately interpreting human driver input within an autonomous steering system.

In some illustrative variations, the logic of any autonomous steering module may be responsive to both autonomous input and driver input. The human driver's input may be associated with a human driver's intent. In some cases, full steering control can be handed over to the human driver if the human driver's intent reaches a certain threshold. In some cases it can Full steering control may be reserved exclusively for autonomous input when the steering system, or another system controlling the steering system, determines that driver input should not be given priority or decides that the human driver's intent is low or absent. As a non-limiting example, the autonomous input, excluding human driver input, may be given sole control of the steering system in a situation where it has been determined by the autonomous steering module that the human driver has a high probability of inadvertently interacting with a steering interface or accidentally performed a recognized gesture.

In some illustrative variations, a vehicle may include a steering system having a steering interface and a set of steerable road wheels. The steering system may be of the electric power steering type, where physical connections mechanically transmit actuation of the steering interface to the steerable wheels. The steering system may be of the "steer-by-wire" type, wherein physical mechanisms do not mechanically transmit actuation of the steering interface to the steerable road wheels, and wherein actuation of the steering interface affects associated actuation of the steerable road wheels via communication of electronic devices, such as, but not limited to sensors, transceivers and electronically excited actuators. In such a system, a steering interface actuation system may include actuators or solenoids or the like to provide visual steering feedback at least in the case where the actuated steering interface is a mechanically actuable steering interface such as, but not limited to, a hand wheel. In a steering system with a digital steering interface, a steering interface actuation system may include any logic modules used in determining how to update any digital data in any digital display for the digital steering interface. Accordingly, if the driver desires to cooperate or intervene with the autonomous steering, the driver may be less disoriented when attempting to take over a steering interface since the steering interface has been autonomously actuated in a manner that is proportional to the vehicle's actual steering maneuver.

In some illustrative variations, a vehicle includes an electronic power steering system. In such a case, the steering interface mechanically communicates with any mechanism used to steer the vehicle in accordance with a steering ratio. Accordingly, if a vehicle having an electronic power steering system has fully autonomous steering or semi-autonomous steering that allows for driver cooperation or intervention, the steering system can communicate visual feedback from the steering to any interface through which the driver can can steer the vehicle. By way of non-limiting example, when a vehicle is in fully autonomous steering mode, it may automatically rotate a vehicle's handwheel in conjunction with the autonomous steering system, which rotates the vehicle's steerable road wheels such that a non-intervening or uncooperative driver may prevent the vehicle's automatic rotating connects the vehicle's hand wheel to the turning of the vehicle's steerable road wheels and intuitively recognizes, assumes or estimates the steering commands of the autonomous steering system. Thus, if the driver desires to cooperate or intervene with the automated steering, the driver may be less disoriented than attempting to take over a steering interface that appears disconnected or misaligned with the vehicle's road wheels.

In some illustrative variations, a steering ratio is defined as the ratio that describes a relationship between manipulation of a steering interface and manipulation of a steering device. As a non-limiting example, a steering ratio for a boat may be defined as the ratio related to how far a rudder of the boat turns in response to a turn of a rudder. As a further non-limiting example, a steering ratio for an aircraft may be defined as a ratio related to how far an aircraft's aileron raises or lowers in response to rotation of a pilot's steering wheel. As a still further non-limiting example, a steering ratio for a wheeled vehicle may be defined as a ratio related to how far a steerable road wheel of the vehicle turns in response to rotation of a hand wheel.

In some illustrative variations, any autonomous actuation of any steering interface may be controlled by the autonomous steering system to make a driver's driving and ride-sharing experience more comfortable and intuitive during a transition from or to autonomous driving, cooperative driving, or fully driver-controlled driving close. In such illustrative variations, autonomous actuation of the steering interface may function as visual steering feedback which allows the driver to intuitively recognize or anticipate the steering instructions transmitted to and interpreted by the autonomous steering system and how these instructions affect the steering of the vehicle via a simultaneous, connected, autonomous actuation of the steering interface. To further simplify the driver's intuitive driving and ride-sharing experience, autonomous steering instructions may be interpreted by the systems affecting the autonomous actuation of the steering interface in a manner intended to be particularly accessible, comfortable, or ergonomic for a human driver to be. As a non-limiting example, a damping factor may be applied to any raw command of the steering interface to slow down any actuation of the steering interface when actuation of the steering interface is terminated by the driver, as well as a timing of the adjustment of the position and force generated by the Steering system acts on the steering interface via a steering interface actuator to control so that the actuation of the steering interface is synchronous with the movement of the vehicle. In this way, the driver may be more likely to find comfort in an autonomously guided vehicle when the visual steering feedback provided to the steering interface by the autonomous steering system appears less jerky, robotic, or inhuman and consequently less alienating to a human observer is, in particular, when attempting to take over the steering interface.

In some illustrative variations, if a vehicle has a steer-by-wire steering system, the visual feedback from the steering may need to be simulated for the driver to provide an intuitive semi-autonomous steering experience for the driver. If a vehicle has a steer-by-wire steering system that has fully autonomous steering or semi-autonomous steering that allows for driver cooperation or intervention, the steering system can simulate visual feedback from the steering at any interface through which the driver can steer the vehicle. By way of non-limiting example, when the vehicle is in fully autonomous steering mode, it may automatically rotate a vehicle handwheel in conjunction with the autonomous steering system, which rotates the vehicle's steerable road wheels such that an uninterrupted or uncooperative driver prevents the vehicle from automatically rotating the handwheel of the vehicle to the turning of the steerable road wheels of the vehicle and intuitively recognizes, guesses or estimates a steering ratio. Thus, if the driver desires to cooperate or intervene with the automated steering, the driver may be less disoriented than attempting to take over a steering interface that appears disconnected or misaligned with the vehicle's road wheels. This concept can be taken into account each time the human limb tracking module detects a driver's intention.

In some illustrative variations, an autonomous steering system may include at least one steering interface, at least one sensor or camera configured to measure or collect optical data, and at least one logic module to determine based on any raw or processed, sensed or acquired optical data whether a human driver desires to take control of the at least one steering interface. Such data may include data such as, but not limited to, a movement velocity, a movement speed, a movement direction, a movement acceleration of a human limb, a limb position, a limb posture, a limb shape , or any other data relating to the appearance, movement, or position of a limb. In some cases, any tracked human limb may be identified and classified as a particular human limb such as, but not limited to, a hand, knee, finger, elbow, or any other human limb using a neural network or database how to use a local or cloud database. Any such identification or classification may be correlated with any limb appearance, motion, or position data to provide accurate context for data collected by cameras or sensors. Any logic module that provides context or analysis for data captured by any such cameras or sensors may be referred to as a human limb tracking module.

To support a cooperative steering system, in some illustrative variations, the sensors of an autonomous steering system may be configured to detect and identify a steering input as a driver input. In some cases, detecting human driver input may cause the autonomous steering system to delegate all driving tasks to a human driver, such as, but not limited to, acceleration, braking, and steering. In some cases, the autonomous steering system can issue only one or more, but not all, of the driving tasks in response to input from a human driver. In some cases, detecting human driver input may cause the steering interface to adjust its position or orientation. By way of non-limiting example, in cases where at least one steering interface includes a handwheel, the handwheel may automatically adjust its telescoping or tilting configuration to be more accessible to the human driver upon detecting human driver input. It may also be contemplated that any change in control or transfer of control related to the autonomous steering system may be accompanied by changes in control or further transfer in other vehicle systems, such as, but not limited to, engine control systems, brake control systems, or powertrain systems.

In some illustrative variations, an autonomous vehicle may include at least one human limb tracking module to determine whether a human driver desires to take control of the driving interface based on any raw, or processed, sensed, or collected optical data. In some such cases, the logic module may determine at least one hand position or hand gesture characteristic of a human driver and based thereon determine whether the human driver intends to take control of the steering interface. In some illustrative variations, any optical data may be processed or correlated with any available data about applied pressure on a steering interface, capacitive touch data from the steering interface, steering interface torque data, steering interface position data, steering interface movement data, or any other data relating to movement or position of a steering interface become.

In some illustrative variations, a neural network may be used to train and build a library about human driver limb positions or postures indicative of driver intent. In such cases, the neural network can be entrusted with the task of image analysis with the aim of identifying and marking at least one human hand within the image. Likewise, the neural network can also be assigned the task of classifying the posture of the identified and marked human hand and determining the position of the human hand with respect to a steering interface. Such a neural network can be trained by collecting thousands of images of many different human drivers and manually marking the human driver's hand in each of the images. The tagged images can then be used to train the neural network to recognize a human driver's hands and to recognize the pose and position of any human driver's hand relative to the steering interface. Any system or module in communication with such a neural network can monitor the identifications, classifications, and determinations based on image analysis and can use operational logic to determine a driver's intent and communicate it to the autonomous driving system . Likewise, the neural network itself can be trained to perform this communication. As a non-limiting example, a movement of all five fingers on one or both hands may be a gesture to be recognized as a gesture indicating a human driver's intention to delegate the task of steering to an autonomous steering system. In one such non-limiting example, a neural network may be used to recognize a particular hand gesture, as well as others, within a set of acquired visual data and to communicate via a logic module to the autonomous steering system that the human driver is the steering system's task would like to hand over to the autonomous steering system for the exclusive execution of the steering. Upon receiving such a communication, the autonomous steering system can assert exclusive control over the task of steering the vehicle. It can also be contemplated that a neural network can be trained to identify and track the position or posture of any other limb of a human driver, such as but not limited to a knee, an elbow, or a forearm. In some variations, the steering system can be combined with torque data to further improve the reliability of the hands-on/off detection. Hand position may be used in conjunction with other sensors in the steering system or other monitoring systems to understand driver intent and prepare autonomous driving assistance system (ADAS) functions to act prior to physical contact with the steering interface to reduce latency. Sensor fusion of hand detection using other driver monitoring techniques may also be employed to understand driver intentions to support better ADAS uptake. Adding hand detection based on its position relative to the steering interface such as a steering wheel, and as this is fused with other physical steering sensors and driver monitoring systems, provides a much better proxy for driver intent. Such a system provides an early indication of physical contact with the steering wheel. The current driver or demonstration that captures your needs for taking additional actions such as pressing buttons on a graphical user interface or a physical button to start 'autonomous mode' in a self-driving vehicle. This is not very intuitive and causes delays between the driver taking his hands off the wheel and the vehicle driving autonomously. Some variations and gestures that may be very similar or identical to removing hands from the steering interface, and in some variations the system may use an occurrence of that gesture to transition to autonomous mode.

In some variations, to interpret the driver's intentions, the position of the driver's hand relative to the steering interface, such as a steering wheel, may be tracked. The steering interface zone is also captured by a similar pre-trained neural network and the zone moves with the tilt and telescoping of the steering system. If the driver's hands are within the zone for a certain length of time and/or with a certain gesture, this can be interpreted as the driver's intention to take over or relinquish control of the vehicle including the steering system. This signal can be sensor fused with physical sensors such as torque to prepare for driver handover by means of a torque burst being eliminated to allow for a smooth handover to the driver. By detecting the driver's intention based on their location relative to the steering wheel, faster responses can be achieved through ADAS functions by pre-processing certain functions based on the interpretation of the driver's intention.

In some illustrative variations, a human limb tracking module may be able to analyze optical data and determine that a human driver desires to take control of a steering interface by using a different extension of the human driver's body like, but not limited to a knee or a thigh. In some such cases, the human limb tracking module may comprise a neural network that has been trained to track the posture or position of at least one limb of a human driver, or capable of being trained to determine the posture or position to track at least one limb of the human driver. Such a human limb tracking module may be configured to track at least one limb of a human driver relative to a specific observation zone within the cabin of the vehicle. As a non-limiting example, the posture and position of at least one limb of a human driver may be tracked with respect to the center point of a handwheel or with respect to an emblem of the handwheel. In this way, the human limb tracking module may be configured to determine that at least one tracked limb of a human driver has changed position and crossed the center point of the handwheel. The human limb tracking module may identify this movement of the human driver's limb as a gesture intended for the human limb tracking module, which should be interpreted as a request by the human driver to an autonomous steering system to perform an action autonomously, such as, but not limited to autonomous lane changing. In some such cases, if telescoping, tilting, or if the position of the center of a handwheel or emblem is moved or adjusted, the observation zone may be employed by at least one optical sensor or camera for the purpose of image or data acquisition and analysis, and also be moved with the handwheel. Where appropriate, the need or desire for such a sensor or camera to adjust perspective or focus to support an adjusted observation zone by the human limb tracking module may be mapped to any suitable mechanism, system or module for adjusting such a camera or a sensor can be communicated.

In some illustrative variations, the speed and direction of movement of the human driver's limb may be tracked by a human limb tracking module. In some such illustrative variations, at least one of the methods of the human limb tracking modules may include a tracking algorithm that accounts for objects partially or completely covering the tracked limb, such as, but not limited to, gloves, watches, pants, shoes, or any other An article of clothing, an accessory, a tool, or an object that requires a complete view of an optical detection system or Imaging system on a human limb fully or partially obscured.

In some illustrative variations, an optical detection system or imaging system such as, but not limited to, a camera system, an infrared detection or imaging system, an acoustic detection or imaging system, a laser detection or imaging system, or any other optical detection or imaging system can be used to track at least one limb of a human driver such as but not limited to hands, forearms, elbows, arms, shoulders, neck, head, thighs, knees, lower legs, feet, or any other body part. An image processing algorithm may be used to remove noise from any captured images or optical data, such as, but not limited to, lighting differences, color differences, or any other image noise that impairs a human limb tracking module's ability to detect, classify, or track a human limb tracking module human limb affected.

In some illustrative variations, while the vehicle is steering autonomously, the driver may attempt to cooperate or interfere with the steering of the vehicle by interacting with at least one steering interface. In some cases, upon determining the driver's intent based on at least one image or visual data analysis disclosed herein, the steering system may allow the driver to temporarily override or cooperate with the steering of the vehicle. This temporary intervention may be enabled based on a determination by a human limb tracking module, in combination with other sensors related to the at least one steering interface, that a driver only temporarily intends to intervene in the task of steering. In such cases, the autonomous steering system can also calculate the deviation from the direction of movement or path of the vehicle, and can continuously calculate new directions of movement or paths of the vehicle when the driver intervenes, so that when the driver stops intervening, the vehicle can continue steering autonomously according to one of the new directions of movement or one of the paths.

In some illustrative variations, a hand wheel serves as the steering interface for an autonomous steering system for a vehicle having steerable road wheels. In some such variations, if the driver intervenes or cooperates with the steering, any mechanism used to automatically rotate the vehicle's handwheel in conjunction with the autonomous steering system that rotates the vehicle's steerable road wheels may require that the driver carries out a steering effort with or against him, while the autonomous steering system continues to steer the vehicle at the same time. In some cases, the autonomous steering system can negotiate the driver input and an autonomous steering input to determine a net steering input and can use the net steering input as a control signal for any turning of the steerable road wheels and associated force feedback for the steering interface, like, but not limited to an automatic turning of a hand wheel. As a non-limiting example, if a driver begins to loosely grip a vehicle's handwheel while the vehicle is in autonomous steering mode, the autonomous steering system may continue to steer the vehicle autonomously and may indicate to the driver his intention to steer to the right, by turning the hand wheel to the right in conjunction with turning the vehicle's steerable road wheels to the right while the driver still has a loose grip on the steering wheel, the driver can sense this intention of the autonomous steering system to turn to the right and can decide to intervene by gripping the handwheel more firmly before increasing the amount of steering. In some such cases, if the driver tries to prevent the hand wheel from turning to the right, the autonomous steering system can detect this and determine a driver's intention based on the driver's intention obtained through analysis of image data or optical Correlates data pertaining to the driver's grip on the handwheel sensed within the vehicle cabin with other sensor data suitable for determining driver intent, such as, but not limited to, pressure data from any pressure sensor used embedded in the handwheel, or any similar embedded or connected touch sensor of any type, or torque sensors trained on, connected to, or embedded in a steering column connected to the handwheel. In some such cases, as a non-limiting example, the autonomous steering system can significantly reduce, but not eliminate, the amount of force it applies to the hand wheel as a component of the autonomous steering system, making it easier for the driver to take control of the vehicle without the To give the driver the impression that the autonomous steering system has completely handed over control to the driver. In this way, the driver can be assured that the autonomous steering system is still trying to control the vehicle and to control the driving vehicle will return after the operator releases the hand wheel. As a further non-limiting example, in some such cases, the autonomous system may completely eliminate the amount of force it applies to the hand wheel as a component of the autonomous steering system, thereby making it easier for the driver to take control of the vehicle and providing additional driver comfort indicate that the autonomous steering system has fully relinquished control to the driver. In some such cases, this may inform the driver via an audible or visual warning that the autonomous steering system is disabled and no longer attempting to steer the vehicle. In such cases, this can provide additional information to the driver that the steering system will not return to control the vehicle after the driver releases the handwheel until autonomous steering is reactivated. In some such cases, the reactivation of autonomous steering or driving may be performed through use of a detected movement or posture of a limb by a human driver. In some such cases, the human driver may employ a recognized hand gesture, such as, but not limited to, a gesture in which each finger of the human driver's at least one hand has been moved from a grasping or clasping posture to a fully extended position. This five-finger movement can be tracked by a human limb tracking system and can be detected by a trained neural network and correlated with other sensor data to determine the driver's intention to autonomously drive or steer based on that gesture to reactivate.

In some illustrative variations, if the vehicle steers autonomously and there is no driver intervention or cooperation, and the driver attempts to intervene or cooperate with the steering, the steering system may determine that the driver's intention is low or absent. the steering system may ignore any unintended driver input regarding steering and instead steer the vehicle by ignoring the unintended driver input. In the case of a non-limiting example of a steer-by-wire system, in the event that the vehicle sensors and systems, or any other system informing the vehicle that a steering maneuver is required for safety reasons, the vehicle may determine any unintended Ignore driver input and perform the steering maneuver autonomously, regardless of any unintended driver input.

In some illustrative variations, the human limb tracking system may assign at least one confidence level to a tracked limb. This confidence level may relate to the human limb tracking system's confidence that the tracked limb is performing or has performed a recognized gesture. In such a case it is possible to register a gesture only if the confidence level for that particular gesture of a particular limb exceeds a certain threshold. Other sensor data such as, but not limited to, sensor data related to the guidance interface or from sensors embedded in the guidance interface may be correlated with image or optical data analyzed by the human limb tracking system and beyond at least inform a confidence level. As a non-limiting example, a lane change gesture may only be recognized if the confidence in that gesture exceeds 90%. In these cases, some limb movements that are very similar to a hand-swipe-over-the-center-of-the-handwheel lane-change gesture, but were not performed by a human driver to indicate a lane-change request, as if a human driver were using their own body reaches over to grab a drink from the center console, but doesn't intend to perform a lane change gesture. In some cases, more than one human limb can be tracked at a time. In some such cases, when more than one human limb is being tracked and it is determined by the human limb tracking module that conflicting gestures or the same gesture were performed with different confidence levels, the guidance system or the human limb tracking system use a prioritization algorithm or decision tree to determine which gesture to count as input, if any, and in what order to address the gesture input.

In some illustrative variations, the steering interface may include a joystick, trackball, slider, throttle, push button, toggle switch, lever, touch screen, mouse, or any other means of user input. In cases where driver input is via a driver interface that does not offer an obvious facility for mechanically powered visual or force feedback, such as, but not limited to, variations that involve a mouse or touchscreen as the primary means of driver input, the automatic Steering system responsive to driver input via visual cues such as but not limited to a flash or animation in a display, audio cues such as but not limited to clicks, beeps, or an instructional voice recording, generated voice, or live voice, or haptic cues such as but not limited to a vibration of a screen, or any other indication that can indicate to the driver how a driver input - which may occur simultaneously with an autonomous input - is countered by the steering system, how it is cooperated with, how it is integrated or how it is handled .

Reference is now made to 1 , which is an illustrative variation in which an autonomous guidance system 101 steers a vehicle 102 along a first autonomous path 103. FIG. In this illustrative variation, a human limb tracking module 115 may analyze or tag images or optical data 116 captured by an optical detection system or imaging system 104 embedded in a rearview mirror 105 . The optical acquisition system or imaging system 104 may acquire images or optical data 116 of an observation zone 106 continuously via a mirror (not shown) positioned behind a human driver 107 . When the human driver 107 attempts to intervene in the steering of the vehicle 102 by manipulating a steering interface 108, the visual capture system or imaging system 104 can capture images or visual data 116 of this human driver 107 intervention attempt by using images or visual Data 116 of the hands 109, 110 of the human driver 107 are recorded. The human limb tracking module 115 may analyze the images or optical data 116 for recognizable limbs and recognizable positions or postures relative to the guidance interface 108 . In this illustrative variation, the human limb tracking module 115 has recognized the hands 109, 110 of the human driver 107 and marked them with markers 111, 112 and assigned them gesture confidence scores 113, 114 by using the images or visual data 116 with the knowledge of a neural Network 117 viewing images or visual data of recognizable human hand gestures associated with steering a vehicle via a handwheel using human hands. The human limb tracking module 115 may correlate these gesture confidence scores 113, 114 with touch or pressure data 118, 119 sensed by touch or pressure sensors 121, 122 fed into the steering interface for the purpose of determining human driver 107 intent 120 108 are embedded while the steering interface 108 is operated in this way. In this illustrative variation, by correlating the gesture confidence scores 113, 114 with the touch or pressure data 118, 119 from the touch or pressure sensors 121, 122, the human limb tracking module 115 may determine that the intention 120 of the human driver 107 is to intervene in the steering. This intent 120 may be communicated through the human limb tracking module 115 to the autonomous steering system 101 , which may then relinquish steering control of the vehicle to the human driver 107 . The human driver 107 can then manually steer the vehicle 102 via the steering interface 108, free from the influence of the autonomous steering system 101.

Referring to 2 1 shows an illustrative variation in which a human driver 207 initially manually steers a vehicle 202 having an autonomous steering system 201 . In this illustrative variation, a human limb tracking module 215 may analyze or tag images or optical data 216 captured by an optical detection system or imaging system 204 embedded in a rearview mirror 205 . The optical acquisition system or imaging system 204 may acquire images or optical data 216 of an observation zone 206 continuously via a mirror (not shown) positioned behind a human driver 207 . The human driver 207 can begin to let go of the steering interface 108 by releasing the grip on the steering interface 208 . The visual capture system or imaging system 204 may capture images or visual data 216 of this human driver 207 release by capturing images or visual data 216 of the human driver's 209 , 210 hands 209 , 210 . The human limb tracking module 215 may analyze the images or optical data 216 for recognizable limbs and recognizable positions or postures relative to the guidance interface 208 . In this illustrative variation, the human limb tracking module 215 has recognized the hands 209, 210 of the human driver 207 and marked them with markers 211, 212 and assigned them gesture confidence scores 213, 214 by using the images or visual data 116 with the knowledge of a neural Network 217 viewing images or visual data of recognizable human hand gestures associated with steering a vehicle via a handwheel using human hands. The human limb tracking module 215 can use this gesture confidence scores 213, 214 with touch or pressure data 218, 219 sensed by touch or pressure sensors 221, 222 embedded in the steering interface 208 for the purpose of determining intent 220 of the human driver 207 when the steering interface 208 is on is no longer operated in this way. In this illustrative variation, by correlating the gesture confidence scores 213, 214 with the touch or pressure data 218, 219 from the touch or pressure sensors 221, 222, the human limb tracking module 215 may determine that the intention 220 of the human driver 207 is to to no longer actuate the steering interface 208 and to hand over the task of steering the vehicle 202 to the autonomous steering system 201 . This intent 220 may be communicated by the human limb tracking module 215 to the autonomous steering system 201 , which then assumes sole control of the task of steering the vehicle 207 . The autonomous steering system 201 may then begin to autonomously steer the vehicle 202 along an autonomous path 203 and preclude any steering inputs by the human driver 207 that have been determined to be unintended.

3A 11 is an illustration of a driver's hand 110 on a steering wheel 108 with a plurality of fingers 123 wrapped around and/or grasping the steering wheel 108, according to some illustrative variations.

3B 11 illustrates the driver's hand 110, such as in FIG 3A 1, in the field of view of an optical sensor positioned to view a front surface 200 of the steering wheel, according to some variations.

3C 11 illustrates the driver's hand 110, such as in FIG 3A 12, in the field of view of an optical sensor positioned to view a lateral edge 201 of the steering wheel 110, according to some variations.

4A 11 is an illustration of a driver's hand 110 in the field of view of an optical sensor positioned to view a front face 200 of the steering wheel 108, with the driver performing a hand gesture indicating the driver's intention is to have steering control to hand over to the vehicle's autonomous driving system by moving a plurality of fingers 123 of the driver's hand, wherein the plurality of fingers 123 are extended and not wrapped around or gripping the steering wheel, according to some variations.

4B 11 illustrates the driver's hand 110, such as in FIG 4A 12, in the field of view of an optical sensor positioned to view a front surface of steering wheel 200, according to some variations.

4C 11 is an illustration of a driver's hand 110 in the field of view of an optical sensor positioned to view a lateral edge 201 of the steering wheel 108, with the driver performing a hand gesture indicating that the driver's intent is steering control to hand over to the vehicle's autonomous driving system by moving a plurality of fingers 123 of the driver, wherein the plurality of fingers 123 are extended and not wrapped around the steering wheel 108 to grasp the steering wheel, according to some variations.

4D 11 illustrates data that an optical sensor may generate regarding a driver's hand 110 in a field of view of an optical sensor positioned to see a lateral edge of the steering wheel, with the driver performing a hand gesture indicating that there is intent of the driver is to hand over steering control to the vehicle's autonomous driving system by moving a plurality of fingers 123 of the driver, wherein the plurality of fingers are extended and not wrapped around or grasping the steering wheel, according to some variations.

4E 14 is an illustration of a driver indicating the driver's intention to transfer steering control to the vehicle's autonomous driving system by completely removing the driver's hand from the steering wheel 108, according to some variations.

5A 11 illustrates a driver's hand 110 in the field of view of an optical sensor positioned to view a lateral edge 201 of the steering wheel 108, with a plurality of fingers 123 of a driver's hand 110 wrapped around the steering wheel, indicating the driver's intention indicates to take steering control from the vehicle's steering system, subject to some variations.

5B 12 illustrates data that an optical sensor may generate regarding a driver's hand 110 in the field of view of an optical sensor positioned to view a lateral edge of the steering wheel with a plurality of fingers 123 of a driver's hand 110 wrapped around the steering wheel is, indicating the driver's intent to take steering control from the vehicle's steering system.

6A Figure 12 illustrates a driver's hand 109, 110 in the field of view of an optical sensor positioned to view a lateral edge 201 of the steering wheel 108, with a plurality of fingers 123 of each driver's hand 109, 110 wrapped around the steering wheel 108, indicating the driver's intent to take control of the vehicle's steering system, according to some variations.

6B Figure 12 illustrates data that an optical sensor can generate regarding a driver's hand 109, 110 in the field of view of an optical sensor positioned to see a lateral edge of the steering wheel, a plurality of fingers 123 of each driver's hand 109, 110 wrapped around the steering wheel, indicating the driver's intent to take control of the vehicle's steering system.

7 Figure 12 illustrates the field of view of an optical sensor positioned to see a front surface 200 of the steering wheel 108 and shows a detection zone 210 in which the driver has placed his hands 109, 110 indicating that it is the driver's intention to take control of the vehicle's steering system. As in 7 As illustrated, object detection software may be employed to determine within a predefined level of acceptance/confidence, for example within a 99% confidence level, that the driver's hands 109, 110 are within the detection zone 210. As in the 7 and 8th As shown, the detection zone 210 may be generally oval, but it may also be round, square, rectangular, diamond-shaped, or any other shape sufficiently large enough that a predefined portion or a sufficient portion of a location of at least one driver's hand is at a steering interface within the detection zone 210, indicating the driver's intent to take control of the vehicle's steering system.

8th Figure 12 illustrates the field of view of an optical sensor positioned to see a front surface of the steering wheel and indicates a detection zone in which the driver has moved his hands outward, indicating the driver's intent to control the vehicle's steering system deliver the autonomous driving system, according to some variations. As in 8th As illustrated, object recognition software may be employed to determine within a predefined level of acceptance/confidence, for example within a 99% confidence level, that a predefined portion or sufficient portion of each of the driver's hands 109, 110 is outside of the Detection zone 210 is located, indicating the driver's intention to relinquish control of the vehicle's steering system to the autonomous driving system.

In some illustrative variations, any number of modules can be combined together or broken down into smaller modules.

The following description of variants is merely illustrative of components, elements, acts, products and methods which are considered to be within the scope of the invention and is in no way intended to limit such scope by what is specifically disclosed or not expressly disclosed is executed. The components, elements, acts, products, and methods as described herein may be combined and rearranged otherwise than as expressly described herein and are still considered to be within the scope of the invention.

Variation 1 may include a product comprising: an autonomous steering system having a steering interface and which is constructed and configured to simultaneously receive human driver steering instructions input by a human driver via at least one steering interface and autonomous steering instructions of an autonomous steering system accept and interpret, wherein the autonomous steering system is able to switch at least between a first steering mode and a second steering mode, wherein the first steering mode is a steering mode in which the steering takes place exclusively on the basis of autonomous steering instructions issued by the autonomous steering system are generated and wherein the second steering mode is a steering mode in which the steering takes place exclusively on the basis of steering instructions from a human driver, which are generated by a human driver via the operation of the at least one steering interface, the product also having at least one camera or comprises an optical sensor configured to track at least one of the position or posture of a human limb and wherein the autonomous steering system is configured to switch between the steering modes based at least on the tracked position or posture of the at least one human limb, wherein the at least one optical sensor or camera for tracking the position or posture of the at least one human limb comprises at least one camera positioned to capture images of at least one of The human driver's hands are detected on or near a steering wheel, and the autonomous steering system is configured to use an image analysis system to determine that each finger of the at least one human driver's hand has moved from a grasping or clasping posture to a fully extended position position has been moved and to change the steering modes based at least on this determination.

Variation 2 may comprise the product of any of Variations 1, or 3-11, wherein the autonomous steering system is further capable of transitioning to a third steering mode in which the at least one steering interface is managed by both the human driver and the autonomous steering system is operated.

Variation 3 may include the product of either of variations 1-2, or 4-11, wherein the at least one camera or optical sensor includes a stereo camera assembly.

Variation 4 may include the product of any of variations 1-3, or 5-11, wherein the product wherein the at least one camera or optical sensor is configured to track the speed of movement of the at least one human limb.

Variation 5 may include the product of any one of variations 1-4, or 6-11, wherein the at least one camera or optical sensor is configured to track the proximity of the at least one human limb relative to the at least one steering interface.

Variation 6 may include the product of any one of variations 1-5, or 7-11, the product further comprising at least one torque sensor for sensing a torque input by a human driver with respect to the steering interface.

Variation 7 may include the product of variation 6, where the torque input is sensed by torque on the steering column.

Variation 8 may include the product of any of Variations 1-7, or 9-11, wherein the product further includes at least one pressure sensor coupled to the steering interface to sense pressure applied to the steering interface by the human driver.

Variation 9 may include the product of any of Variations 1-8, or 10, wherein the product further includes at least one capacitive touch sensor coupled to the steering interface to detect a change in capacitance present at the capacitive touch sensor. when the human driver touches the steering interface.

Variation 10 may include a method comprising: using at least one camera or optical sensor to track at least one of the position or posture of at least one limb, namely a hand, of a human driver, wherein the at least one optical sensor or camera includes at least one a camera positioned to capture images of at least one of the human driver's hands on or near a steering wheel; generating correlation data by correlating the at least one of the position or posture of the at least one limb of the human driver with data acquired from at least one steering interface or from at least one sensor configured to allow interaction with the at least one steering interface by the capture human drivers; changing a steering mode of an autonomous steering system based at least on the correlated data; using an image analysis system to determine that each finger of the at least one hand of the human driver has been moved from a grasping or clasping posture to a fully extended position; and changing steering modes of the autonomous steering system based at least on said determining.

Variation 11 may include the method of variation 10, or 12-15, wherein the at least one camera or optical sensor includes a stereo camera assembly.

Variation 12 may include the method of any of Variations 10-11, or 13-15, wherein the data acquired from at least one steering interface or from at least one sensor configured to facilitate interaction with the at least one steering interface by the human driver include data from at least one capacitive touch sensor embedded in the at least one steering interface.

Variation 13 may include the method of any of Variations 10-12 or 14-15, wherein the data acquired from at least one steering interface or from at least one sensor configured to allow interaction with the at least one steering interface by the human driver detect, comprise at least one torque sensor for detecting a torque input to the steering interface by the human driver.

Variation 14 may include the method of variation 13 wherein the torque input is torque sensed at a steering column.

Variation 15 may include the method of any of Variations 10-14, wherein the data acquired from at least one steering interface or from at least one sensor configured to detect interaction with the at least one steering interface by the human driver includes at least one Comprise a pressure sensor for detecting a pressure exerted on the steering interface by the human driver.

Variation 16 may include a product comprising: an autonomous steering system having a steering interface and constructed and configured to simultaneously receive human driver steering instructions input by a human driver via at least one steering interface and autonomous steering inputs input by an autonomous steering system are entered to accept and interpret, wherein the autonomous steering system is able to switch between at least a first steering mode and a second steering mode, wherein the first steering mode is a steering mode in which the steering takes place exclusively on the basis of autonomous steering instructions, which are provided by the autonomous steering system can be generated and wherein the second steering mode is a steering mode in which the steering occurs exclusively on the basis of steering instructions from a human driver, which are generated by a human driver via the operation of the at least one steering interface, the product also having at least one A camera or an optical sensor configured to track at least one of the position or posture of a human limb and wherein the autonomous steering system is configured to switch between the steering modes based at least on the tracked position or posture of the at least a human limb, wherein the at least one camera or optical sensor is positioned to view a front surface of the steering interface, and wherein the autonomous steering system is configured to use an image analysis system to determine that each finger of the at least one hand of the human driver has been moved from a grasping or clutching posture to a fully extended position and changing steering modes based at least on this determination.

Variation 17 may include a product as set forth in variation 16 wherein the human limb comprises a human hand.

Variation 18 may include a method comprising: providing an autonomous steering system comprising a steering interface and which is constructed and configured to simultaneously accept steering instructions from a human driver input by a human driver via at least one steering interface and steering instructions from an autonomous steering system and to interpret, which are entered by an autonomous steering system, the autonomous steering system being able to switch between at least a first steering mode and a second steering mode, the first steering mode being a steering mode in which the steering is exclusively based on the autonomous steering instructions generated by the autonomous steering system and the second steering mode is a steering mode in which the steering occurs exclusively based on the human driver's steering instructions generated by the human driver via the operation of the at least one steering interface, using at least one camera or an optical sensor to track at least one of the position or posture of the human limb and wherein the autonomous steering system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human limb, wherein the at least a camera or the optical sensor is positioned to view a front face of the steering interface, using an image analysis system to determine that each finger of the at least one human driver's hand has been moved from a grasping or clasping posture to a fully extended position, and changing steering modes of the autonomous steering system based at least on said determining.

Variation 19 may include a method as set forth in variation 18 wherein the human limb comprises a human hand.

The above description of selected variations within the scope of the invention is merely illustrative in nature and, consequently, variations or variants thereof are not to be regarded as a departure from the spirit and scope of the invention.

Claims (19)

Product comprising: • an autonomous steering system, which has a steering interface and which is constructed and set up to simultaneously steering instructions from a human driver, which by a human driver via at least one steering interface and to accept and interpret autonomous steering instructions of an autonomous steering system, where • the autonomous steering system is able to switch at least between a first steering mode and a second steering mode, • the first steering mode is a steering mode in which steering is exclusive takes place on the basis of autonomous steering instructions generated by the autonomous steering system, • the second steering mode is a steering mode in which steering takes place exclusively on the basis of steering instructions from a human driver, which are generated by a human driver via the actuation of the at least one steering interface are generated, • the product also has at least one camera or an optical sensor, which (r) is set up to track a position and / or posture of a human limb, • the autonomous steering system is set up to switch between the steering modes, based at least on the tracked position or posture of the at least one human limb, • the at least one optical sensor or camera for tracking the position or posture of the at least one human limb comprises at least one camera positioned to capture images of at least one of the The human driver's hands are detected on or near a steering wheel, and • the autonomous steering system is configured to use an image analysis system to determine that each finger of the at least one human driver's hand has moved from a grasping or clasping posture to a fully extended position position has been moved and to change the steering modes based at least on this determination. product after claim 1 , wherein the autonomous steering system is further capable of transitioning to a third steering mode wherein the at least one steering interface is actuated by both the human driver and the autonomous steering system. product after claim 1 or 2 , wherein the at least one camera or the optical sensor has a stereo camera arrangement. Product according to one of the preceding claims, wherein the at least one camera or the optical sensor is set up to detect the movement speed of the at least one human limb. A product according to any one of the preceding claims, wherein the at least one camera or the optical sensor is arranged to track the proximity of the at least one human limb with respect to the at least one steering interface. The product of any preceding claim, wherein the product further comprises at least one torque sensor for detecting torque input by the human driver into the steering interface. product after claim 6 , wherein the torque input is sensed by a torque on the steering column. The product of any preceding claim, wherein the product further comprises at least one pressure sensor coupled to the steering interface to sense pressure applied to the steering interface by the human driver. The product of any preceding claim, wherein the product further comprises at least one capacitive touch sensor coupled to the steering interface to detect a change in capacitance present at the capacitive touch sensor when the human driver touches the steering interface. Process comprising: • Using at least one camera and/or one optical sensor to detect at least one of the position and/or posture of at least one limb, namely a hand, of a human driver, wherein the at least one optical sensor or camera comprises at least one camera positioned to capture images of at least one of the human driver's hands on or near a steering wheel, • generating correlation data by correlating at least one of the position or posture of the at least one limb of the human driver with data acquired from at least one steering interface or from at least one sensor configured to allow interaction with the at least one steering interface by the capture human drivers; and • changing a steering mode of an autonomous steering system based at least on the correlation data, • using an image analysis system to determine that each finger of the at least one human driver's hand has been moved from a grasping or clasping posture to a fully extended position, and • Changing the steering modes of the autonomous steering system based at least on this determination. procedure after claim 10 , wherein the at least one camera and/or the optical sensor has a stereo camera arrangement. procedure after claim 10 or 11 , wherein the data acquired from the at least one steering interface or the at least one sensor configured to detect an interaction with the at least one steering interface by the human driver comprises data from at least one capacitive touch sensor integrated into the at least one Steering interface is embedded. Procedure according to one of Claims 10 until 12 , wherein the data acquired from the at least one steering interface or from the at least one sensor configured to detect an interaction with the at least one steering interface by the human driver comprises at least one torque sensor for detecting a torque input into the steering interface by the have human drivers. Procedure according to one of Claims 10 until 13 , wherein the torque input is sensed by a torque on a steering column. Procedure according to one of Claims 10 until 14 , wherein the data acquired from the at least one steering interface or from the at least one sensor configured to detect an interaction with the at least one steering interface by the human driver comprises at least one pressure sensor coupled to the steering interface, to detect a pressure exerted on the steering interface by the human driver. Product having: • an autonomous steering system having a steering interface and which is constructed and configured to simultaneously accept and interpret human driver steering instructions input by a human driver via at least one steering interface and autonomous steering steering instructions input by an autonomous steering system , where • the autonomous steering system is able to switch at least between a first steering mode and a second steering mode, • the first steering mode is a steering mode in which steering takes place exclusively on the basis of steering instructions generated by the autonomous steering system, • the second steering mode is a steering mode in which steering takes place exclusively on the basis of steering instructions from a human driver, which are generated by a human driver via an actuation of the at least one steering interface, • the product also has at least one camera or optical sensor that is set up to track a position and/or posture of a human limb, • the autonomous steering system is configured to switch between the steering modes based at least on the tracked position or posture of the at least one human limb, • the at least one camera or optical sensor is positioned to view a front face of the steering interface, and • the autonomous steering system is configured to use an image analysis system to determine that each finger of the at least one hand of the human driver has been moved from a grasping or clasping posture to a fully extended position and to change steering modes based at least on this determination. product after Claim 16 , wherein the human limb has a human hand. Method comprising: • Providing an autonomous steering system having a steering interface and which is constructed and set up to simultaneously accept and accept steering instructions from a human driver entered by a human driver via at least one steering interface and autonomous steering instructions entered by an autonomous steering system interpret, wherein the autonomous steering system is able to switch at least between a first steering mode and a second steering mode, wherein the first steering mode is a steering mode in which the steering takes place exclusively on the basis of autonomous steering instructions, which are generated by the autonomous steering system and wherein the second steering mode is a steering mode in which the steering is exclusively based on human driver steering instructions generated by a human driver via the operation of at least one steering interface, • using at least one camera or an optical sensor to at least track a position and/or a posture of a human limb and wherein the autonomous guidance system is configured to switch between steering modes based at least on the tracked position or posture of the at least one human limb, wherein the at least one camera or the optical sensor is positioned to see a front face of the steering interface, • using an image analysis system to determine that each finger of the at least one human driver's hand has been moved from a grasping or clasping posture to a fully extended position, and • changing steering modes of the autonomous steering system based at least on this determination. • procedure after Claim 18 , wherein the human limb comprises a human hand.

Images