US20190228243A1 - Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction - Google Patents
Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction Download PDFInfo
- Publication number
- US20190228243A1 US20190228243A1 US15/877,906 US201815877906A US2019228243A1 US 20190228243 A1 US20190228243 A1 US 20190228243A1 US 201815877906 A US201815877906 A US 201815877906A US 2019228243 A1 US2019228243 A1 US 2019228243A1
- Authority
- US
- United States
- Prior art keywords
- target position
- user
- vector
- hand
- vehicle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000001514 detection method Methods 0.000 claims abstract description 40
- 230000004044 response Effects 0.000 claims abstract description 25
- 230000015654 memory Effects 0.000 claims abstract description 19
- 238000012544 monitoring process Methods 0.000 claims abstract description 16
- 239000013598 vector Substances 0.000 claims description 84
- 210000003811 finger Anatomy 0.000 description 34
- 210000003128 head Anatomy 0.000 description 17
- 238000004891 communication Methods 0.000 description 16
- 230000003287 optical effect Effects 0.000 description 6
- 238000013459 approach Methods 0.000 description 5
- 230000002452 interceptive effect Effects 0.000 description 5
- 230000033001 locomotion Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 210000004247 hand Anatomy 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 206010042008 Stereotypy Diseases 0.000 description 1
- 239000004020 conductor Substances 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 210000003813 thumb Anatomy 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
- B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
-
- G06K9/00845—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/0304—Detection arrangements using opto-electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/0416—Control or interface arrangements specially adapted for digitisers
- G06F3/0418—Control or interface arrangements specially adapted for digitisers for error correction or compensation, e.g. based on parallax, calibration or alignment
-
- G06K9/00355—
-
- G06K9/00604—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/19—Sensors therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/146—Instrument input by gesture
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/146—Instrument input by gesture
- B60K2360/1464—3D-gesture
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
- B60K2360/149—Instrument input by detecting viewing direction not otherwise provided for
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
- B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
Definitions
- Embodiments described herein generally relate to vehicles and, more specifically, to vehicles having systems that determine a target position that a user is gesturing towards, where the user's gaze is directed towards a location other than the target position.
- Some types of vehicle systems may allow a driver or passenger to provide input without manipulating buttons or other tactile inputs. More specifically, the vehicle may receive nonverbal communication from an individual using hand gestures.
- the vehicle includes sensors to detect the movement and position of an individual's hand, and determines the information the individual is attempting to convey based on the movement and position of the hand.
- some challenges may exist in certain situations that can limit the system's ability to interpret some of the information expressed by an individual's hands.
- some vehicles include an eye-tracking system that is capable of tracking the driver's gaze direction.
- the driver's gaze direction may be used to determine the driver's level of awareness as he or she is operating the vehicle.
- the driver normally directs his or her attention towards the environment located in front of the vehicle. Therefore, drivers typically direct their gaze away from the front of the road for only a few moments at a time while operating the vehicle. For example, a driver may turn his or her head to the side and look out of one of the side windows of a vehicle, but only for a few seconds. Accordingly, eye-tracking systems are limited in their ability to determine commands based on the gaze direction of the driver.
- a vehicle in one embodiment, includes a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position.
- the vehicle also includes a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user.
- the vehicle also includes one or more processors and one or more non-transitory memory modules communicatively coupled to the one or more processors.
- the processors store machine-readable instructions that, when executed, cause the processors to determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system. The first point and the second point define a pointing axis of the hand of the user.
- the processors are also caused to calculate a virtual eye position based at least in part on a first point located on the hand of the user and the actual eye position.
- the processors are further caused to calculate a first target position based on the virtual eye position and a second target position based on the pointing axis of the hand of the user.
- the processors are caused to determine the final target position based on the first target position and the second target position.
- the processors are caused to control at least one vehicle system based at least in part on the final target position.
- a vehicle in another embodiment, includes a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position.
- the vehicle also includes a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user.
- the vehicle also includes one or more processors and one or more non-transitory memory modules communicatively coupled to the one or more processors.
- the processors store machine-readable instructions that, when executed, cause the one or more processors to determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system. The first point and the second point define a pointing axis of the hand of the user.
- the processors are also caused to calculate a virtual eye position based at least in part on a first point located on the hand of the user and the actual eye position.
- the processors are further caused to calculate a first target position based on the virtual eye position and a second target position based on the pointing axis of the hand of the user.
- the processors are caused to compare a first accuracy associated with the first target position with a second accuracy associated with the second target position. In response to determining the first accuracy and the second accuracy are equal to one another, the processors are caused to determine the final target position based on weighting the first target position and the second target position equally.
- the processors are caused to determine the final target position based on weighting the first target position and the second target position unequally. Finally, the processors are caused to control at least one vehicle system based at least in part on the final target position.
- a method for determining a final target position that a user of a vehicle is gesturing towards includes determining, by a computer, a first point and a second point located on a hand of the user based at least in part on a gesture signal generated by a user detection system. The first point and the second point define a pointing axis of the hand of the user. The method also includes calculating a virtual eye position based at least in part on a first point located on the hand of the user and an actual eye position of the user, where an eye location signal generated by a user gaze monitoring system generates the actual eye position. The method also includes calculating, by the computer, a first target position based on the virtual eye position.
- the method further includes calculating, by the computer, a second target position based on the pointing axis of the hand of the user.
- the method also includes determining the final target position based on the first target position and the second target position by the computer.
- the method includes controlling at least one vehicle system based at least in part on the final target position.
- FIG. 1 depicts a schematic view of an example vehicle configured to determine a final target position based on a virtual eye position and a direction that a hand of a user is pointing towards, according to one or more embodiments shown and described herein;
- FIG. 2 depicts a schematic top view of a user of the vehicle pointing towards a first target position, according to one or more embodiments shown and described herein;
- FIG. 3 is a perspective view of the user driving the vehicle and pointing towards the first target position, according to one or more embodiments shown and described herein;
- FIG. 4 is a perspective view of the user pointing towards the object and also directing his or her gaze towards the first target position, according to one or more embodiments shown and described herein;
- FIG. 5 depicts a schematic top view of the user and includes the first target position, a second target position, and the final target position, according to one or more embodiments shown and described herein;
- FIG. 6 depicts a flowchart of an example method for determining the final target position, according to one or more embodiments shown and described herein.
- the embodiments disclosed herein are directed to vehicle systems and methods to determine a target position that a user is pointing towards when his or her gaze is directed in a location other than the target position.
- the user is a driver of the vehicle.
- the disclosure may also be applied towards a passenger of the vehicle as well.
- the driver When operating a vehicle, the driver normally directs his or her attention towards the environment located in front of the vehicle, and may turn his or her head momentarily to glance at objects located on either side of the vehicle.
- the driver may point or gesture towards a particular object or direction located in the surrounding using his or her hands, where the gesture may be used to convey information to one or more vehicle systems. For example, the driver may point towards a specific landmark, and the landmark is conveyed to a GPS or an interactive dialog system.
- the direction of the driver's gaze may not align with the direction indicated by the hand.
- the direction indicated by a particular driver's hand may be relatively accurate. That is, a driver may tend to point in a direction that is relatively close to or on the target position. However, another driver may not point towards the target position as accurately because he or she is gesturing towards the target position in a different manner. In other words, the accuracy of the direction intended by the driver's hand depends on upon the specific pointing style used by the driver. For example, some people may point to an object using their index finger. However, other people may point using a different finger. Some people may even point using another object that they grasp in their hand.
- the object grasped by the hand may be, for example, a writing instrument like a pen or an object located within proximity of the driver (e.g., a pair of sunglasses or the driver's cellular telephone).
- the disclosed system also determines a simulated or virtual position of the driver's eyes.
- the virtual position of the driver's eyes is aligned with the target object and the tip of the driver's finger, and is used to determine the target position that the driver is pointing towards.
- the driver is looking in a direction other than the target location (e.g., towards the front of the vehicle).
- the virtual eye position may be calculated by the location of the hand of the driver and the actual position of the driver's eyes.
- the location of the driver is determined based on a gesture signal that is generated by an object detection system.
- the system determines an actual position of the driver's eyes (i.e., the real position of the eyes) based at least in part on an eye location signal generated by a driver gaze monitoring system.
- the system determines the virtual eye position by rotating the actual position of the driver's eyes about a vertical axis of the driver until a midpoint measured between the driver's eyes is aligned with the driver's fingertip.
- the position of the driver's eyes when aligned with the driver's fingertip represents the virtual eye position.
- the virtual eye position may also vary in accuracy between different users. That is, the virtual eye position corresponding to one particular individual may be more accurate or less accurate than the virtual eye position that corresponds to another individual.
- the system may determine a directional vector that is oriented towards a first target position.
- the first target position represents the target position calculated based on the virtual eye approach.
- the system also determines a directional vector pointing towards a second target position.
- the second target positon is based on the pointing direction that the driver's hand is gesturing towards.
- the system then calculates an actual or final target position that the driver is gesturing towards by combining the first target position with the second target position.
- the first target position and the second target position are weighted equally to determine the final target position. Specifically, the first target position and the second target position are weighted equally when both the first target position and the second target position provide equal accuracies between one another. In another embodiment the first target position and the second target position are weighted unequally. More specifically, the first target position may be weighted more heavily if the virtual eye approach provides a more accurate result when compared to the driver's hand. Similarly, the second target position may be weighted more heavily if the driver's hand provides a more accurate result when compared to the virtual eye approach.
- the vehicle 100 may be any passenger vehicle such as, for example, a terrestrial, aquatic, and/or airborne vehicle.
- the vehicle 100 includes a communication path 104 , an electronic control unit 102 , an object detection system 130 , a driver detection system 140 , a driver gaze monitoring system 170 (which may also be referred to as a user gaze monitoring system), and one or more vehicle systems 180 .
- the electronic control unit 102 includes one or more processors 105 and one or more memory modules 106 . Referring to FIGS. 1 and 5 , the electronic control unit 102 is configured to calculate a final target position 112 that is located in an environment surrounding the vehicle 100 .
- the final target position 112 may represent an object that the driver is pointing at such as a person, an animal, a landmark, another vehicle, a building, and the like.
- the location represented by the final target position 112 represents where a driver of the vehicle 100 is attempting to point or gesture towards using his or her hand 110 , and is based on two discrete target locations, namely a first target location T 1 and a second target location T 2 that are calculated based on two different approaches that are described in greater detail below.
- the communication path 104 provides data interconnectivity between various modules disposed within the vehicle 100 .
- each of the modules may operate as a node that may send and/or receive data.
- the communication path 104 includes a conductive material that permits the transmission of electrical data signals to processors, memories, sensors, and actuators throughout the vehicle 100 .
- the communication path 104 can be a bus, such as, for example, a LIN bus, a CAN bus, a VAN bus, and the like.
- the communication path 104 may be wireless and/or an optical waveguide.
- Components that are communicatively coupled may include components capable of exchanging data signals with one another such as, for example, electrical signals via conductive medium, electromagnetic signals via air, optical signals via optical waveguides, and the like.
- the communication path 104 may be formed from any medium that is capable of transmitting a signal such as, for example, conductive wires, conductive traces, optical waveguides, or the like. Moreover, the communication path 104 may be formed from a combination of mediums capable of transmitting signals. In some embodiments, the communication path 104 comprises a combination of conductive traces, conductive wires, connectors, and buses that cooperate to permit the transmission of electrical data signals to components such as processors, memories, sensors, input devices, output devices, and communication devices.
- signal means a waveform (e.g., electrical, optical, magnetic, mechanical or electromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave, square-wave, vibration, and the like, capable of traveling through a medium.
- waveform e.g., electrical, optical, magnetic, mechanical or electromagnetic
- the electronic control unit 102 may be any computing device.
- the electronic control unit 102 may be any type of vehicle-installed, handheld, laptop, or other form of single computing device, or may be composed of multiple computing devices.
- the electronic control unit 102 includes one or more processors 105 for controlling operations of the electronic control unit 102 .
- the one or more processors 105 may include any device capable of executing machine-readable instructions stored on a non-transitory computer-readable medium. Accordingly, each of the one or more processors 105 may include a controller, an integrated circuit, a microchip, a computer, and/or any other computing device.
- the electronic control unit 102 further includes one or more memory modules 106 communicatively coupled to the one or more processors 105 .
- the one or more memory modules 106 may be configured as volatile and/or nonvolatile memory and, as such, may include random access memory (including SRAM, DRAM, and/or other types of RAM), flash memory, secure digital (SD) memory, registers, compact discs (CD), digital versatile discs (DVD), and/or other types of non-transitory computer-readable mediums. Depending on the particular embodiment, these non-transitory computer-readable mediums may reside within the electronic control unit 102 and/or external to the electronic control unit 102 .
- the one or more memory modules 106 may be configured to store one or more pieces of logic as described in more detail below. The embodiments described herein may utilize a distributed computing arrangement to perform any portion of the logic described herein.
- Embodiments of the present disclosure include logic that includes machine-readable instructions and/or an algorithm written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, and/or 5GL) such as, machine language that may be directly executed by the processor, assembly language, object-oriented programming (OOP), scripting languages, microcode, etc., that may be compiled or assembled into machine-readable instructions and stored on a machine-readable medium.
- the logic and/or algorithm may be written in a hardware description language (HDL), such as logic implemented via either a field-programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), and their equivalents.
- HDL hardware description language
- the logic may be implemented in any conventional computer programming language, as pre-programmed hardware elements, and/or as a combination of hardware and software components.
- Logic stored on the one or more memory modules 106 may include, for example, object recognition logic, speech recognition logic, risk determination logic, notification generation logic, and autonomous vehicle control logic.
- the electronic control unit 102 includes logic to calculate the virtual eye position and then determine the location of the final target position 112 .
- the logic stored on the one or more memory modules 106 may include object recognition logic.
- the object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment.
- Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like).
- SIFT scale-invariant feature transform
- SURF speeded up robust features
- GLOH gradient location and orientation histogram
- the logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within the vehicle 100 . Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic.
- the one or more memory modules 106 and the one or more processors 105 are integral with the electronic control unit 102 .
- the electronic control unit 102 , the one or more memory modules 106 , and the one or more processors 105 may be discrete components communicatively coupled to one another without departing from the scope of the present disclosure.
- one or more processors and one or more memory modules 106 of the electronic control unit 102 may be remote to the vehicle 100 .
- the vehicle 100 may be in wireless communication (e.g., using a wireless communication system) with a remote server storing logic and data that is configured to perform at least some of the functionalities described herein.
- the object detection system 130 is communicatively coupled to the electronic control unit 102 over the communication path 104 .
- the object detection system 130 may include any device configured to detect the presence of an object within the surrounding environment of the vehicle 100 . More specifically, the object detection system 130 is configured to detect the presence of an object within the vicinity of the vehicle 100 .
- the object detection system 130 may include an object detection sensor 132 configured to output an object signal indicative of the presence of one or more objects within the vicinity of the vehicle 100 . Based on the object signal of the object detection sensor 132 , the electronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification.
- the object detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, the object detection system 130 includes more than one object detection sensor 132 .
- the driver detection system 140 is communicatively coupled to the electronic control unit 102 over the communication path 104 .
- the driver detection system 140 may include any device configured to detect the presence, movements and/or actions of the driver of the vehicle 100 .
- the driver detection system 140 may include one or more driver detection sensors 142 .
- the driver detection sensors 142 may include, but are not limited to, a camera with a field of view on a face and the surrounding area of the driver. Referring to FIGS. 1 and 5 , the driver detection sensors 142 are configured to output a gesture signal that is indicative of a first point 200 located on the driver's hand 110 .
- the gesture signal is created in response to the driver raising his or her hand 110 away from a steering wheel 144 of the vehicle 100 attempting to point or otherwise draw attention to the second target position T 2 . That is, the gesture signal is created in response to the driver gesturing towards one or more objects using his or her hand 110 .
- the first point 200 of the driver's hand 110 represents a tip 208 of one of the fingers 114 located on the hand 110 of the driver. More specifically, the first point 200 is at the tip 208 of the driver's index finger 114 . This is because the driver is gesturing towards the second target position T 2 by pointing his or her index finger 114 along a pointing direction P. Accordingly, the electronic control unit 102 determines that the driver is pointing towards the second target position T 2 using his or her hand 110 , and sets the first point 200 as the tip 208 of the driver's index finger 114 .
- the disclosure is not limited to gesturing towards the second target position T 2 using an index finger of an individual. Instead, the driver may point using one or more digits of the hand 110 (e.g., the thumb, ring finger, etc.), where the tip of one or more digits of the user's hand 110 represent the first point 200 .
- the driver may point using one or more digits of the hand 110 (e.g., the thumb, ring finger, etc.), where the tip of one or more digits of the user's hand 110 represent the first point 200 .
- the first point 200 may not be located on the driver's hand 110 .
- the driver may grasp an article or item with his or her hand. The driver may then use the object to point towards an object.
- the driver may point towards the second target position T 2 using elongated articles such as a pen, a pencil, a stick, or the like. Therefore, the tip 208 of the driver's finger 114 is actually represented by an endpoint of the article that is being grasped by the driver.
- a distal end of the article may be considered the tip 208 of the driver's finger 114 .
- the distal end of the article points towards the second target position T 2 ( FIG.
- the disclosure is not limited to elongated object.
- the driver may also gesture using any other article that is sized to be grasped by the hand 110 of the driver, and can be manipulated to point towards a particular direction.
- the article may be an item that the driver usually wears or keeps nearby while driving the vehicle 100 such as, but not limited to, a cellular telephone, a pair of sunglasses, and the like.
- the driver gaze monitoring system 170 is communicatively coupled to the electronic control unit 102 over the communication path 104 .
- the driver gaze monitoring system 170 may include any device configured to monitor the driver's gaze associated movement. More specifically, the driver gaze monitoring system 170 includes one or more devices to monitor a direction and motion of the eyes 116 of the driver relative to his or her head 126 . As such, the driver gaze monitoring system 170 may include one or more eye tracking systems 172 configured to output a direction signal indicative of the driver's gaze direction, which is referred to in FIG. 2 as D 1 . The eye tracking systems 172 may also output an eye location signal that indicates an actual position of the eyes 116 of the driver.
- the eye tracking system may include one or more cameras or some other optical sensors for detecting light reflected back from the driver's eyes.
- the light reflected back from the driver's eyes may be near infrared light, which may range from about 700 nanometers to 2500 nanometers in the electromagnetic spectrum.
- FIG. 2 is a schematic top view illustrating a head 126 and a finger 114 of the driver, where the driver is pointing towards a first target position T 1 .
- FIG. 3 is a perspective view of the driver gesturing towards the first target position T 1 shown in FIG. 2 .
- the driver is attempting to gesture towards the first target position T 1 using one of the fingers 114 on his or her hand 110 , where the driver is pointing or gesturing in the pointing direction P ( FIG. 3 ), but the driver's gaze is directed towards the driver's gaze direction D 1 .
- the driver's gaze direction D 1 is pointing towards a location that is different than the first target position T 1 .
- the driver's gaze direction D 1 is directed towards a driving route 118 of the vehicle 100 .
- both the first target position T 1 as well as the second target position T 2 are shown, where the second target position T 2 is based on the direction indicated by an index finger 114 of the driver's hand 110 (i.e., the pointing direction P).
- the electronic control unit 102 calculates the first target position T 1 based on the virtual eye position, the second target position T 2 , and then determines the final target position 112 based on the first target position T 1 and the second target position T 2 . More specifically, in the non-limiting embodiment as shown in FIG. 5 , the final target position 112 is determined by dividing an angle A created by a first vector representing the driver's gaze direction D 1 and a second vector representing the pointing direction P into two different sub-angles.
- the angle A is divided into a first sub-angle ⁇ 1 and a second sub-angle ⁇ 2 .
- FIG. 5 illustrates the angles ⁇ 1 and ⁇ 2 being equal to one another, it should be appreciated that the disclosure is not limited to this particular configuration. That is, the angles ⁇ 1 and ⁇ 2 may also be unequal as well.
- the electronic control unit 102 determines the position of the eyes 116 of the driver. The electronic control unit 102 then calculates a midpoint M 1 between the two eyes 116 of the driver. More specifically, as seen in FIG. 2 the electronic control unit 102 calculates the midpoint M 1 by determining a first line segment S 1 relative to the driver's head 126 .
- the first line segment S 1 intersects both eyes 116 of the driver, is tangent with respect to the driver's head 126 , and is substantially perpendicular with respect to the vector representing the driver's gaze direction D 1 .
- the midpoint M 1 represents a middle point between the two eyes 116 of the driver that intersects the first line segment S 1 .
- the driver's head 126 is drawn as a circle for purposes of simplicity and clarity in the illustration. However, it should be appreciated that the illustration shown in FIG. 2 is not limiting.
- the electronic control unit 102 may calculate the line segment S 1 and the midpoint M 1 by modeling the driver's head using any number of shapes or profiles.
- the electronic control unit 102 may then calculate the virtual eye position, which represents a simulated or virtual position of the driver's eyes.
- a set of virtual eyes 120 of the driver is shown.
- the virtual eyes 120 are directed towards the first point 200 on the driver's hand 110 . That is, FIG. 2 illustrates the pair of virtual eyes 120 directed towards the tip 208 of the finger 114 .
- the electronic control unit 102 may calculate the position of the set of virtual eyes 120 by rotating the actual position of the driver's eyes 116 about a vertical axis A-A of the head 126 of the driver until the midpoint M 1 measured between the driver's eyes 116 is aligned with the first point 200 located at the tip 208 of the driver's finger 114 when viewed in a plane substantially perpendicular to the axis A-A of the driver (i.e., a horizontal plane).
- the electronic control unit 102 simulates the driver turning his or her head 126 such that the virtual eyes 120 are positioned to see the first target position T 1 .
- the vertical axis A-A of the driver's head 126 represents the vertical axis of the driver's body.
- the vertical extends from the head to the feet of a human's body.
- humans are capable of rotating about their respective vertical axis to perform a three hundred and sixty degree turn.
- FIG. 4 illustrates the driver directing his or her gaze towards the first target position T 1 while also pointing in the pointing direction P.
- the driver has turned his or her head 126 such that his or her eyes 116 are aligned with the first target position T 1 .
- the driver's eyes 116 are directed towards a third gaze direction D 3 that is aligned with the tip 208 of the driver's finger 114 and is directed towards the first target position T 1 . Therefore, the driver is indicating the general direction of the first target position T 1 by pointing his or her finger 114 towards the direction P.
- the driver is also gazing past the tip 208 of his or her finger 114 towards the first target position T 1 .
- a midpoint between the virtual eyes 120 of the driver is indicated as midpoint M 2 .
- the electronic control unit 102 calculates the midpoint M 2 by first determining a second line segment S 2 relative to the driver's head 126 .
- the second line segment S 2 intersects both of the virtual eyes 120 and is tangent with respect to the driver's head 126 .
- the midpoint M 2 represents a middle point between the two virtual eyes 120 that intersect the second line segment S 2 .
- the electronic control unit calculates a virtual gaze direction D 2 of the driver.
- the electronic control unit 102 calculates the virtual gaze direction D 2 by determining a vector that originates at the midpoint M 2 and intersects the first point 200 of the driver's hand 110 . As seen in FIG. 2 , the vector is substantially perpendicular with respect to the second line segment S 2 .
- the virtual gaze direction D 2 represents a simulated gaze direction of the driver, where the driver is looking towards the first target position T 1 . That is, the driver's head 126 is turned such that the eyes 116 would be directed towards the first target position T 1 . More specifically, the virtual gaze direction D 2 of FIG. 2 is the same as the third gaze direction D 3 illustrated in FIG. 3 . The third gaze direction D 3 is aligned with the tip 208 of the driver's finger 114 and is directed towards the first target position T 1 .
- the electronic control unit 102 determines the first target position T 1 based on the virtual gaze direction D 2 .
- the electronic control unit 102 determines one or more objects indicated by the object signal generated by the object detection system 130 and the virtual gaze direction D 2 .
- the electronic control unit 102 determines the first target position T 1 by identifying one or more objects that intersect with the vector associated with the virtual gaze direction D 2 .
- the electronic control unit 102 may execute object recognition logic to classify the objects.
- the electronic control unit 102 may employ the object recognition logic configured to recognize specific landmarks. Some examples of landmarks that may be recognized include, but are not limited to, the Golden Gate Bridge in San Francisco, the Empire State Building in New York City, the Eiffel Tower in Paris, and the like.
- the electronic control unit 102 receives as input the gesture signal generated by the driver detection sensors 142 .
- the gesture signal indicates at least two points located on the driver's hand 110 . More specifically, the gesture signal indicates the first point 200 as well as a second point 202 , where both points 200 , 202 are located along the finger 114 of the driver's hand 110 .
- the first point 200 is located at the tip 208 of the driver's finger 114 .
- the second point 202 is located along the driver's finger 114 , at a location below the tip 208 , and is collinear with respect to the first point 200 .
- the points 200 , 202 define a line segment 204 , where the line segment 204 represents a pointing axis of the driver's finger 114 .
- the pointing axis indicates the direction that the driver's hand 110 is pointing or gesturing towards. More specifically, in the embodiment as illustrated in FIG. 5 the pointing axis (i.e., the line segment 204 ) is aligned with the driver's index finger 114 .
- the electronic control unit 102 then extends the line segment 204 representing the pointing axis beyond the tip 208 of the driver's finger 114 to determine a vector that represents the pointing direction P.
- the electronic control unit 102 determines the second target position T 2 based on the pointing direction P. In some embodiments, the electronic control unit 102 may determine the second target position T 2 based on one or more objects indicated by the object signal generated by the object detection system 130 and the vector representing the pointing direction P. The electronic control unit 102 determines the second target position T 2 by identifying one or more objects that intersect with the vector associated with the pointing direction P 2 . Once the first target position T 1 and the second target position T 2 are calculated, the electronic control unit 102 may then determine the final target position 112 .
- the electronic control unit 102 may first determine an angle A that is created by the first vector representing the virtual gaze direction D 2 and the second vector representing the pointing direction P intersecting one another. The electronic control unit 102 then calculates an intersecting vector 206 that divides the angle A into two separate angles. Specifically, the angle A is divided into the first sub-angle ⁇ 1 and the second sub-angle ⁇ 2 . The first sub-angle ⁇ 1 is measured between the vector representing the pointing direction P and the intersecting vector 206 , and the second sub-angle ⁇ 2 is measured between the intersecting vector 206 and the driver's gaze direction D 2 .
- the electronic control unit 102 compares a first accuracy associated with the first target position T 1 with a second accuracy associated with the second target position T 2 , and determines that both the first accuracy and the second accuracy are equal to one another. Accordingly, the first target position T 1 and the second target position T 2 are weighted equally to determine the final target position 112 .
- the intersecting vector 206 bisects the angle A to create two equal sub-angles ⁇ 1 and ⁇ 2 . That is, the intersecting vector 206 divides the angle A into two equal parts.
- the position of the intersecting vector 206 is not limited to the embodiment as shown in FIG. 5 .
- the electronic control unit 102 compares the first accuracy associated with the first target position T 1 with the second accuracy associated with the second target position T 2 , and determines that the first accuracy is unequal with respect to the second accuracy. In response to determining the first target position T 1 and the second target position T 2 provide unequal accuracies, the electronic control unit 102 determines that the intersecting vector 206 divides the angle A into two unequal sub-angles ⁇ 1 and ⁇ 2 . More specifically, in response to determining the first accuracy is greater accuracy than the second accuracy, the electronic control unit 102 divides the angle A by the intersecting vector 206 such that the first sub-angle ⁇ 1 is greater than the second sub-angle ⁇ 2 .
- the intersecting vector 206 is directed towards the final target position 112 . Accordingly, the virtual gaze direction D 2 influences the final target position 112 more than the pointing direction P. Similarly, in response to determining the second accuracy is greater than the first accuracy, the electronic control unit 102 divides the angle A by the intersecting vector 206 such that the first sub-angle ⁇ 1 is less than the second sub-angle ⁇ 2 Accordingly, the intersecting vector 206 is drawn closer towards the pointing direction P.
- the weight or influence given to either the pointing direction P or the virtual gaze direction D 2 when calculating the final target position 112 is based at least in part by the particular identity of the driver. That is, the weight or influence of the pointing direction P and the virtual gaze direction D 2 may vary between individuals. This is because different people point and look towards an object in a variety of configurations or position. More specifically, the accuracy of the virtual gaze direction D 2 varies based on the identity of the driver since some people may tend to look directly at an object of interest while driving, while others do not look at the object directly. The accuracy of the pointing direction P is also variable since different people gesture towards an object by their hand using a variety of approaches.
- some people may gesture towards an object or position by pointing their index finger towards the object (i.e., as shown in the figures).
- other individuals may point towards an object may gesture towards an object using a different finger of their hand.
- some individuals may even use another object grasp by their hand to gesture in a particular direction.
- the electronic control module 102 weights the pointing direction P and the virtual gaze direction D 2 equally. In response to determining the pointing direction P provides greater accuracy when compared to the virtual gaze direction D 2 , the electronic control unit 102 weights the pointing direction P more than the virtual gaze direction D 2 . Accordingly, the intersecting vector 206 in FIG. 5 is positioned closer to the pointing direction P and the second target T 2 . Similarly, in response to determining the virtual gaze direction D 2 provides more accuracy than the pointing direction P, the electronic control unit 102 weights the virtual gaze direction D 2 more than the pointing direction P. The intersecting vector 206 is also located closer towards the virtual gaze direction D 2 and the first target location T 1 .
- the final target position 112 is determined based on the object signal generated by the object detection system 130 and the intersecting vector 206 .
- the electronic control unit 102 determines one or more objects based on the object signal that intersect with the intersecting vector 206 , and identifies the object that intersects with the vector as the final target position 112 .
- the object may be a person, another vehicle, a building, and the like.
- the virtual gaze direction D 2 , the final target position 112 , and the object classified using object recognition logic are communicated to one or more vehicle systems 180 from the electronic control unit 102 via the communication path 104 .
- the vehicle systems 180 may be controlled based on at least one of the virtual gaze direction D 2 , the final target position 112 , and the object classified using object recognition logic.
- the vehicle systems 180 may include, but are not limited to, a GPS system or an interactive dialog system.
- An interactive dialog system converses with the driver using text, speech, gestures, haptics, and the like.
- the interactive dialog system may assist the driver with identifying objects located in the final target position 112 . For example, if the driver points towards the Empire State Building in New York City and asks “What am I pointing to?”, then the interactive dialog system would answer back “The Empire State Building”.
- a flowchart 600 depicting a method for determining the final target position 112 is graphically depicted.
- the final target position 112 is based on the virtual eye position as well as the direction that the driver points towards using his or her hand 110 . It should be understood that embodiments are not limited by the order of steps of the flowchart 600 of FIG. 6 .
- the electronic control unit 102 receives the gesture signal generated by the driver detection system 140 .
- the gesture signal indicates that the driver is raising his or her hand 110 away from the steering wheel 144 of the vehicle 100 to point or otherwise draw attention to an object or a position located within the environment surrounding the vehicle 100 .
- the method may then proceed to decision block 604 .
- the electronic control unit 102 determines if the driver's gaze is directed towards the final target position 112 . More specifically, the electronic control unit 102 determines if the driver's gaze direction D 1 is directed towards the final target position 112 based on the direction signal generated by the driver gaze monitoring system 170 and the first point 200 on the driver's hand 110 . If the driver's gaze direction D 1 intersects the first point 200 , then the electronic control unit 102 determines that the driver's gaze direction D 1 is directed towards the final target position 112 . The method may then terminate. However, if the electronic control unit 102 determines that the driver's gaze direction D 1 does not intersect the first point 200 on the driver's hand 110 , then the method may proceed to block 606 .
- the electronic control unit 102 determines the position of the driver's eyes 116 ( FIG. 2 ) based on the eye location signal generated by the driver gaze monitoring system 170 . The method may then proceed to block 608 .
- the electronic control unit 102 determines the first point 200 on the driver's hand 110 ( FIG. 2 ) based on the gesture signal generated by the driver detection system 140 .
- the first point 200 represents the tip 208 of the driver's index finger 114 .
- the disclosure is not limited to the driver's index finger 114 .
- the first point 200 may be an object that the driver grasps in his or her hand such as a pen, a pencil, a pair of sunglasses, and the like. The method may then proceed to block 610 .
- the electronic control unit 102 determines the position of the virtual eyes 120 .
- the virtual eyes 120 are calculated by rotating the actual position of the driver's eyes 116 about the vertical axis A-A of the driver's head 126 until the midpoint M 1 measured between the driver's eyes 116 is aligned with the first point 200 of the driver's finger 114 . The method may then proceed to block 612 .
- the electronic control unit 102 calculates the virtual gaze direction D 2 by determining a vector that originates at the midpoint M 2 and intersects the first point 200 of the driver's hand 110 . The method may then proceed to block 614 .
- the electronic control unit 102 determines the second target position T 2 based on the pointing direction P. In some embodiments, the electronic control unit 102 may also determine one or more objects based on the object signal generated by the object detection system 130 and the vector representing the pointing direction P. More specifically, the object or objects indicated by the object signal that intersect with the vector as the second target position T 2 may represent the second target position T 2 . The method may then proceed to block 616 .
- the electronic control unit 102 determines the second target position T 2 as seen in FIG. 5 .
- the second target position T 2 is determined based on the pointing direction P that indicates the direction that the driver is gesturing towards. More specifically, as seen in FIG. 5 the pointing direction P is collinear with respect to the pointing axis of the driver's finger 114 (the pointing axis is illustrated in FIG. 5 as the line segment 204 ). The method may then proceed to block 618 .
- the electronic control unit 102 determines the intersecting vector 206 shown in FIG. 5 .
- the intersecting vector 206 divides the angle A into the first sub-angle ⁇ 1 and the second sub-angle ⁇ 2 .
- the first sub-angle ⁇ 1 is measured between the vector representing the pointing direction P and the intersecting vector 206
- the second sub-angle ⁇ 2 is measured between the intersecting vector 206 and the virtual gaze direction gaze direction D 2 .
- the method may then proceed to block 620 .
- the electronic control unit determines the final target position 112 based on the object signal generated by the object detection system 130 and the intersecting vector 206 . Specifically, the electronic control unit 102 determines one or more objects based on the object signal that intersect with the intersecting vector 206 , and identifies the object that intersects with the vector as the final target position 112 . The method may then terminate.
- embodiments described herein are directed to vehicle systems that determine a target position that a driver or passenger of the vehicle is attempting to point or otherwise gesture towards. More specifically, the disclosed system determines the final target position based on the first target position and the second target position, where the first target position is based on the virtual eye position and the second target position is based on the direction that the driver is pointing towards.
- the accuracy of the first target position and the second target position are based on the specific identity of the driver, since people tend to point and look at objects using different mannerisms. Thus, in some instances the first target position may be more accurate when compared to the second target position, and in other instances the second target position may be more accurate when compared to the first target position.
- the disclosed system takes advantage of the fluctuations in accuracy by weighting the target position having the greatest accuracy more heavily when determining the final target position.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Ophthalmology & Optometry (AREA)
- Chemical & Material Sciences (AREA)
- Combustion & Propulsion (AREA)
- Transportation (AREA)
- Mechanical Engineering (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- Embodiments described herein generally relate to vehicles and, more specifically, to vehicles having systems that determine a target position that a user is gesturing towards, where the user's gaze is directed towards a location other than the target position.
- Some types of vehicle systems may allow a driver or passenger to provide input without manipulating buttons or other tactile inputs. More specifically, the vehicle may receive nonverbal communication from an individual using hand gestures. The vehicle includes sensors to detect the movement and position of an individual's hand, and determines the information the individual is attempting to convey based on the movement and position of the hand. However, some challenges may exist in certain situations that can limit the system's ability to interpret some of the information expressed by an individual's hands.
- In addition to systems that allow for a driver to provide input based on hand gestures, some vehicles include an eye-tracking system that is capable of tracking the driver's gaze direction. The driver's gaze direction may be used to determine the driver's level of awareness as he or she is operating the vehicle. The driver normally directs his or her attention towards the environment located in front of the vehicle. Therefore, drivers typically direct their gaze away from the front of the road for only a few moments at a time while operating the vehicle. For example, a driver may turn his or her head to the side and look out of one of the side windows of a vehicle, but only for a few seconds. Accordingly, eye-tracking systems are limited in their ability to determine commands based on the gaze direction of the driver.
- In one embodiment, a vehicle includes a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position. The vehicle also includes a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user. The vehicle also includes one or more processors and one or more non-transitory memory modules communicatively coupled to the one or more processors. The processors store machine-readable instructions that, when executed, cause the processors to determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system. The first point and the second point define a pointing axis of the hand of the user. The processors are also caused to calculate a virtual eye position based at least in part on a first point located on the hand of the user and the actual eye position. The processors are further caused to calculate a first target position based on the virtual eye position and a second target position based on the pointing axis of the hand of the user. The processors are caused to determine the final target position based on the first target position and the second target position. Finally, the processors are caused to control at least one vehicle system based at least in part on the final target position.
- In another embodiment, a vehicle includes a user detection system configured to output a gesture signal in response to a hand of a user performing at least one gesture to indicate a final target position. The vehicle also includes a user gaze monitoring system configured to output an eye location signal that indicates an actual eye position of the user. The vehicle also includes one or more processors and one or more non-transitory memory modules communicatively coupled to the one or more processors. The processors store machine-readable instructions that, when executed, cause the one or more processors to determine a first point and a second point located on the hand of the user based at least in part on the gesture signal from the user detection system. The first point and the second point define a pointing axis of the hand of the user. The processors are also caused to calculate a virtual eye position based at least in part on a first point located on the hand of the user and the actual eye position. The processors are further caused to calculate a first target position based on the virtual eye position and a second target position based on the pointing axis of the hand of the user. The processors are caused to compare a first accuracy associated with the first target position with a second accuracy associated with the second target position. In response to determining the first accuracy and the second accuracy are equal to one another, the processors are caused to determine the final target position based on weighting the first target position and the second target position equally. In response to determining the first accuracy and the second accuracy are unequal to one another, the processors are caused to determine the final target position based on weighting the first target position and the second target position unequally. Finally, the processors are caused to control at least one vehicle system based at least in part on the final target position.
- In yet another embodiment, a method for determining a final target position that a user of a vehicle is gesturing towards is disclosed. The method includes determining, by a computer, a first point and a second point located on a hand of the user based at least in part on a gesture signal generated by a user detection system. The first point and the second point define a pointing axis of the hand of the user. The method also includes calculating a virtual eye position based at least in part on a first point located on the hand of the user and an actual eye position of the user, where an eye location signal generated by a user gaze monitoring system generates the actual eye position. The method also includes calculating, by the computer, a first target position based on the virtual eye position. The method further includes calculating, by the computer, a second target position based on the pointing axis of the hand of the user. The method also includes determining the final target position based on the first target position and the second target position by the computer. Finally, the method includes controlling at least one vehicle system based at least in part on the final target position.
- These and additional features provided by the embodiments of the present disclosure will be more fully understood in view of the following detailed description, in conjunction with the drawings.
- The embodiments set forth in the drawings are illustrative and exemplary in nature and not intended to limit the disclosure. The following detailed description of the illustrative embodiments can be understood when read in conjunction with the following drawings, where like structure is indicated with like reference numerals and in which:
-
FIG. 1 depicts a schematic view of an example vehicle configured to determine a final target position based on a virtual eye position and a direction that a hand of a user is pointing towards, according to one or more embodiments shown and described herein; -
FIG. 2 depicts a schematic top view of a user of the vehicle pointing towards a first target position, according to one or more embodiments shown and described herein; -
FIG. 3 is a perspective view of the user driving the vehicle and pointing towards the first target position, according to one or more embodiments shown and described herein; -
FIG. 4 is a perspective view of the user pointing towards the object and also directing his or her gaze towards the first target position, according to one or more embodiments shown and described herein; -
FIG. 5 depicts a schematic top view of the user and includes the first target position, a second target position, and the final target position, according to one or more embodiments shown and described herein; and -
FIG. 6 depicts a flowchart of an example method for determining the final target position, according to one or more embodiments shown and described herein. - The embodiments disclosed herein are directed to vehicle systems and methods to determine a target position that a user is pointing towards when his or her gaze is directed in a location other than the target position. In the embodiments as described in the present disclosure, the user is a driver of the vehicle. However, it should be appreciated that that the disclosure may also be applied towards a passenger of the vehicle as well.
- When operating a vehicle, the driver normally directs his or her attention towards the environment located in front of the vehicle, and may turn his or her head momentarily to glance at objects located on either side of the vehicle. The driver may point or gesture towards a particular object or direction located in the surrounding using his or her hands, where the gesture may be used to convey information to one or more vehicle systems. For example, the driver may point towards a specific landmark, and the landmark is conveyed to a GPS or an interactive dialog system. However, since the driver is usually directing his or her attention towards the front of the vehicle, the direction of the driver's gaze may not align with the direction indicated by the hand.
- In some instances, the direction indicated by a particular driver's hand may be relatively accurate. That is, a driver may tend to point in a direction that is relatively close to or on the target position. However, another driver may not point towards the target position as accurately because he or she is gesturing towards the target position in a different manner. In other words, the accuracy of the direction intended by the driver's hand depends on upon the specific pointing style used by the driver. For example, some people may point to an object using their index finger. However, other people may point using a different finger. Some people may even point using another object that they grasp in their hand. The object grasped by the hand may be, for example, a writing instrument like a pen or an object located within proximity of the driver (e.g., a pair of sunglasses or the driver's cellular telephone).
- The disclosed system also determines a simulated or virtual position of the driver's eyes. The virtual position of the driver's eyes is aligned with the target object and the tip of the driver's finger, and is used to determine the target position that the driver is pointing towards. As mentioned above, the driver is looking in a direction other than the target location (e.g., towards the front of the vehicle). The virtual eye position may be calculated by the location of the hand of the driver and the actual position of the driver's eyes. The location of the driver is determined based on a gesture signal that is generated by an object detection system. The system determines an actual position of the driver's eyes (i.e., the real position of the eyes) based at least in part on an eye location signal generated by a driver gaze monitoring system.
- The system then determines the virtual eye position by rotating the actual position of the driver's eyes about a vertical axis of the driver until a midpoint measured between the driver's eyes is aligned with the driver's fingertip. The position of the driver's eyes when aligned with the driver's fingertip represents the virtual eye position. Similar to the pointing direction indicated by the user, the virtual eye position may also vary in accuracy between different users. That is, the virtual eye position corresponding to one particular individual may be more accurate or less accurate than the virtual eye position that corresponds to another individual.
- Once the virtual eye position is determined, the system may determine a directional vector that is oriented towards a first target position. The first target position represents the target position calculated based on the virtual eye approach. The system also determines a directional vector pointing towards a second target position. The second target positon is based on the pointing direction that the driver's hand is gesturing towards. The system then calculates an actual or final target position that the driver is gesturing towards by combining the first target position with the second target position.
- In some embodiments, the first target position and the second target position are weighted equally to determine the final target position. Specifically, the first target position and the second target position are weighted equally when both the first target position and the second target position provide equal accuracies between one another. In another embodiment the first target position and the second target position are weighted unequally. More specifically, the first target position may be weighted more heavily if the virtual eye approach provides a more accurate result when compared to the driver's hand. Similarly, the second target position may be weighted more heavily if the driver's hand provides a more accurate result when compared to the virtual eye approach.
- Various embodiments of vehicles and methods for determining a target position that the user is gesturing towards are disclosed below.
- Referring now to
FIG. 1 , an embodiment of avehicle 100 is schematically depicted. Thevehicle 100 may be any passenger vehicle such as, for example, a terrestrial, aquatic, and/or airborne vehicle. Thevehicle 100 includes acommunication path 104, anelectronic control unit 102, anobject detection system 130, adriver detection system 140, a driver gaze monitoring system 170 (which may also be referred to as a user gaze monitoring system), and one ormore vehicle systems 180. Theelectronic control unit 102 includes one ormore processors 105 and one ormore memory modules 106. Referring toFIGS. 1 and 5 , theelectronic control unit 102 is configured to calculate afinal target position 112 that is located in an environment surrounding thevehicle 100. Thefinal target position 112 may represent an object that the driver is pointing at such as a person, an animal, a landmark, another vehicle, a building, and the like. The location represented by thefinal target position 112 represents where a driver of thevehicle 100 is attempting to point or gesture towards using his or herhand 110, and is based on two discrete target locations, namely a first target location T1 and a second target location T2 that are calculated based on two different approaches that are described in greater detail below. - Referring to
FIG. 1 , thecommunication path 104 provides data interconnectivity between various modules disposed within thevehicle 100. Specifically, each of the modules may operate as a node that may send and/or receive data. In some embodiments, thecommunication path 104 includes a conductive material that permits the transmission of electrical data signals to processors, memories, sensors, and actuators throughout thevehicle 100. In some embodiments, thecommunication path 104 can be a bus, such as, for example, a LIN bus, a CAN bus, a VAN bus, and the like. In some embodiments, thecommunication path 104 may be wireless and/or an optical waveguide. Components that are communicatively coupled may include components capable of exchanging data signals with one another such as, for example, electrical signals via conductive medium, electromagnetic signals via air, optical signals via optical waveguides, and the like. - Accordingly, the
communication path 104 may be formed from any medium that is capable of transmitting a signal such as, for example, conductive wires, conductive traces, optical waveguides, or the like. Moreover, thecommunication path 104 may be formed from a combination of mediums capable of transmitting signals. In some embodiments, thecommunication path 104 comprises a combination of conductive traces, conductive wires, connectors, and buses that cooperate to permit the transmission of electrical data signals to components such as processors, memories, sensors, input devices, output devices, and communication devices. Additionally, it is noted that the term “signal” means a waveform (e.g., electrical, optical, magnetic, mechanical or electromagnetic), such as DC, AC, sinusoidal-wave, triangular-wave, square-wave, vibration, and the like, capable of traveling through a medium. - Still referring to
FIG. 1 , theelectronic control unit 102 may be any computing device. For instance theelectronic control unit 102 may be any type of vehicle-installed, handheld, laptop, or other form of single computing device, or may be composed of multiple computing devices. Theelectronic control unit 102 includes one ormore processors 105 for controlling operations of theelectronic control unit 102. The one ormore processors 105 may include any device capable of executing machine-readable instructions stored on a non-transitory computer-readable medium. Accordingly, each of the one ormore processors 105 may include a controller, an integrated circuit, a microchip, a computer, and/or any other computing device. - The
electronic control unit 102 further includes one ormore memory modules 106 communicatively coupled to the one ormore processors 105. The one ormore memory modules 106 may be configured as volatile and/or nonvolatile memory and, as such, may include random access memory (including SRAM, DRAM, and/or other types of RAM), flash memory, secure digital (SD) memory, registers, compact discs (CD), digital versatile discs (DVD), and/or other types of non-transitory computer-readable mediums. Depending on the particular embodiment, these non-transitory computer-readable mediums may reside within theelectronic control unit 102 and/or external to theelectronic control unit 102. The one ormore memory modules 106 may be configured to store one or more pieces of logic as described in more detail below. The embodiments described herein may utilize a distributed computing arrangement to perform any portion of the logic described herein. - Embodiments of the present disclosure include logic that includes machine-readable instructions and/or an algorithm written in any programming language of any generation (e.g., 1GL, 2GL, 3GL, 4GL, and/or 5GL) such as, machine language that may be directly executed by the processor, assembly language, object-oriented programming (OOP), scripting languages, microcode, etc., that may be compiled or assembled into machine-readable instructions and stored on a machine-readable medium. Similarly, the logic and/or algorithm may be written in a hardware description language (HDL), such as logic implemented via either a field-programmable gate array (FPGA) configuration or an application-specific integrated circuit (ASIC), and their equivalents. Accordingly, the logic may be implemented in any conventional computer programming language, as pre-programmed hardware elements, and/or as a combination of hardware and software components. Logic stored on the one or
more memory modules 106 may include, for example, object recognition logic, speech recognition logic, risk determination logic, notification generation logic, and autonomous vehicle control logic. Thus, theelectronic control unit 102 includes logic to calculate the virtual eye position and then determine the location of thefinal target position 112. - As noted above, the logic stored on the one or
more memory modules 106 may include object recognition logic. The object recognition logic may include any known or yet-to-be-developed object recognition algorithms that may be utilized to detect objects within an environment. Example object recognition algorithms include, but are not limited to, edge detection algorithms, corner detection algorithms, blob detection algorithms, and feature description algorithms (e.g., scale-invariant feature transform (“SIFT”), speeded up robust features (“SURF”), gradient location and orientation histogram (“GLOH”), and the like). The logic stored on the electronic control unit may also include speech recognition logic used to detect the words spoken by the driver and/or passengers within thevehicle 100. Any known or yet-to-be-developed speech recognition algorithms may be used for the speech recognition logic. - In the embodiments described herein, the one or
more memory modules 106 and the one ormore processors 105 are integral with theelectronic control unit 102. However, it is noted that theelectronic control unit 102, the one ormore memory modules 106, and the one ormore processors 105 may be discrete components communicatively coupled to one another without departing from the scope of the present disclosure. As an example and not a limitation, one or more processors and one ormore memory modules 106 of theelectronic control unit 102 may be remote to thevehicle 100. For example, thevehicle 100 may be in wireless communication (e.g., using a wireless communication system) with a remote server storing logic and data that is configured to perform at least some of the functionalities described herein. - The
object detection system 130 is communicatively coupled to theelectronic control unit 102 over thecommunication path 104. Theobject detection system 130 may include any device configured to detect the presence of an object within the surrounding environment of thevehicle 100. More specifically, theobject detection system 130 is configured to detect the presence of an object within the vicinity of thevehicle 100. Theobject detection system 130 may include anobject detection sensor 132 configured to output an object signal indicative of the presence of one or more objects within the vicinity of thevehicle 100. Based on the object signal of theobject detection sensor 132, theelectronic control unit 102 may execute object recognition logic to detect an object and classify the detected object into a classification. Theobject detection sensor 132 may include, but is not limited to, a camera, a LiDAR sensor, a RADAR sensor, a sonar sensor, a proximity sensor, and the like. In some embodiments, theobject detection system 130 includes more than oneobject detection sensor 132. - The
driver detection system 140 is communicatively coupled to theelectronic control unit 102 over thecommunication path 104. Thedriver detection system 140 may include any device configured to detect the presence, movements and/or actions of the driver of thevehicle 100. As such, thedriver detection system 140 may include one or moredriver detection sensors 142. Thedriver detection sensors 142 may include, but are not limited to, a camera with a field of view on a face and the surrounding area of the driver. Referring toFIGS. 1 and 5 , thedriver detection sensors 142 are configured to output a gesture signal that is indicative of afirst point 200 located on the driver'shand 110. The gesture signal is created in response to the driver raising his or herhand 110 away from asteering wheel 144 of thevehicle 100 attempting to point or otherwise draw attention to the second target position T2. That is, the gesture signal is created in response to the driver gesturing towards one or more objects using his or herhand 110. - In the exemplary embodiment as shown in
FIGS. 2, 3 and 5 , thefirst point 200 of the driver'shand 110 represents atip 208 of one of thefingers 114 located on thehand 110 of the driver. More specifically, thefirst point 200 is at thetip 208 of the driver'sindex finger 114. This is because the driver is gesturing towards the second target position T2 by pointing his or herindex finger 114 along a pointing direction P. Accordingly, theelectronic control unit 102 determines that the driver is pointing towards the second target position T2 using his or herhand 110, and sets thefirst point 200 as thetip 208 of the driver'sindex finger 114. Although theindex finger 114 is described, it should be appreciated that the disclosure is not limited to gesturing towards the second target position T2 using an index finger of an individual. Instead, the driver may point using one or more digits of the hand 110 (e.g., the thumb, ring finger, etc.), where the tip of one or more digits of the user'shand 110 represent thefirst point 200. - In another embodiment, the
first point 200 may not be located on the driver'shand 110. Instead, the driver may grasp an article or item with his or her hand. The driver may then use the object to point towards an object. For example, instead of using his or herindex finger 114 the driver may point towards the second target position T2 using elongated articles such as a pen, a pencil, a stick, or the like. Therefore, thetip 208 of the driver'sfinger 114 is actually represented by an endpoint of the article that is being grasped by the driver. Specifically, a distal end of the article may be considered thetip 208 of the driver'sfinger 114. The distal end of the article points towards the second target position T2 (FIG. 5 ) and a proximate end of the article is grasped by the driver. Furthermore, although elongated objects such as pencils and pens are described, the disclosure is not limited to elongated object. The driver may also gesture using any other article that is sized to be grasped by thehand 110 of the driver, and can be manipulated to point towards a particular direction. For example, the article may be an item that the driver usually wears or keeps nearby while driving thevehicle 100 such as, but not limited to, a cellular telephone, a pair of sunglasses, and the like. - Referring to both
FIGS. 1 and 2 , the drivergaze monitoring system 170 is communicatively coupled to theelectronic control unit 102 over thecommunication path 104. The drivergaze monitoring system 170 may include any device configured to monitor the driver's gaze associated movement. More specifically, the drivergaze monitoring system 170 includes one or more devices to monitor a direction and motion of theeyes 116 of the driver relative to his or herhead 126. As such, the drivergaze monitoring system 170 may include one or moreeye tracking systems 172 configured to output a direction signal indicative of the driver's gaze direction, which is referred to inFIG. 2 as D1. Theeye tracking systems 172 may also output an eye location signal that indicates an actual position of theeyes 116 of the driver. As an example and not a limitation, the eye tracking system may include one or more cameras or some other optical sensors for detecting light reflected back from the driver's eyes. As a non-limiting example, the light reflected back from the driver's eyes may be near infrared light, which may range from about 700 nanometers to 2500 nanometers in the electromagnetic spectrum. -
FIG. 2 is a schematic top view illustrating ahead 126 and afinger 114 of the driver, where the driver is pointing towards a first target position T1.FIG. 3 is a perspective view of the driver gesturing towards the first target position T1 shown inFIG. 2 . As seen inFIGS. 2 and 3 , the driver is attempting to gesture towards the first target position T1 using one of thefingers 114 on his or herhand 110, where the driver is pointing or gesturing in the pointing direction P (FIG. 3 ), but the driver's gaze is directed towards the driver's gaze direction D1. The driver's gaze direction D1 is pointing towards a location that is different than the first target position T1. For example, the driver's gaze direction D1 is directed towards a drivingroute 118 of thevehicle 100. - Referring to
FIGS. 1 and 5 , both the first target position T1 as well as the second target position T2 are shown, where the second target position T2 is based on the direction indicated by anindex finger 114 of the driver's hand 110 (i.e., the pointing direction P). Theelectronic control unit 102 calculates the first target position T1 based on the virtual eye position, the second target position T2, and then determines thefinal target position 112 based on the first target position T1 and the second target position T2. More specifically, in the non-limiting embodiment as shown inFIG. 5 , thefinal target position 112 is determined by dividing an angle A created by a first vector representing the driver's gaze direction D1 and a second vector representing the pointing direction P into two different sub-angles. More specifically, the angle A is divided into a first sub-angle α1 and a second sub-angle α2. AlthoughFIG. 5 illustrates the angles α1 and α2 being equal to one another, it should be appreciated that the disclosure is not limited to this particular configuration. That is, the angles α1 and α2 may also be unequal as well. - Calculation of the first target position T1 (
FIG. 2 ) based on the virtual eye position will now be described. Referring to bothFIGS. 1 and 2 , in response to receiving the eye location signal from theeye tracking systems 172, theelectronic control unit 102 determines the position of theeyes 116 of the driver. Theelectronic control unit 102 then calculates a midpoint M1 between the twoeyes 116 of the driver. More specifically, as seen inFIG. 2 theelectronic control unit 102 calculates the midpoint M1 by determining a first line segment S1 relative to the driver'shead 126. The first line segment S1 intersects botheyes 116 of the driver, is tangent with respect to the driver'shead 126, and is substantially perpendicular with respect to the vector representing the driver's gaze direction D1. The midpoint M1 represents a middle point between the twoeyes 116 of the driver that intersects the first line segment S1. - In the embodiment as shown in
FIG. 2 , the driver'shead 126 is drawn as a circle for purposes of simplicity and clarity in the illustration. However, it should be appreciated that the illustration shown inFIG. 2 is not limiting. Theelectronic control unit 102 may calculate the line segment S1 and the midpoint M1 by modeling the driver's head using any number of shapes or profiles. - The
electronic control unit 102 may then calculate the virtual eye position, which represents a simulated or virtual position of the driver's eyes. As seen inFIG. 2 , a set ofvirtual eyes 120 of the driver is shown. Thevirtual eyes 120 are directed towards thefirst point 200 on the driver'shand 110. That is,FIG. 2 illustrates the pair ofvirtual eyes 120 directed towards thetip 208 of thefinger 114. Theelectronic control unit 102 may calculate the position of the set ofvirtual eyes 120 by rotating the actual position of the driver'seyes 116 about a vertical axis A-A of thehead 126 of the driver until the midpoint M1 measured between the driver'seyes 116 is aligned with thefirst point 200 located at thetip 208 of the driver'sfinger 114 when viewed in a plane substantially perpendicular to the axis A-A of the driver (i.e., a horizontal plane). In other words, theelectronic control unit 102 simulates the driver turning his or herhead 126 such that thevirtual eyes 120 are positioned to see the first target position T1. It should be appreciated that the vertical axis A-A of the driver'shead 126 represents the vertical axis of the driver's body. The vertical extends from the head to the feet of a human's body. Furthermore, it should be understood that humans are capable of rotating about their respective vertical axis to perform a three hundred and sixty degree turn. -
FIG. 4 illustrates the driver directing his or her gaze towards the first target position T1 while also pointing in the pointing direction P. In other words, the driver has turned his or herhead 126 such that his or hereyes 116 are aligned with the first target position T1. More specifically, the driver'seyes 116 are directed towards a third gaze direction D3 that is aligned with thetip 208 of the driver'sfinger 114 and is directed towards the first target position T1. Therefore, the driver is indicating the general direction of the first target position T1 by pointing his or herfinger 114 towards the direction P. The driver is also gazing past thetip 208 of his or herfinger 114 towards the first target position T1. - Turning back to
FIGS. 2 and 3 , a midpoint between thevirtual eyes 120 of the driver is indicated as midpoint M2. Theelectronic control unit 102 calculates the midpoint M2 by first determining a second line segment S2 relative to the driver'shead 126. The second line segment S2 intersects both of thevirtual eyes 120 and is tangent with respect to the driver'shead 126. The midpoint M2 represents a middle point between the twovirtual eyes 120 that intersect the second line segment S2. In response to determining the midpoint M2 between thevirtual eyes 120, the electronic control unit calculates a virtual gaze direction D2 of the driver. Theelectronic control unit 102 calculates the virtual gaze direction D2 by determining a vector that originates at the midpoint M2 and intersects thefirst point 200 of the driver'shand 110. As seen inFIG. 2 , the vector is substantially perpendicular with respect to the second line segment S2. - It should be appreciated that the virtual gaze direction D2 represents a simulated gaze direction of the driver, where the driver is looking towards the first target position T1. That is, the driver's
head 126 is turned such that theeyes 116 would be directed towards the first target position T1. More specifically, the virtual gaze direction D2 ofFIG. 2 is the same as the third gaze direction D3 illustrated inFIG. 3 . The third gaze direction D3 is aligned with thetip 208 of the driver'sfinger 114 and is directed towards the first target position T1. - Referring back to
FIG. 1 , theelectronic control unit 102 determines the first target position T1 based on the virtual gaze direction D2. In some embodiments, theelectronic control unit 102 determines one or more objects indicated by the object signal generated by theobject detection system 130 and the virtual gaze direction D2. Theelectronic control unit 102 determines the first target position T1 by identifying one or more objects that intersect with the vector associated with the virtual gaze direction D2. In such embodiments, theelectronic control unit 102 may execute object recognition logic to classify the objects. In one embodiment, theelectronic control unit 102 may employ the object recognition logic configured to recognize specific landmarks. Some examples of landmarks that may be recognized include, but are not limited to, the Golden Gate Bridge in San Francisco, the Empire State Building in New York City, the Eiffel Tower in Paris, and the like. - Calculation of the second target position T2 based on the position of the driver's
hand 110 will now be described. Referring to bothFIGS. 1 and 5 , theelectronic control unit 102 receives as input the gesture signal generated by thedriver detection sensors 142. The gesture signal indicates at least two points located on the driver'shand 110. More specifically, the gesture signal indicates thefirst point 200 as well as asecond point 202, where bothpoints finger 114 of the driver'shand 110. As mentioned above, thefirst point 200 is located at thetip 208 of the driver'sfinger 114. Thesecond point 202 is located along the driver'sfinger 114, at a location below thetip 208, and is collinear with respect to thefirst point 200. Thepoints line segment 204, where theline segment 204 represents a pointing axis of the driver'sfinger 114. The pointing axis indicates the direction that the driver'shand 110 is pointing or gesturing towards. More specifically, in the embodiment as illustrated inFIG. 5 the pointing axis (i.e., the line segment 204) is aligned with the driver'sindex finger 114. Theelectronic control unit 102 then extends theline segment 204 representing the pointing axis beyond thetip 208 of the driver'sfinger 114 to determine a vector that represents the pointing direction P. - The
electronic control unit 102 then determines the second target position T2 based on the pointing direction P. In some embodiments, theelectronic control unit 102 may determine the second target position T2 based on one or more objects indicated by the object signal generated by theobject detection system 130 and the vector representing the pointing direction P. Theelectronic control unit 102 determines the second target position T2 by identifying one or more objects that intersect with the vector associated with the pointing direction P2. Once the first target position T1 and the second target position T2 are calculated, theelectronic control unit 102 may then determine thefinal target position 112. - Continuing to refer to both
FIGS. 1 and 5 , theelectronic control unit 102 may first determine an angle A that is created by the first vector representing the virtual gaze direction D2 and the second vector representing the pointing direction P intersecting one another. Theelectronic control unit 102 then calculates an intersectingvector 206 that divides the angle A into two separate angles. Specifically, the angle A is divided into the first sub-angle α1 and the second sub-angle α2. The first sub-angle α1 is measured between the vector representing the pointing direction P and the intersectingvector 206, and the second sub-angle α2 is measured between the intersectingvector 206 and the driver's gaze direction D2. - In one embodiment, the
electronic control unit 102 compares a first accuracy associated with the first target position T1 with a second accuracy associated with the second target position T2, and determines that both the first accuracy and the second accuracy are equal to one another. Accordingly, the first target position T1 and the second target position T2 are weighted equally to determine thefinal target position 112. In the non-limiting embodiment as shown inFIG. 5 , the intersectingvector 206 bisects the angle A to create two equal sub-angles α1 and α2. That is, the intersectingvector 206 divides the angle A into two equal parts. However, it is to be appreciated that the position of the intersectingvector 206 is not limited to the embodiment as shown inFIG. 5 . - In another embodiment, the
electronic control unit 102 compares the first accuracy associated with the first target position T1 with the second accuracy associated with the second target position T2, and determines that the first accuracy is unequal with respect to the second accuracy. In response to determining the first target position T1 and the second target position T2 provide unequal accuracies, theelectronic control unit 102 determines that the intersectingvector 206 divides the angle A into two unequal sub-angles α1 and α2. More specifically, in response to determining the first accuracy is greater accuracy than the second accuracy, theelectronic control unit 102 divides the angle A by the intersectingvector 206 such that the first sub-angle α1 is greater than the second sub-angle α2. It should be appreciated that the intersectingvector 206 is directed towards thefinal target position 112. Accordingly, the virtual gaze direction D2 influences thefinal target position 112 more than the pointing direction P. Similarly, in response to determining the second accuracy is greater than the first accuracy, theelectronic control unit 102 divides the angle A by the intersectingvector 206 such that the first sub-angle α1 is less than the second sub-angle α2 Accordingly, the intersectingvector 206 is drawn closer towards the pointing direction P. - The weight or influence given to either the pointing direction P or the virtual gaze direction D2 when calculating the
final target position 112 is based at least in part by the particular identity of the driver. That is, the weight or influence of the pointing direction P and the virtual gaze direction D2 may vary between individuals. This is because different people point and look towards an object in a variety of configurations or position. More specifically, the accuracy of the virtual gaze direction D2 varies based on the identity of the driver since some people may tend to look directly at an object of interest while driving, while others do not look at the object directly. The accuracy of the pointing direction P is also variable since different people gesture towards an object by their hand using a variety of approaches. For instance, some people may gesture towards an object or position by pointing their index finger towards the object (i.e., as shown in the figures). However, other individuals may point towards an object may gesture towards an object using a different finger of their hand. Furthermore, as explained above, some individuals may even use another object grasp by their hand to gesture in a particular direction. - Referring to
FIGS. 1 and 5 , in response to determining that the pointing direction P and the virtual gaze direction D2 provide equal accuracies, theelectronic control module 102 weights the pointing direction P and the virtual gaze direction D2 equally. In response to determining the pointing direction P provides greater accuracy when compared to the virtual gaze direction D2, theelectronic control unit 102 weights the pointing direction P more than the virtual gaze direction D2. Accordingly, the intersectingvector 206 inFIG. 5 is positioned closer to the pointing direction P and the second target T2. Similarly, in response to determining the virtual gaze direction D2 provides more accuracy than the pointing direction P, theelectronic control unit 102 weights the virtual gaze direction D2 more than the pointing direction P. The intersectingvector 206 is also located closer towards the virtual gaze direction D2 and the first target location T1. - Referring to
FIGS. 1 and 5 , thefinal target position 112 is determined based on the object signal generated by theobject detection system 130 and the intersectingvector 206. Specifically, theelectronic control unit 102 determines one or more objects based on the object signal that intersect with the intersectingvector 206, and identifies the object that intersects with the vector as thefinal target position 112. The object may be a person, another vehicle, a building, and the like. Once thefinal target position 112 is determined, the virtual gaze direction D2, thefinal target position 112, and the object classified using object recognition logic are communicated to one ormore vehicle systems 180 from theelectronic control unit 102 via thecommunication path 104. Thevehicle systems 180 may be controlled based on at least one of the virtual gaze direction D2, thefinal target position 112, and the object classified using object recognition logic. - The
vehicle systems 180 may include, but are not limited to, a GPS system or an interactive dialog system. An interactive dialog system converses with the driver using text, speech, gestures, haptics, and the like. In one embodiment, the interactive dialog system may assist the driver with identifying objects located in thefinal target position 112. For example, if the driver points towards the Empire State Building in New York City and asks “What am I pointing to?”, then the interactive dialog system would answer back “The Empire State Building”. - Referring now to
FIG. 6 , aflowchart 600 depicting a method for determining thefinal target position 112 is graphically depicted. As explained above, thefinal target position 112 is based on the virtual eye position as well as the direction that the driver points towards using his or herhand 110. It should be understood that embodiments are not limited by the order of steps of theflowchart 600 ofFIG. 6 . - Referring generally to
FIGS. 1, 2, 3, and 6 , inblock 602 of theflowchart 600 theelectronic control unit 102 receives the gesture signal generated by thedriver detection system 140. The gesture signal indicates that the driver is raising his or herhand 110 away from thesteering wheel 144 of thevehicle 100 to point or otherwise draw attention to an object or a position located within the environment surrounding thevehicle 100. The method may then proceed todecision block 604. - In
decision block 604, theelectronic control unit 102 determines if the driver's gaze is directed towards thefinal target position 112. More specifically, theelectronic control unit 102 determines if the driver's gaze direction D1 is directed towards thefinal target position 112 based on the direction signal generated by the drivergaze monitoring system 170 and thefirst point 200 on the driver'shand 110. If the driver's gaze direction D1 intersects thefirst point 200, then theelectronic control unit 102 determines that the driver's gaze direction D1 is directed towards thefinal target position 112. The method may then terminate. However, if theelectronic control unit 102 determines that the driver's gaze direction D1 does not intersect thefirst point 200 on the driver'shand 110, then the method may proceed to block 606. - In
block 606, theelectronic control unit 102 determines the position of the driver's eyes 116 (FIG. 2 ) based on the eye location signal generated by the drivergaze monitoring system 170. The method may then proceed to block 608. - In
block 608, theelectronic control unit 102 determines thefirst point 200 on the driver's hand 110 (FIG. 2 ) based on the gesture signal generated by thedriver detection system 140. In the embodiment as illustrated, thefirst point 200 represents thetip 208 of the driver'sindex finger 114. However, as explained above, the disclosure is not limited to the driver'sindex finger 114. In fact, thefirst point 200 may be an object that the driver grasps in his or her hand such as a pen, a pencil, a pair of sunglasses, and the like. The method may then proceed to block 610. - In
block 610, theelectronic control unit 102 determines the position of thevirtual eyes 120. Referring toFIG. 2 , thevirtual eyes 120 are calculated by rotating the actual position of the driver'seyes 116 about the vertical axis A-A of the driver'shead 126 until the midpoint M1 measured between the driver'seyes 116 is aligned with thefirst point 200 of the driver'sfinger 114. The method may then proceed to block 612. - In
block 612, theelectronic control unit 102 calculates the virtual gaze direction D2 by determining a vector that originates at the midpoint M2 and intersects thefirst point 200 of the driver'shand 110. The method may then proceed to block 614. - In
block 614, theelectronic control unit 102 determines the second target position T2 based on the pointing direction P. In some embodiments, theelectronic control unit 102 may also determine one or more objects based on the object signal generated by theobject detection system 130 and the vector representing the pointing direction P. More specifically, the object or objects indicated by the object signal that intersect with the vector as the second target position T2 may represent the second target position T2. The method may then proceed to block 616. - In
block 616, theelectronic control unit 102 determines the second target position T2 as seen inFIG. 5 . The second target position T2 is determined based on the pointing direction P that indicates the direction that the driver is gesturing towards. More specifically, as seen inFIG. 5 the pointing direction P is collinear with respect to the pointing axis of the driver's finger 114 (the pointing axis is illustrated inFIG. 5 as the line segment 204). The method may then proceed to block 618. - In
block 618, theelectronic control unit 102 determines the intersectingvector 206 shown inFIG. 5 . The intersectingvector 206 divides the angle A into the first sub-angle α1 and the second sub-angle α2. The first sub-angle α1 is measured between the vector representing the pointing direction P and the intersectingvector 206, and the second sub-angle α2 is measured between the intersectingvector 206 and the virtual gaze direction gaze direction D2. The method may then proceed to block 620. - In
block 620, the electronic control unit determines thefinal target position 112 based on the object signal generated by theobject detection system 130 and the intersectingvector 206. Specifically, theelectronic control unit 102 determines one or more objects based on the object signal that intersect with the intersectingvector 206, and identifies the object that intersects with the vector as thefinal target position 112. The method may then terminate. - It should now be understood that embodiments described herein are directed to vehicle systems that determine a target position that a driver or passenger of the vehicle is attempting to point or otherwise gesture towards. More specifically, the disclosed system determines the final target position based on the first target position and the second target position, where the first target position is based on the virtual eye position and the second target position is based on the direction that the driver is pointing towards. The accuracy of the first target position and the second target position are based on the specific identity of the driver, since people tend to point and look at objects using different mannerisms. Thus, in some instances the first target position may be more accurate when compared to the second target position, and in other instances the second target position may be more accurate when compared to the first target position. The disclosed system takes advantage of the fluctuations in accuracy by weighting the target position having the greatest accuracy more heavily when determining the final target position.
- While particular embodiments have been illustrated and described herein, it should be understood that various other changes and modifications may be made without departing from the spirit and scope of the claimed subject matter. Moreover, although various aspects of the claimed subject matter have been described herein, such aspects need not be utilized in combination. It is therefore intended that the appended claims cover all such changes and modifications that are within the scope of the claimed subject matter.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/877,906 US10706300B2 (en) | 2018-01-23 | 2018-01-23 | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/877,906 US10706300B2 (en) | 2018-01-23 | 2018-01-23 | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction |
Publications (2)
Publication Number | Publication Date |
---|---|
US20190228243A1 true US20190228243A1 (en) | 2019-07-25 |
US10706300B2 US10706300B2 (en) | 2020-07-07 |
Family
ID=67298685
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/877,906 Active 2038-10-19 US10706300B2 (en) | 2018-01-23 | 2018-01-23 | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction |
Country Status (1)
Country | Link |
---|---|
US (1) | US10706300B2 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190228242A1 (en) * | 2018-01-23 | 2019-07-25 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position |
US10706300B2 (en) * | 2018-01-23 | 2020-07-07 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction |
US10817068B2 (en) * | 2018-01-23 | 2020-10-27 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining target based on selecting a virtual eye position or a pointing direction |
US20210056764A1 (en) * | 2018-05-22 | 2021-02-25 | Magic Leap, Inc. | Transmodal input fusion for a wearable system |
EP3920083A1 (en) * | 2020-06-05 | 2021-12-08 | Aptiv Technologies Limited | Pointing to non-mapped objects outside a vehicle |
US11338678B2 (en) * | 2018-01-30 | 2022-05-24 | Toyota Jidosha Kabushiki Kaisha | Vehicle display device |
EP4002064A1 (en) * | 2020-11-18 | 2022-05-25 | XRSpace CO., LTD. | Method and system for showing a cursor for user interaction on a display device |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6702390B2 (en) | 2018-10-09 | 2020-06-03 | トヨタ自動車株式会社 | Vehicle drive control device, vehicle-mounted electronic control unit, learned model, machine learning system, vehicle drive control method, electronic control unit manufacturing method, and output parameter calculation device |
Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100283722A1 (en) * | 2009-05-08 | 2010-11-11 | Sony Ericsson Mobile Communications Ab | Electronic apparatus including a coordinate input surface and method for controlling such an electronic apparatus |
US20110028372A1 (en) * | 2008-03-28 | 2011-02-03 | Novozymes A/S | Triggered Release System |
US20110267265A1 (en) * | 2010-04-30 | 2011-11-03 | Verizon Patent And Licensing, Inc. | Spatial-input-based cursor projection systems and methods |
US20140184494A1 (en) * | 2012-12-31 | 2014-07-03 | Giedrius Tomas Burachas | User Centric Interface for Interaction with Visual Display that Recognizes User Intentions |
US20140226002A1 (en) * | 2011-07-25 | 2014-08-14 | Leica Geosystems Ag | Measuring device that can be operated without contact and control method for such a measuring device |
US20140361977A1 (en) * | 2013-06-07 | 2014-12-11 | Sony Computer Entertainment Inc. | Image rendering responsive to user actions in head mounted display |
US20150135144A1 (en) * | 2012-04-23 | 2015-05-14 | VTouch Co., Ltd. | Apparatus for obtaining virtual 3d object information without requiring pointer |
US20150277700A1 (en) * | 2013-04-12 | 2015-10-01 | Usens, Inc. | System and method for providing graphical user interface |
US20160179205A1 (en) * | 2013-06-27 | 2016-06-23 | Eyesight Mobile Technologies Ltd. | Systems and methods of direct pointing detection for interaction with a digital device |
US20160335475A1 (en) * | 2014-02-04 | 2016-11-17 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. | 3d image analyzer for determining the gaze direction |
US20170068322A1 (en) * | 2015-09-04 | 2017-03-09 | Eyesight Mobile Technologies Ltd. | Gesture recognition control device |
US20170147078A1 (en) * | 2015-09-15 | 2017-05-25 | Alexander van Laack | Method for the target recognition of target objects |
US20170329411A1 (en) * | 2016-05-13 | 2017-11-16 | Alexander van Laack | Method for the Contactless Shifting of Visual Information |
US9823744B2 (en) * | 2014-05-09 | 2017-11-21 | Google Inc. | Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects |
US20180088676A1 (en) * | 2015-04-16 | 2018-03-29 | Rakuten, Inc. | Gesture interface |
US20190228242A1 (en) * | 2018-01-23 | 2019-07-25 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position |
US20190227635A1 (en) * | 2018-01-23 | 2019-07-25 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on selecting a virtual eye position or a pointing direction |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20070188022A1 (en) | 2003-09-02 | 2007-08-16 | Toshiyuki Itabashi | Input device |
JP6106402B2 (en) | 2012-10-23 | 2017-03-29 | 矢崎エナジーシステム株式会社 | In-vehicle device |
US10452136B2 (en) | 2014-10-13 | 2019-10-22 | Thomson Licensing | Method for controlling the displaying of text for aiding reading on a display device, and apparatus adapted for carrying out the method, computer program, and computer readable storage medium |
JP6604151B2 (en) | 2015-11-09 | 2019-11-13 | 三菱自動車工業株式会社 | Speech recognition control system |
WO2018000200A1 (en) | 2016-06-28 | 2018-01-04 | 华为技术有限公司 | Terminal for controlling electronic device and processing method therefor |
US20190318181A1 (en) | 2016-07-01 | 2019-10-17 | Eyesight Mobile Technologies Ltd. | System and method for driver monitoring |
KR102463712B1 (en) | 2017-11-24 | 2022-11-08 | 현대자동차주식회사 | Virtual touch recognition apparatus and method for correcting recognition error thereof |
US11167771B2 (en) | 2018-01-05 | 2021-11-09 | Magna Mirrors Of America, Inc. | Vehicular gesture monitoring system |
US10706300B2 (en) * | 2018-01-23 | 2020-07-07 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction |
-
2018
- 2018-01-23 US US15/877,906 patent/US10706300B2/en active Active
Patent Citations (17)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110028372A1 (en) * | 2008-03-28 | 2011-02-03 | Novozymes A/S | Triggered Release System |
US20100283722A1 (en) * | 2009-05-08 | 2010-11-11 | Sony Ericsson Mobile Communications Ab | Electronic apparatus including a coordinate input surface and method for controlling such an electronic apparatus |
US20110267265A1 (en) * | 2010-04-30 | 2011-11-03 | Verizon Patent And Licensing, Inc. | Spatial-input-based cursor projection systems and methods |
US20140226002A1 (en) * | 2011-07-25 | 2014-08-14 | Leica Geosystems Ag | Measuring device that can be operated without contact and control method for such a measuring device |
US20150135144A1 (en) * | 2012-04-23 | 2015-05-14 | VTouch Co., Ltd. | Apparatus for obtaining virtual 3d object information without requiring pointer |
US20140184494A1 (en) * | 2012-12-31 | 2014-07-03 | Giedrius Tomas Burachas | User Centric Interface for Interaction with Visual Display that Recognizes User Intentions |
US20150277700A1 (en) * | 2013-04-12 | 2015-10-01 | Usens, Inc. | System and method for providing graphical user interface |
US20140361977A1 (en) * | 2013-06-07 | 2014-12-11 | Sony Computer Entertainment Inc. | Image rendering responsive to user actions in head mounted display |
US20160179205A1 (en) * | 2013-06-27 | 2016-06-23 | Eyesight Mobile Technologies Ltd. | Systems and methods of direct pointing detection for interaction with a digital device |
US20160335475A1 (en) * | 2014-02-04 | 2016-11-17 | Fraunhofer-Gesellschaft Zur Foerderung Der Angewandten Forschung E.V. | 3d image analyzer for determining the gaze direction |
US9823744B2 (en) * | 2014-05-09 | 2017-11-21 | Google Inc. | Systems and methods for biomechanically-based eye signals for interacting with real and virtual objects |
US20180088676A1 (en) * | 2015-04-16 | 2018-03-29 | Rakuten, Inc. | Gesture interface |
US20170068322A1 (en) * | 2015-09-04 | 2017-03-09 | Eyesight Mobile Technologies Ltd. | Gesture recognition control device |
US20170147078A1 (en) * | 2015-09-15 | 2017-05-25 | Alexander van Laack | Method for the target recognition of target objects |
US20170329411A1 (en) * | 2016-05-13 | 2017-11-16 | Alexander van Laack | Method for the Contactless Shifting of Visual Information |
US20190228242A1 (en) * | 2018-01-23 | 2019-07-25 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position |
US20190227635A1 (en) * | 2018-01-23 | 2019-07-25 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on selecting a virtual eye position or a pointing direction |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190228242A1 (en) * | 2018-01-23 | 2019-07-25 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position |
US10706300B2 (en) * | 2018-01-23 | 2020-07-07 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction |
US10817068B2 (en) * | 2018-01-23 | 2020-10-27 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining target based on selecting a virtual eye position or a pointing direction |
US10853674B2 (en) * | 2018-01-23 | 2020-12-01 | Toyota Research Institute, Inc. | Vehicle systems and methods for determining a gaze target based on a virtual eye position |
US11338678B2 (en) * | 2018-01-30 | 2022-05-24 | Toyota Jidosha Kabushiki Kaisha | Vehicle display device |
US11820225B2 (en) | 2018-01-30 | 2023-11-21 | Toyota Jidosha Kabushiki Kaisha | Vehicle display device |
US20210056764A1 (en) * | 2018-05-22 | 2021-02-25 | Magic Leap, Inc. | Transmodal input fusion for a wearable system |
US11983823B2 (en) * | 2018-05-22 | 2024-05-14 | Magic Leap, Inc. | Transmodal input fusion for a wearable system |
EP3920083A1 (en) * | 2020-06-05 | 2021-12-08 | Aptiv Technologies Limited | Pointing to non-mapped objects outside a vehicle |
EP4002064A1 (en) * | 2020-11-18 | 2022-05-25 | XRSpace CO., LTD. | Method and system for showing a cursor for user interaction on a display device |
Also Published As
Publication number | Publication date |
---|---|
US10706300B2 (en) | 2020-07-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10706300B2 (en) | Vehicle systems and methods for determining a target based on a virtual eye position and a pointing direction | |
US10853674B2 (en) | Vehicle systems and methods for determining a gaze target based on a virtual eye position | |
US10817068B2 (en) | Vehicle systems and methods for determining target based on selecting a virtual eye position or a pointing direction | |
US20190065872A1 (en) | Behavior recognition apparatus, learning apparatus, and method and program therefor | |
US10890975B2 (en) | Haptic guidance system | |
US9868449B1 (en) | Recognizing in-air gestures of a control object to control a vehicular control system | |
JP4972201B2 (en) | Man-machine interaction based on sign | |
CN112590794A (en) | Method and apparatus for determining an estimate of the ability of a vehicle driver to take over control of a vehicle | |
KR20210135389A (en) | Apparatus for recognizing an obstacle, a vehicle system having the same and method thereof | |
Endres et al. | Geremin" 2D microgestures for drivers based on electric field sensing | |
WO2019028761A1 (en) | Object tracking using depth information | |
Coppola et al. | Social activity recognition on continuous RGB-D video sequences | |
Ouyang et al. | Multiwave: A novel vehicle steering pattern detection method based on smartphones | |
Frigieri et al. | Fast and accurate facial landmark localization in depth images for in-car applications | |
Prasetya et al. | Triangle similarity approach for detecting eyeball movement | |
Aftab et al. | Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects | |
CN105373230A (en) | Gesture recognition method and device of on-board unit | |
CN105759955B (en) | Input device | |
KR20200023670A (en) | Apparatus and Method for Recognizing Gestures and Finger Spellings using Cluster Radar and 3D Convolution Neural Network | |
Kopinski et al. | A real-time applicable 3d gesture recognition system for automobile hmi | |
Kopinski et al. | Neural network based data fusion for hand pose recognition with multiple tof sensors | |
Kopinski et al. | A light-weight real-time applicable hand gesture recognition system for automotive applications | |
Kopinski et al. | Gesture-based human-machine interaction for assistance systems | |
Pech et al. | Real time recognition of non-driving related tasks in the context of highly automated driving | |
KR101870542B1 (en) | Method and apparatus of recognizing a motion |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: TOYOTA RESEARCH INSTITUTE, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YAMAOKA, MASAAKI;HORIUCHI, YUKI;REEL/FRAME:044703/0799 Effective date: 20171215 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: TOYOTA JIDOSHA KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TOYOTA RESEARCH INSTITUTE, INC.;REEL/FRAME:053928/0146 Effective date: 20200921 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |