US20150316998A1 - Touchless human machine interface - Google Patents
Touchless human machine interface Download PDFInfo
- Publication number
- US20150316998A1 US20150316998A1 US14/800,050 US201514800050A US2015316998A1 US 20150316998 A1 US20150316998 A1 US 20150316998A1 US 201514800050 A US201514800050 A US 201514800050A US 2015316998 A1 US2015316998 A1 US 2015316998A1
- Authority
- US
- United States
- Prior art keywords
- finger
- image
- images
- user
- controller
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
- 238000005096 rolling process Methods 0.000 claims description 5
- 238000000034 method Methods 0.000 abstract description 30
- 238000004458 analytical method Methods 0.000 abstract description 3
- 210000003811 finger Anatomy 0.000 description 113
- 210000003813 thumb Anatomy 0.000 description 12
- 238000012545 processing Methods 0.000 description 8
- 230000008859 change Effects 0.000 description 7
- 238000005286 illumination Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000002329 infrared spectrum Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- B60K35/10—
-
- B60K35/29—
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
- G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
- G06F3/044—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by capacitive means
-
- G06K9/00087—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/12—Fingerprints or palmprints
- G06V40/1365—Matching; Classification
-
- B60K2360/11—
-
- B60K2360/146—
-
- B60K2360/1868—
-
- B60K2360/21—
Definitions
- the present invention generally relates to a user interface.
- touch screen technology With the introduction of touch screen technology, users are more comfortable with user interfaces that accept input based on touch interaction with the user. However, a touch screen technology can still be expensive and limits the input to motion that actually contacts the device. In an automotive environment, where a user is confined by a seatbelt, it may not always be convenient to touch the device to provide input.
- a system and method for receiving input from a user includes at least one camera configured to receive an image of a hand of the user and a controller configured to analyze the image and issue a command based on the analysis of the image.
- Camera technology may be used in the vehicle to accept user input while not require the user to physically touch the device.
- the user may make a hand gesture with their hand within the field of view of the camera. Based on the position and orientation of the user's fingers or the finger path over time, the system can identify the gesture based on a plurality of stored gesture models. Further, the cameras may be configured with resolution sufficient to distinguish a finger print on the fingers.
- the finger print may be used for tracking the position and orientation of the finger, as well as, for a security check allowing certain features of the user interface to be unlocked.
- the security check may be for setting option parameters or for even starting the car ignition.
- FIG. 1 is a front view of a system for accepting user input
- FIG. 2 is a side view of the system for accepting user input in FIG. 1 ;
- FIG. 3 is a flowchart illustrating a method for accepting user input
- FIG. 4 Is illustration of an image received by a first camera for accepting user input
- FIG. 5 Is illustration of a subsequent image received by the first camera for accepting user input
- FIG. 6 Is illustration of the finger path and gesture formed by the user in FIG. 4 and FIG. 5 ;
- FIG. 7 Is illustration of the change in position and orientation of the thumb from FIG. 6 ;
- FIG. 8 Is illustration of an image received by a second camera for accepting user input
- FIG. 9 Is illustration of a subsequent image received by the second camera for accepting user input
- FIG. 10 Is illustration of the finger path and gesture formed by the user in FIG. 8 and FIG. 9 ;
- FIG. 11 is a schematic view of one implementation of a controller.
- the system 100 includes a controller 110 and a plurality of cameras 112 , 114 , 116 , 118 .
- the system 100 may also include a display 122 and input controls 124 .
- the cameras 112 , 114 , 116 , 118 may be mounted on a front console 120 of a vehicle.
- the cameras may include a detector with an array of photosensitive elements.
- a CMOS camera may be used.
- a high definition camera may be used and configured to identify detailed features in the image. For example, if a hand is presented to the camera, the camera may be configured with a resolution high enough to distinguish lines on the pad of the finger that comprise a finger print.
- the cameras may be located at different positions along the width of the front console as denoted for example, by cameras 112 , 114 , 118 .
- the cameras may also be mounted at different heights as denoted by cameras 112 , 116 .
- the cameras may be clustered around the center of the console as shown by cameras 112 , 116 , 118 .
- cameras 112 , 116 , 118 are located around the outside of the display 122 on the front console 120 . Further, the cameras are located in the center of the console and are accessible by either the driver of the vehicle or front seat passenger of the vehicle.
- the display 122 may be a light emitting diode (LED) display, liquid crystal display (LCD), or other display appropriate for automotive applications.
- input controls 124 may be provided for receiving user input by the controller 110 .
- the controls 124 may include one or more buttons 126 , knobs 128 , or switches 130 to facilitate providing user input to the controller 110 .
- buttons 126 may be provided within the context of the controls 124 .
- knobs 128 may be provided within the context of the controls 124 .
- the controller 110 , the display 122 , the controls 124 , and the cameras 112 , 114 , 116 , 118 may all be in communication with one another or for example, a communication bus 132 .
- the communications may be multiple separate communication links between each individual device, or a wireless communication link may also be provided between one or more of the above referenced devices.
- the cameras 112 , 114 , 116 , 118 may be configured to view a hand of the user to receive user input. Accordingly, the cameras may capture an image of the hand of the user and further the image of the hand of the user may be displayed on the display 122 . By locating the cameras near the display 122 , the user can view the image of the hand provided on the display 122 , as well as, the user's actual hand. Viewing the image and the hand simultaneously will aid the user in forming gestures that can be understood as user input by the system 100 .
- the controller 110 may also display graphics overlaid on the image of the hand to aid in forming recognizable hand gestures and/or communicate the controller's interpretation of the hand gesture.
- the cameras 112 , 114 , 116 , 118 may have overlapping fields of use such that multiple of the cameras are viewing the hand at the same time. This can also be seen in the side view provided in FIG. 2 . Accordingly, the field of view 220 of the camera 112 overlaps with the field of view 222 of camera 116 such that both cameras view a hand 224 of the user simultaneously and from different vantage points. By using two cameras that view the hand 224 from different vantage points, various stereo vision techniques can be used to determine the location of the hand and orientation of the hand in three dimensional space, as well as, the position and location of fingers 228 and 226 on the hand 224 .
- the controller 110 may then issue a command to one or more vehicle systems as denoted by reference numeral 140 .
- vehicle systems may include climate control systems, entertainment systems, door lock systems, lighting systems, or other well known systems.
- lights 134 , 136 may be provided, for example on the front console, to illuminate the hand during night driving conditions or when ambient illumination is not sufficient for the functioning of the system 100 .
- the lights 134 , 136 are associated with particular cameras 112 and 116 .
- the lighting may be provided at a central location and provided independent of any of the cameras particularly.
- the lights 134 , 136 may be LED illumination, and further, the lights 134 , 136 may provide illumination in the near infrared spectrum. As such the illumination may not be visible to the user, however, the illumination could be readily sensed by the cameras.
- a sensor 138 such as a photo detector may be used to determine if the ambient lighting of a vehicle is above a certain threshold level and as such, may provide a signal to activate the lights 134 , 136 .
- the sensor 138 may have one or more photodiodes located at one or more positions within the vehicle to determine the directionality of the ambient light and as such the controller 110 may light a subset of the lights based on the directionality of the ambient light sensed by the sensor 138 to provide even illumination of the hand to the system 100 .
- a background image is formed by averaging a series of background images. This may be accomplished by acquiring many images over a period of time and averaging the results of the series of background images.
- the background image may include a rolling series of images such that a stored background image is the average of a series of background images over a given time period, for example, the last hour. By using a rolling window of background images, the changing lighting conditions in the vehicle may be accounted for as well as other environmental changes.
- an image is captured by the camera where multiple cameras are used, for example, to determine the three dimensional position and orientation of the hand and/or fingers of the user, then multiple images may be captured such as one image for each camera.
- each image is analyzed.
- the system may determine if the hand is present in the image by comparing the recently acquired image to the background image. For example, the background image may be subtracted from the recently captured image to provide a difference image.
- the difference image may be thresholded and an analysis of the resulting shapes may identify if the shapes contained in the image correspond to the shape of a hand. This may be done by various methods which may be as simple as a pixel count, or as complicated as scoring a model of a representative hand and comparing the shapes and the image to the representative model to determine the likelihood that a hand exists in the image.
- a stored pattern of the finger print of one or more of the fingers may be used to identify the hand.
- the method follows line 320 to block 322 .
- the background series of images is updated and a new background image is stored.
- the method then follows line 324 to block 314 , where the method continues.
- the hand is in the image and a potential gesture exists, then the method follows line 326 to block 328 .
- the hand is located in the image space and/or in three dimensional space using images from multiple cameras.
- the position and orientation of each of the fingers may be determined.
- each finger may be first identified in the image space and then a position of the orientation of the finger determined based on images from multiple cameras using triangulation and other stereo imaging techniques to determine the three dimensional position and orientation of each finger. For example, various models may be used to determine the position and orientation of the fingers including a model of the finger print for each finger.
- another image is captured by one or more of the cameras to image the hand.
- the fingers are tracked in block 334 .
- the fingers may be tracked by utilizing the same steps described above with respect to blocks 328 and 330 .
- the system may use the position and orientation of the fingers in the previous set of images as a starting point and determine the change in position and orientation of the fingers relative to the previous series images.
- the system may determine a three dimensional finger path for each finger and match one or more of the finger paths to a predetermined model of finger paths that represents a gesture.
- a system determines if a gesture is recognized, for example, based on the plurality of finger paths. If a gesture is not recognized in block 336 , the method follows line 338 to block 340 .
- the system determines if the hand has been removed from the image. If the hand has not been removed from the image, the method follows line 342 to block 332 where the method continues. If the hand has been removed from the image in block 340 , the method follows line 346 to block 332 .
- the background image is updated and stored and the method follows line 324 to block 314 , where the method continues.
- the method follows 348 to block 350 .
- the system sends a command that is associated with the recognized gesture to a vehicle subsystem. For example, a gesture where each of the fingers are moved toward one another may cause the controller to send a command to the vehicle entertainment system to change the volume of the vehicle.
- the thumb and pinky may be extended from a closed fist and as such, the vaulted position of the pinky and thumb to the rest of the fingers may indicate a gesture to initiate a phone call based only on the positioning orientation of the fingers relative to one another. Accordingly, the system may send a command to initiate a phone call to a telematics controller.
- image 400 from one camera is provided.
- the image 400 includes a hand 401 within the field of view of the camera.
- An index finger is denoted by arrow 410 and various image processing techniques may be used to determine the location of the index finger within the image as denoted by reference numeral 420 .
- arrow 412 denotes a middle finger
- arrow 414 denotes a ring finger
- arrow 416 denotes a pinky finger and the system may use various image processing techniques to determine the locations of such fingers including a middle finger position 422 , a ring finger position 424 , and a pinky finger position 426 .
- thumb is denoted by arrow 418 and the thumb position is denoted by reference number 428 . It is further noted that while the two dimensional positions of the fingers can be determined in two dimensional image space, the components of multiple images from different cameras may be used to determine the three dimensional positions and orientations of each of the fingers through various stereo imaging processing techniques.
- a subsequent image 500 from the same camera as FIG. 4 is provided.
- the system may identify each of the fingers on the hand 401 .
- a second position for each of the fingers can be determined in the subsequent image in particular, an index finger position 502 , a middle finger position 522 , a ring finger position 524 , a pink finger position 526 , and a thumb position 528 are shown within the image.
- FIG. 6 a superimposed image of FIG. 4 and FIG. 5 is provided showing the initial finger positions 420 , 422 , 424 , 426 , 428 and the subsequent finger positions 520 , 522 , 524 , 526 , 528 .
- the difference between the finger positions can be calculated in two dimensional or three dimensional space.
- the change in the finger positions from the initial finger position to the subsequent finger position can be used to calculate a finger path denoted by line 620 . While line 620 represents an index finger path, the same process may be used for each of the fingers thereby creating a middle finger path 622 , a ring finger path 624 , a pinky finger path 626 and a thumb finger path 628 . As illustrated in FIG.
- the fingers were moved from a more open spread position to a close together pinched position.
- this gesture imitates a person grasping a knob or squeezing an object.
- the system may interpret the gesture as making something smaller, for example, the volume, or may interpret this combination of finger paths as grasping a knob, for example, to change a radio station and a subsequent rotating of the finger positions while grasping as an indication of turning a representative knob to a higher or lower station.
- the system may determine whether the user input is provided by the driver of the vehicle or the passenger of a vehicle based on which hand (right hand or left hand) is presented to the camera.
- the fingerprint matching may also be used to identify each of the fingers on the hand as well as which user is providing input to the system. Further, the fingerprints may be used to provide access to certain features or unlock certain functions of the car using the fingerprint scan as a security feature.
- the system determines the position and orientation of one or more of the fingers based on a model that is matched to the finger.
- the initial and subsequent images may again be used to determine a finger path, as well as, a set of positions and orientations of each finger in three dimensional space. Again, this may also be done using stereo vision techniques and also using multiple sets of sequential images for multiple cameras.
- an initial position and orientation of a finger may be determined as denoted by reference numeral 712 .
- a diameter of the finger 710 or change in diameter may be used to determine a model of the finger as denoted by axis 714 and geometry 716 .
- the model may include just a position and principle axis 714 or may also include a geometric model as denoted by shape 716 including a cylindrical model or a more complex three dimensional model.
- a second position and orientation of the finger may be determined as denoted by reference numeral 732 .
- an outline of the finger 730 in the subsequent image may be used to determine an axis 734 and model 736 for the finger in the subsequent image.
- the gesture may be determined based on both the initial position and orientation and subsequent position and orientations in a set of sequential images for each camera to determine a gesture based on such three dimensional position and orientation information, as well as, contiguous finger path information.
- a second set of sequential images is provided from another camera which may be used in combination with the first set of images 4 - 5 from the first camera.
- the features of the first set of images can be combined with the features of the second set of images using triangulation or stereo vision techniques to determine the set of three dimensional positions and orientations of each finger in three dimensional space, as well as, three dimensional finger paths for each finger which may be used in combination to identify a gesture and issue a corresponding command.
- index finger 410 the middle finger 412 , the ring finger 414 , the pinky finger 416 , and the thumb 418 are identified and an index finger position 810 , a middle finger position 812 , a ring finger position 814 , a pinky finger position 816 , and a thumb position 818 are determined based on the image 800 .
- image 800 is provided from a different position and angle than images 400 and 500 from the first camera to facilitate various triangulation and stereo imaging techniques to determine the position of and orientation of each finger in three dimensional space using a combination of images taken from different cameras at the same time.
- the fingers in image 800 are presented with the pads of the fingers presented to the camera.
- the finger print from each finger may also be used to identify the location and orientation of each finger.
- a subsequent image 900 is provided from the same camera that acquired image 800 and also at the same subsequent time as image 500 .
- the system determines a position of each finger, specifically an index finger position 910 , a middle finger position 912 , a ring finger position 914 , a pinky finger position 916 , and a thumb position 918 .
- each of the initial and subsequent positions for each finger is used to determine a finger path for each corresponding finger.
- a finger path for each corresponding finger For example, an index finger path 1010 , a middle finger path 1012 , a ring finger path 1014 , a pinky finger path 1016 , and a thumb finger path 1018 .
- Each of the finger paths as well as the position and orientations of each finger may be compared to various gesture models to determine a corresponding gesture 1020 . Accordingly, the system may issue a command corresponding to the gesture based on one or more combinations of the position orientation and finger paths of each finger.
- the computer system 1100 includes a processor 1110 for executing instructions such as those described in the methods discussed above.
- the instructions may be stored in a computer readable medium such as memory 1112 or storage devices 1114 , for example a disk drive, CD, or DVD.
- the computer may include a display controller 1116 responsive to instructions to generate a textual or graphical display on a display device 1118 , for example a computer monitor.
- the processor 1110 may communicate with a network controller 1120 to communicate data or instructions to other systems, for example other general computer systems.
- the network controller 1120 may communicate over Ethernet or other known protocols to distribute processing or provide remote access to information over a variety of network topologies, including local area networks, wide area networks, the Internet, or other commonly used network topologies.
- dedicated hardware implementations such as application specific integrated circuits, programmable logic arrays and other hardware devices, can be constructed to implement one or more of the methods described herein.
- Applications that may include the apparatus and systems of various embodiments can broadly include a variety of electronic and computer systems.
- One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices with related control and data signals that can be communicated between and through the modules, or as portions of an application-specific integrated circuit. Accordingly, the present system encompasses software, firmware, and hardware implementations.
- the methods described herein may be implemented by software programs executable by a computer system.
- implementations can include distributed processing, component/object distributed processing, and parallel processing.
- virtual computer system processing can be constructed to implement one or more of the methods or functionality as described herein.
- computer-readable medium includes a single medium or multiple media, such as a centralized or distributed database, and/or associated caches and servers that store one or more sets of instructions.
- computer-readable medium shall also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a processor or that cause a computer system to perform any one or more of the methods or operations disclosed herein.
Abstract
A system and method for receiving input from a user is provided. The system includes at least one camera configured to receive an image of a hand of the user and a controller configured to analyze the image and issue a command based on the analysis of the image.
Description
- 1. Field of the Invention
- The present invention generally relates to a user interface.
- 2. Description of Related Art
- With the introduction of touch screen technology, users are more comfortable with user interfaces that accept input based on touch interaction with the user. However, a touch screen technology can still be expensive and limits the input to motion that actually contacts the device. In an automotive environment, where a user is confined by a seatbelt, it may not always be convenient to touch the device to provide input.
- In view of the above, it is apparent that there is a need for an improved user interface.
- A system and method for receiving input from a user is provided. The system includes at least one camera configured to receive an image of a hand of the user and a controller configured to analyze the image and issue a command based on the analysis of the image.
- Camera technology may be used in the vehicle to accept user input while not require the user to physically touch the device. The user may make a hand gesture with their hand within the field of view of the camera. Based on the position and orientation of the user's fingers or the finger path over time, the system can identify the gesture based on a plurality of stored gesture models. Further, the cameras may be configured with resolution sufficient to distinguish a finger print on the fingers. The finger print may be used for tracking the position and orientation of the finger, as well as, for a security check allowing certain features of the user interface to be unlocked. The security check may be for setting option parameters or for even starting the car ignition.
- Further objects, features and advantages of this invention will become readily apparent to persons skilled in the art after a review of the following description, with reference to the drawings and claims that are appended to and form a part of this specification.
-
FIG. 1 is a front view of a system for accepting user input; -
FIG. 2 is a side view of the system for accepting user input inFIG. 1 ; -
FIG. 3 is a flowchart illustrating a method for accepting user input; -
FIG. 4 . Is illustration of an image received by a first camera for accepting user input; -
FIG. 5 . Is illustration of a subsequent image received by the first camera for accepting user input; -
FIG. 6 . Is illustration of the finger path and gesture formed by the user inFIG. 4 andFIG. 5 ; -
FIG. 7 Is illustration of the change in position and orientation of the thumb fromFIG. 6 ; -
FIG. 8 Is illustration of an image received by a second camera for accepting user input; -
FIG. 9 . Is illustration of a subsequent image received by the second camera for accepting user input; -
FIG. 10 . Is illustration of the finger path and gesture formed by the user inFIG. 8 andFIG. 9 ; and -
FIG. 11 is a schematic view of one implementation of a controller. - Now referring to
FIG. 1 , asystem 100 is provided for receiving user input. Thesystem 100 includes acontroller 110 and a plurality ofcameras system 100 may also include adisplay 122 andinput controls 124. Thecameras front console 120 of a vehicle. The cameras may include a detector with an array of photosensitive elements. In one example, a CMOS camera may be used. Further, a high definition camera may be used and configured to identify detailed features in the image. For example, if a hand is presented to the camera, the camera may be configured with a resolution high enough to distinguish lines on the pad of the finger that comprise a finger print. - The cameras may be located at different positions along the width of the front console as denoted for example, by
cameras cameras cameras cameras display 122 on thefront console 120. Further, the cameras are located in the center of the console and are accessible by either the driver of the vehicle or front seat passenger of the vehicle. - The
display 122 may be a light emitting diode (LED) display, liquid crystal display (LCD), or other display appropriate for automotive applications. Further,input controls 124 may be provided for receiving user input by thecontroller 110. Thecontrols 124 may include one ormore buttons 126,knobs 128, orswitches 130 to facilitate providing user input to thecontroller 110. Although it is also understood that other known physical controls may be provided within the context of thecontrols 124. - As such, the
controller 110, thedisplay 122, thecontrols 124, and thecameras communication bus 132. Although it is also understood that the communications may be multiple separate communication links between each individual device, or a wireless communication link may also be provided between one or more of the above referenced devices. - The
cameras display 122. By locating the cameras near thedisplay 122, the user can view the image of the hand provided on thedisplay 122, as well as, the user's actual hand. Viewing the image and the hand simultaneously will aid the user in forming gestures that can be understood as user input by thesystem 100. Thecontroller 110 may also display graphics overlaid on the image of the hand to aid in forming recognizable hand gestures and/or communicate the controller's interpretation of the hand gesture. Further, thecameras FIG. 2 . Accordingly, the field ofview 220 of thecamera 112 overlaps with the field ofview 222 ofcamera 116 such that both cameras view ahand 224 of the user simultaneously and from different vantage points. By using two cameras that view thehand 224 from different vantage points, various stereo vision techniques can be used to determine the location of the hand and orientation of the hand in three dimensional space, as well as, the position and location offingers hand 224. Using successive images taken by the cameras, allows motion of the hand, as well as, motion of individual fingers to be identified and interpreted as a gesture by thecontroller 110. Upon interpreting a gesture, thecontroller 110 may then issue a command to one or more vehicle systems as denoted byreference numeral 140. The vehicle systems may include climate control systems, entertainment systems, door lock systems, lighting systems, or other well known systems. - Further,
lights system 100. In one example, thelights particular cameras lights lights sensor 138 such as a photo detector may be used to determine if the ambient lighting of a vehicle is above a certain threshold level and as such, may provide a signal to activate thelights sensor 138 may have one or more photodiodes located at one or more positions within the vehicle to determine the directionality of the ambient light and as such thecontroller 110 may light a subset of the lights based on the directionality of the ambient light sensed by thesensor 138 to provide even illumination of the hand to thesystem 100. - Now referring to
FIG. 3 , a method for providing user input is illustrated. The method starts inblock 310. Inblock 312, a background image is formed by averaging a series of background images. This may be accomplished by acquiring many images over a period of time and averaging the results of the series of background images. In one implementation, the background image may include a rolling series of images such that a stored background image is the average of a series of background images over a given time period, for example, the last hour. By using a rolling window of background images, the changing lighting conditions in the vehicle may be accounted for as well as other environmental changes. Inblock 314, an image is captured by the camera where multiple cameras are used, for example, to determine the three dimensional position and orientation of the hand and/or fingers of the user, then multiple images may be captured such as one image for each camera. - In
block 316, each image is analyzed. The system may determine if the hand is present in the image by comparing the recently acquired image to the background image. For example, the background image may be subtracted from the recently captured image to provide a difference image. The difference image may be thresholded and an analysis of the resulting shapes may identify if the shapes contained in the image correspond to the shape of a hand. This may be done by various methods which may be as simple as a pixel count, or as complicated as scoring a model of a representative hand and comparing the shapes and the image to the representative model to determine the likelihood that a hand exists in the image. In another example, a stored pattern of the finger print of one or more of the fingers may be used to identify the hand. If the system does not determine that a hand exists in the image, the method followsline 320 to block 322. Inblock 322, the background series of images is updated and a new background image is stored. The method then followsline 324 to block 314, where the method continues. If the hand is in the image and a potential gesture exists, then the method followsline 326 to block 328. Inblock 328, the hand is located in the image space and/or in three dimensional space using images from multiple cameras. Inblock 330, the position and orientation of each of the fingers may be determined. Accordingly, each finger may be first identified in the image space and then a position of the orientation of the finger determined based on images from multiple cameras using triangulation and other stereo imaging techniques to determine the three dimensional position and orientation of each finger. For example, various models may be used to determine the position and orientation of the fingers including a model of the finger print for each finger. Inblock 332, another image is captured by one or more of the cameras to image the hand. The fingers are tracked inblock 334. The fingers may be tracked by utilizing the same steps described above with respect toblocks block 336, a system determines if a gesture is recognized, for example, based on the plurality of finger paths. If a gesture is not recognized inblock 336, the method followsline 338 to block 340. In block 340, the system determines if the hand has been removed from the image. If the hand has not been removed from the image, the method followsline 342 to block 332 where the method continues. If the hand has been removed from the image in block 340, the method followsline 346 to block 332. Inblock 332, the background image is updated and stored and the method followsline 324 to block 314, where the method continues. - Now referring again to block 336, if in
block 336 the gesture is recognized based on the positions and orientations of the fingers, or more particularly based on the plurality of finger paths, the method follows 348 to block 350. Inblock 350, the system sends a command that is associated with the recognized gesture to a vehicle subsystem. For example, a gesture where each of the fingers are moved toward one another may cause the controller to send a command to the vehicle entertainment system to change the volume of the vehicle. Another example, the thumb and pinky may be extended from a closed fist and as such, the vaulted position of the pinky and thumb to the rest of the fingers may indicate a gesture to initiate a phone call based only on the positioning orientation of the fingers relative to one another. Accordingly, the system may send a command to initiate a phone call to a telematics controller. - Now referring to
FIG. 4 ,image 400 from one camera is provided. Theimage 400 includes ahand 401 within the field of view of the camera. An index finger is denoted byarrow 410 and various image processing techniques may be used to determine the location of the index finger within the image as denoted byreference numeral 420. Similarly,arrow 412 denotes a middle finger,arrow 414 denotes a ring finger, andarrow 416 denotes a pinky finger and the system may use various image processing techniques to determine the locations of such fingers including amiddle finger position 422, aring finger position 424, and apinky finger position 426. Further, the thumb is denoted byarrow 418 and the thumb position is denoted byreference number 428. It is further noted that while the two dimensional positions of the fingers can be determined in two dimensional image space, the components of multiple images from different cameras may be used to determine the three dimensional positions and orientations of each of the fingers through various stereo imaging processing techniques. - Now referring to
FIG. 5 , asubsequent image 500 from the same camera asFIG. 4 is provided. InFIG. 5 , again the system may identify each of the fingers on thehand 401. For example, theindex finger 410, themiddle finger 412, thering finger 414, thepinky finger 416 and thethumb 418. Further, a second position for each of the fingers can be determined in the subsequent image in particular, an index finger position 502, amiddle finger position 522, aring finger position 524, apink finger position 526, and athumb position 528 are shown within the image. - Now referring to
FIG. 6 , a superimposed image ofFIG. 4 andFIG. 5 is provided showing the initial finger positions 420, 422, 424, 426, 428 and the subsequent finger positions 520, 522, 524, 526, 528. The difference between the finger positions can be calculated in two dimensional or three dimensional space. The change in the finger positions from the initial finger position to the subsequent finger position can be used to calculate a finger path denoted byline 620. Whileline 620 represents an index finger path, the same process may be used for each of the fingers thereby creating amiddle finger path 622, aring finger path 624, apinky finger path 626 and athumb finger path 628. As illustrated inFIG. 6 , the fingers were moved from a more open spread position to a close together pinched position. For example, this gesture imitates a person grasping a knob or squeezing an object. The system may interpret the gesture as making something smaller, for example, the volume, or may interpret this combination of finger paths as grasping a knob, for example, to change a radio station and a subsequent rotating of the finger positions while grasping as an indication of turning a representative knob to a higher or lower station. In addition, the system may determine whether the user input is provided by the driver of the vehicle or the passenger of a vehicle based on which hand (right hand or left hand) is presented to the camera. Further, if a very high resolution camera is used, it may be possible to image the fingerprint of one or more of the fingers and determine the location of the fingers based on a matching of a stored fingerprint within the field of the view of the camera. In this scenario, the fingerprint matching may also be used to identify each of the fingers on the hand as well as which user is providing input to the system. Further, the fingerprints may be used to provide access to certain features or unlock certain functions of the car using the fingerprint scan as a security feature. - Now referring to
FIG. 7 , one example is provided where the system determines the position and orientation of one or more of the fingers based on a model that is matched to the finger. In this implementation, the initial and subsequent images may again be used to determine a finger path, as well as, a set of positions and orientations of each finger in three dimensional space. Again, this may also be done using stereo vision techniques and also using multiple sets of sequential images for multiple cameras. In this example, an initial position and orientation of a finger may be determined as denoted byreference numeral 712. A diameter of thefinger 710 or change in diameter may be used to determine a model of the finger as denoted byaxis 714 andgeometry 716. The model may include just a position andprinciple axis 714 or may also include a geometric model as denoted byshape 716 including a cylindrical model or a more complex three dimensional model. In a subsequent image, again a second position and orientation of the finger may be determined as denoted byreference numeral 732. Again, an outline of thefinger 730 in the subsequent image may be used to determine anaxis 734 andmodel 736 for the finger in the subsequent image. Accordingly, the gesture may be determined based on both the initial position and orientation and subsequent position and orientations in a set of sequential images for each camera to determine a gesture based on such three dimensional position and orientation information, as well as, contiguous finger path information. - Now referring to
FIG. 8-9 , a second set of sequential images is provided from another camera which may be used in combination with the first set of images 4-5 from the first camera. The features of the first set of images can be combined with the features of the second set of images using triangulation or stereo vision techniques to determine the set of three dimensional positions and orientations of each finger in three dimensional space, as well as, three dimensional finger paths for each finger which may be used in combination to identify a gesture and issue a corresponding command. - Now referring to
FIG. 8 , again theindex finger 410, themiddle finger 412, thering finger 414, thepinky finger 416, and thethumb 418 are identified and anindex finger position 810, amiddle finger position 812, aring finger position 814, apinky finger position 816, and athumb position 818 are determined based on theimage 800. Further,image 800 is provided from a different position and angle thanimages image 800 are presented with the pads of the fingers presented to the camera. As discussed previously, the finger print from each finger may also be used to identify the location and orientation of each finger. - Now referring to
FIG. 9 , asubsequent image 900 is provided from the same camera that acquiredimage 800 and also at the same subsequent time asimage 500. Again, the system determines a position of each finger, specifically anindex finger position 910, amiddle finger position 912, aring finger position 914, apinky finger position 916, and athumb position 918. - Now referring to
FIG. 10 , each of the initial and subsequent positions for each finger is used to determine a finger path for each corresponding finger. For example, anindex finger path 1010, amiddle finger path 1012, aring finger path 1014, apinky finger path 1016, and athumb finger path 1018. Each of the finger paths as well as the position and orientations of each finger may be compared to various gesture models to determine acorresponding gesture 1020. Accordingly, the system may issue a command corresponding to the gesture based on one or more combinations of the position orientation and finger paths of each finger. - Any of the controllers, devices, modules, servers, or engines described may be implemented in one or more computer systems. One exemplary system is provided in
FIG. 11 . Thecomputer system 1100 includes aprocessor 1110 for executing instructions such as those described in the methods discussed above. The instructions may be stored in a computer readable medium such asmemory 1112 orstorage devices 1114, for example a disk drive, CD, or DVD. The computer may include adisplay controller 1116 responsive to instructions to generate a textual or graphical display on adisplay device 1118, for example a computer monitor. In addition, theprocessor 1110 may communicate with anetwork controller 1120 to communicate data or instructions to other systems, for example other general computer systems. Thenetwork controller 1120 may communicate over Ethernet or other known protocols to distribute processing or provide remote access to information over a variety of network topologies, including local area networks, wide area networks, the Internet, or other commonly used network topologies. - In other embodiments, dedicated hardware implementations, such as application specific integrated circuits, programmable logic arrays and other hardware devices, can be constructed to implement one or more of the methods described herein. Applications that may include the apparatus and systems of various embodiments can broadly include a variety of electronic and computer systems. One or more embodiments described herein may implement functions using two or more specific interconnected hardware modules or devices with related control and data signals that can be communicated between and through the modules, or as portions of an application-specific integrated circuit. Accordingly, the present system encompasses software, firmware, and hardware implementations.
- In accordance with various embodiments of the present disclosure, the methods described herein may be implemented by software programs executable by a computer system. Further, in an exemplary, non-limited embodiment, implementations can include distributed processing, component/object distributed processing, and parallel processing. Alternatively, virtual computer system processing can be constructed to implement one or more of the methods or functionality as described herein.
- Further, the methods described herein may be embodied in a computer-readable medium. The term “computer-readable medium” includes a single medium or multiple media, such as a centralized or distributed database, and/or associated caches and servers that store one or more sets of instructions. The term “computer-readable medium” shall also include any medium that is capable of storing, encoding or carrying a set of instructions for execution by a processor or that cause a computer system to perform any one or more of the methods or operations disclosed herein.
- As a person skilled in the art will readily appreciate, the above description is meant as an illustration of the principles of the invention. This description is not intended to limit the scope or application of the invention in that the invention is susceptible to modification, variation and change, without departing from spirit of the invention, as defined in the following claims.
Claims (18)
1. In a non-transitory computer readable storage medium having stored therein instructions executable by a programmed processor for receiving input to a vehicle system from a user of the system, the storage medium comprising instructions for:
capturing a series of background images of the interior of a vehicle over a predetermined time to produce a rolling window of background images;
after the rolling window of background averages is produced, capturing a plurality of captured images of background and an image of a finger print of a user's finger;
subtracting the rolling window of background images from the plurality of captured images of the background and the image of a finger print of the user's finger to produce a plurality of images of the finger print of user's finger;
comparing the plurality of images of the finger print of the user's finger to a stored finger print pattern to identify the user's finger and to determine a position, orientation, and three-dimensional path of the user's finger;
analyzing the position, orientation, and three dimensional path of the user's finger to determine a gesture;
issuing a command to the vehicle system responsive to the determined gesture.
2. The computer readable storage medium according to claim 1 , wherein the image is one of a plurality of images captured by a plurality of cameras on a front console of a vehicle.
3. The computer readable storage medium according to claim 1 , further comprising instructions for determining positions of at least two fingers relative to one another in each image of a series of images to determine a gesture based on each finger path of the at least two fingers in three dimensional space.
4. The computer readable storage medium according to claim 3 , further comprising instructions for determining the positions of the at least two fingers relative to one another in each image based on triangulation and the position and orientation of each camera.
5. The computer readable storage medium according to claim 4 , further comprising instructions for determining the positions of the at least two fingers by generating an average of a plurality of background images and calculating a difference between each image in the series of images and the average of the plurality of background images.
6. A system for receiving input from a user of a vehicle, the system comprising:
at least two cameras configured to selectively capture images, a first image being an image of a background, a second image being captured after the first image and including both background and an image of a finger print of a user's finger;
and
a controller coupled to the cameras and configured to subtract the first image from the second image to produce a difference image comprising an image of the finger print of the user, the controller being additionally configured to compare the image of the finger print to a stored pattern and determine from said comparison the identity of a finger of a user, the controller being additionally configured to determine the location of the finger of the user within a field of view of at least one of the cameras, the controller being additionally configured to determine a three dimensional path of the finger based on a model of the finger print in three dimensional space without the finger contacting a surface.
7. The system according to claim 6 , wherein the controller is configured to match captured images of finger prints obtained from at least two fingers, to stored images of finger prints and wherein the controller is configured to determine the positions of the at least two fingers relative to one another in a plurality of successive images based on triangulation.
8. The system according to claim 7 , wherein the controller is configured to determine the positions of the at least two fingers on the hand by first generating an average of a plurality of background images that do not include the hand and thereafter calculating a difference between each image in the series of images that include the hand and captured images of the finger prints of the at least two fingers.
9. A system for receiving input command gestures from a user in a vehicle, the system comprising:
at least one camera located in the vehicle and configured to receive a first series of images of a background obtained from inside the vehicle and, after the at least one camera receives the first series of images of background, receive a second image of a finger print of a user's finger captured in the first series of images; and
a controller configured to average the first series of images of the background captured prior to capturing the second image and to subtract the average of the first series of images of the background from the second image to produce an image of the finger print of a user's finger, the controller additionally configured to determine the identity of the user's finger by comparing the image of the finger print of the user's finger to a stored pattern of a finger print and issue a command based on whether the image of the finger print of the user's finger matches the stored pattern of a finger print, the controller being additionally configured to determine a three dimensional path of the finger based on a model of the finger print to determine a gesture of the finger in three dimensional space.
10. The system according to claim 9 , wherein the controller is configured to determine the position of one of more fingers in the image.
11. The system according to claim 10 , further comprising a display screen in the vehicle, and wherein the at least one camera is a plurality of cameras located in predetermined positions inside the vehicle, each camera being configured and located to capture a series of images of finger prints, the controller being configured to identify the positions of the one or more fingers responsive to a determination that images of the finger prints of one or more fingers match stored patterns of finger prints.
12. The system according to claim 11 , wherein the controller is configured to analyze each series of images to identify positions of the one or more fingers in three dimensional space.
13. The system according to claim 11 , wherein the controller is configured to determine the positions of at least two fingers relative to one another.
14. The system according to claim 9 , wherein the controller is configured to determine the positions of at least one finger in each image of a series of images to determine a finger path.
15. The system according to claim 9 , wherein the controller is configured to determine the positions of at least two fingers relative to one another in each image of a series of images to determine a gesture based on each finger path of the at least two fingers in three dimensional space.
16. The system according to claim 9 , wherein the controller is configured to determine the positions of at least two fingers relative to one another in each image based on triangulation and a position and orientation of each camera.
17. The system according to claim 9 , wherein the controller is configured to determine the positions of at least two fingers by generating an average of a plurality of background images and calculating a difference between each image of a series of images and the average of the plurality of background images.
18. The system according to claim 9 , wherein the controller is configured to match a stored model of a finger print to a finger print on at least one of the fingers.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/800,050 US20150316998A1 (en) | 2011-02-10 | 2015-07-15 | Touchless human machine interface |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/024,392 US10025388B2 (en) | 2011-02-10 | 2011-02-10 | Touchless human machine interface |
US14/800,050 US20150316998A1 (en) | 2011-02-10 | 2015-07-15 | Touchless human machine interface |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/024,392 Division US10025388B2 (en) | 2011-02-10 | 2011-02-10 | Touchless human machine interface |
Publications (1)
Publication Number | Publication Date |
---|---|
US20150316998A1 true US20150316998A1 (en) | 2015-11-05 |
Family
ID=45567127
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/024,392 Active 2031-07-29 US10025388B2 (en) | 2011-02-10 | 2011-02-10 | Touchless human machine interface |
US14/800,050 Abandoned US20150316998A1 (en) | 2011-02-10 | 2015-07-15 | Touchless human machine interface |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/024,392 Active 2031-07-29 US10025388B2 (en) | 2011-02-10 | 2011-02-10 | Touchless human machine interface |
Country Status (2)
Country | Link |
---|---|
US (2) | US10025388B2 (en) |
WO (1) | WO2012108998A1 (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140136024A1 (en) * | 2011-07-15 | 2014-05-15 | Brose Fahrzeugteile GmbH & Co., KG, Hallstadt | Error avoidance in the gesture-controlled opening of a motor vehicle door and trunk |
CN108694366A (en) * | 2017-04-05 | 2018-10-23 | 映智科技股份有限公司 | Fingerprint sensor and method for reducing noise interference |
US10259681B2 (en) | 2013-10-24 | 2019-04-16 | Otis Elevator Company | Elevator dispatch using fingerprint recognition |
Families Citing this family (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8988394B2 (en) * | 2011-10-25 | 2015-03-24 | Semiconductor Components Industries, Llc | Electronic devices with camera-based user interfaces |
WO2013154557A1 (en) * | 2012-04-12 | 2013-10-17 | Hewlett-Packard Development Company, L.P. | Non-contact fingerprinting systems wth afocal optical systems |
US9747306B2 (en) * | 2012-05-25 | 2017-08-29 | Atheer, Inc. | Method and apparatus for identifying input features for later recognition |
JP2014021933A (en) * | 2012-07-23 | 2014-02-03 | Ricoh Co Ltd | Projection device, and projection method |
US9690384B1 (en) * | 2012-09-26 | 2017-06-27 | Amazon Technologies, Inc. | Fingertip location determinations for gesture input |
ES2451849B1 (en) * | 2012-09-28 | 2015-03-10 | Fundacion Para La Promocion De La Innovacion Invest Y Desarrollo Tecnologico En La Ind De Automocion | Procedure and system for gestural interaction with a vehicle |
EP2738645A1 (en) * | 2012-11-30 | 2014-06-04 | Harman Becker Automotive Systems GmbH | Vehicle gesture recognition system and method |
WO2014096896A1 (en) * | 2012-12-20 | 2014-06-26 | Renault Trucks | A method of selecting display data in a display system of a vehicle |
JP6322364B2 (en) * | 2013-01-29 | 2018-05-09 | 矢崎総業株式会社 | Electronic control unit |
US9829984B2 (en) * | 2013-05-23 | 2017-11-28 | Fastvdo Llc | Motion-assisted visual language for human computer interfaces |
DE102013010932B4 (en) * | 2013-06-29 | 2015-02-12 | Audi Ag | Method for operating a user interface, user interface and motor vehicle with a user interface |
AU2013206685A1 (en) * | 2013-07-04 | 2015-01-22 | Canon Kabushiki Kaisha | Projected user interface system for multiple users |
DE102013012466B4 (en) | 2013-07-26 | 2019-11-07 | Audi Ag | Operating system and method for operating a vehicle-side device |
TW201510772A (en) * | 2013-09-09 | 2015-03-16 | Novatek Microelectronics Corp | Gesture determination method and electronic device |
CN104460963A (en) * | 2013-09-22 | 2015-03-25 | 联咏科技股份有限公司 | Gesture judgment method and electronic device |
US10296281B2 (en) * | 2013-11-05 | 2019-05-21 | LiveStage, Inc. | Handheld multi vantage point player |
KR101537936B1 (en) * | 2013-11-08 | 2015-07-21 | 현대자동차주식회사 | Vehicle and control method for the same |
TWI506483B (en) * | 2013-12-13 | 2015-11-01 | Ind Tech Res Inst | Interactive writing device and operating method thereof using adaptive color identification mechanism |
ITCO20130068A1 (en) * | 2013-12-18 | 2015-06-19 | Nu Tech S A S Di De Michele Marco & Co | METHOD TO PROVIDE USER COMMANDS TO AN ELECTRONIC PROCESSOR AND RELATED PROGRAM FOR PROCESSING AND ELECTRONIC CIRCUIT. |
US20150309681A1 (en) * | 2014-04-23 | 2015-10-29 | Google Inc. | Depth-based mode switching for touchless gestural interfaces |
US9330306B2 (en) | 2014-06-11 | 2016-05-03 | Panasonic Intellectual Property Management Co., Ltd. | 3D gesture stabilization for robust input control in mobile environments |
JP6361332B2 (en) * | 2014-07-04 | 2018-07-25 | 富士通株式会社 | Gesture recognition apparatus and gesture recognition program |
KR101524575B1 (en) * | 2014-08-20 | 2015-06-03 | 박준호 | Wearable device |
DE102014224632A1 (en) * | 2014-12-02 | 2016-06-02 | Robert Bosch Gmbh | Method for operating an input device, input device |
TWI552892B (en) * | 2015-04-14 | 2016-10-11 | 鴻海精密工業股份有限公司 | Control system and control method for vehicle |
Citations (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4782328A (en) * | 1986-10-02 | 1988-11-01 | Product Development Services, Incorporated | Ambient-light-responsive touch screen data input method and system |
US6061064A (en) * | 1993-08-31 | 2000-05-09 | Sun Microsystems, Inc. | System and method for providing and using a computer user interface with a view space having discrete portions |
US6535622B1 (en) * | 1999-04-26 | 2003-03-18 | Veridicom, Inc. | Method for imaging fingerprints and concealing latent fingerprints |
US20040032906A1 (en) * | 2002-08-19 | 2004-02-19 | Lillig Thomas M. | Foreground segmentation for digital video |
US6929180B2 (en) * | 2001-08-10 | 2005-08-16 | S-Staff Corporation | Personal identification method and personal identification device |
US7170492B2 (en) * | 2002-05-28 | 2007-01-30 | Reactrix Systems, Inc. | Interactive video display system |
US7243945B2 (en) * | 1992-05-05 | 2007-07-17 | Automotive Technologies International, Inc. | Weight measuring systems and methods for vehicles |
US20090167682A1 (en) * | 2006-02-03 | 2009-07-02 | Atsushi Yamashita | Input device and its method |
US20100231522A1 (en) * | 2005-02-23 | 2010-09-16 | Zienon, Llc | Method and apparatus for data entry input |
US7843429B2 (en) * | 1997-08-22 | 2010-11-30 | Pryor Timothy R | Interactive video based games using objects sensed by TV cameras |
US20110205186A1 (en) * | 2009-12-04 | 2011-08-25 | John David Newton | Imaging Methods and Systems for Position Detection |
US8472665B2 (en) * | 2007-05-04 | 2013-06-25 | Qualcomm Incorporated | Camera-based user input for compact devices |
US8515128B1 (en) * | 2009-10-07 | 2013-08-20 | Qualcomm Incorporated | Hover detection |
US8514251B2 (en) * | 2008-06-23 | 2013-08-20 | Qualcomm Incorporated | Enhanced character input using recognized gestures |
US8576199B1 (en) * | 2000-02-22 | 2013-11-05 | Apple Inc. | Computer control systems |
US8610809B2 (en) * | 2009-11-02 | 2013-12-17 | Sony Corporation | Solid-state imaging device and camera system that controls a unit of plural rows |
US9262016B2 (en) * | 2009-01-05 | 2016-02-16 | Smart Technologies Ulc | Gesture recognition method and interactive input system employing same |
US9311528B2 (en) * | 2007-01-03 | 2016-04-12 | Apple Inc. | Gesture learning |
US9513744B2 (en) * | 1994-08-15 | 2016-12-06 | Apple Inc. | Control systems employing novel physical controls and touch screens |
Family Cites Families (21)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5914748A (en) * | 1996-08-30 | 1999-06-22 | Eastman Kodak Company | Method and apparatus for generating a composite image using the difference of two images |
US6072494A (en) | 1997-10-15 | 2000-06-06 | Electric Planet, Inc. | Method and apparatus for real-time gesture recognition |
US7050606B2 (en) | 1999-08-10 | 2006-05-23 | Cybernet Systems Corporation | Tracking and gesture recognition system particularly suited to vehicular control applications |
US6624833B1 (en) | 2000-04-17 | 2003-09-23 | Lucent Technologies Inc. | Gesture-based input interface system with shadow detection |
US6678413B1 (en) * | 2000-11-24 | 2004-01-13 | Yiqing Liang | System and method for object identification and behavior characterization using video analysis |
US7457425B2 (en) * | 2001-02-09 | 2008-11-25 | Thx Ltd. | Vehicle sound system |
US6603462B2 (en) * | 2001-03-21 | 2003-08-05 | Multidigit, Inc. | System and method for selecting functions based on a finger feature such as a fingerprint |
EP3432181B1 (en) * | 2004-11-12 | 2019-10-02 | Koninklijke Philips N.V. | Distinctive user identification and authentication for multiple user access to display devices |
GB2440683B (en) * | 2005-02-23 | 2010-12-08 | Zienon L L C | Method and apparatus for data entry input |
US20070038059A1 (en) * | 2005-07-07 | 2007-02-15 | Garrett Sheffer | Implant and instrument morphing |
EP2428413B1 (en) * | 2005-07-11 | 2013-03-27 | Volvo Technology Corporation | Methods and arrangement for performing driver identity verification |
US8086971B2 (en) * | 2006-06-28 | 2011-12-27 | Nokia Corporation | Apparatus, methods and computer program products providing finger-based and hand-based gesture commands for portable electronic device applications |
US7654425B2 (en) * | 2006-10-31 | 2010-02-02 | Honda Motor Co., Ltd. | Adjustable dashboard-mounted camera mount |
US8793621B2 (en) * | 2006-11-09 | 2014-07-29 | Navisense | Method and device to control touchless recognition |
US7971156B2 (en) | 2007-01-12 | 2011-06-28 | International Business Machines Corporation | Controlling resource access based on user gesturing in a 3D captured image stream of the user |
US8230367B2 (en) * | 2007-09-14 | 2012-07-24 | Intellectual Ventures Holding 67 Llc | Gesture-based user interactions with status indicators for acceptable inputs in volumetric zones |
EP2304527A4 (en) | 2008-06-18 | 2013-03-27 | Oblong Ind Inc | Gesture-based control system for vehicle interfaces |
US8400398B2 (en) * | 2009-08-27 | 2013-03-19 | Schlumberger Technology Corporation | Visualization controls |
US8677284B2 (en) * | 2009-11-04 | 2014-03-18 | Alpine Electronics, Inc. | Method and apparatus for controlling and displaying contents in a user interface |
US8620024B2 (en) * | 2010-09-17 | 2013-12-31 | Sony Corporation | System and method for dynamic gesture recognition using geometric classification |
US20120169671A1 (en) * | 2011-01-03 | 2012-07-05 | Primax Electronics Ltd. | Multi-touch input apparatus and its interface method using data fusion of a single touch sensor pad and an imaging sensor |
-
2011
- 2011-02-10 US US13/024,392 patent/US10025388B2/en active Active
-
2012
- 2012-01-23 WO PCT/US2012/022149 patent/WO2012108998A1/en active Application Filing
-
2015
- 2015-07-15 US US14/800,050 patent/US20150316998A1/en not_active Abandoned
Patent Citations (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US4782328A (en) * | 1986-10-02 | 1988-11-01 | Product Development Services, Incorporated | Ambient-light-responsive touch screen data input method and system |
US7243945B2 (en) * | 1992-05-05 | 2007-07-17 | Automotive Technologies International, Inc. | Weight measuring systems and methods for vehicles |
US6061064A (en) * | 1993-08-31 | 2000-05-09 | Sun Microsystems, Inc. | System and method for providing and using a computer user interface with a view space having discrete portions |
US9513744B2 (en) * | 1994-08-15 | 2016-12-06 | Apple Inc. | Control systems employing novel physical controls and touch screens |
US7843429B2 (en) * | 1997-08-22 | 2010-11-30 | Pryor Timothy R | Interactive video based games using objects sensed by TV cameras |
US6535622B1 (en) * | 1999-04-26 | 2003-03-18 | Veridicom, Inc. | Method for imaging fingerprints and concealing latent fingerprints |
US8576199B1 (en) * | 2000-02-22 | 2013-11-05 | Apple Inc. | Computer control systems |
US6929180B2 (en) * | 2001-08-10 | 2005-08-16 | S-Staff Corporation | Personal identification method and personal identification device |
US7170492B2 (en) * | 2002-05-28 | 2007-01-30 | Reactrix Systems, Inc. | Interactive video display system |
US20040032906A1 (en) * | 2002-08-19 | 2004-02-19 | Lillig Thomas M. | Foreground segmentation for digital video |
US20100231522A1 (en) * | 2005-02-23 | 2010-09-16 | Zienon, Llc | Method and apparatus for data entry input |
US20090167682A1 (en) * | 2006-02-03 | 2009-07-02 | Atsushi Yamashita | Input device and its method |
US9311528B2 (en) * | 2007-01-03 | 2016-04-12 | Apple Inc. | Gesture learning |
US8472665B2 (en) * | 2007-05-04 | 2013-06-25 | Qualcomm Incorporated | Camera-based user input for compact devices |
US8514251B2 (en) * | 2008-06-23 | 2013-08-20 | Qualcomm Incorporated | Enhanced character input using recognized gestures |
US9262016B2 (en) * | 2009-01-05 | 2016-02-16 | Smart Technologies Ulc | Gesture recognition method and interactive input system employing same |
US8515128B1 (en) * | 2009-10-07 | 2013-08-20 | Qualcomm Incorporated | Hover detection |
US8610809B2 (en) * | 2009-11-02 | 2013-12-17 | Sony Corporation | Solid-state imaging device and camera system that controls a unit of plural rows |
US20110205186A1 (en) * | 2009-12-04 | 2011-08-25 | John David Newton | Imaging Methods and Systems for Position Detection |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140136024A1 (en) * | 2011-07-15 | 2014-05-15 | Brose Fahrzeugteile GmbH & Co., KG, Hallstadt | Error avoidance in the gesture-controlled opening of a motor vehicle door and trunk |
US9283905B2 (en) * | 2011-07-15 | 2016-03-15 | Brose Fahrzeugteile Gmbh & Co. Kg, Hallstadt | Error avoidance in the gesture-controlled opening of a motor vehicle door and trunk |
US10259681B2 (en) | 2013-10-24 | 2019-04-16 | Otis Elevator Company | Elevator dispatch using fingerprint recognition |
CN108694366A (en) * | 2017-04-05 | 2018-10-23 | 映智科技股份有限公司 | Fingerprint sensor and method for reducing noise interference |
Also Published As
Publication number | Publication date |
---|---|
US20120207345A1 (en) | 2012-08-16 |
US10025388B2 (en) | 2018-07-17 |
WO2012108998A1 (en) | 2012-08-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10025388B2 (en) | Touchless human machine interface | |
US11726577B2 (en) | Systems and methods for triggering actions based on touch-free gesture detection | |
US9235269B2 (en) | System and method for manipulating user interface in vehicle using finger valleys | |
JP6703486B2 (en) | Vehicle trunk opening device and method, and recording medium recording program for executing this method | |
US20110260965A1 (en) | Apparatus and method of user interface for manipulating multimedia contents in vehicle | |
JP5261554B2 (en) | Human-machine interface for vehicles based on fingertip pointing and gestures | |
US20140139429A1 (en) | System and method for computer vision based hand gesture identification | |
CN105849673A (en) | Human-to-computer natural three-dimensional hand gesture based navigation method | |
CN113646736A (en) | Gesture recognition method, device and system and vehicle | |
JP6671288B2 (en) | Gesture device, operation method thereof, and vehicle equipped with the same | |
KR101438615B1 (en) | System and method for providing a user interface using 2 dimension camera in a vehicle | |
CN113330395B (en) | Multi-screen interaction method and device, terminal equipment and vehicle | |
US9760758B2 (en) | Determining which hand is being used to operate a device using a fingerprint sensor | |
US20160104313A1 (en) | Method and apparatus for rendering object for multiple 3d displays | |
WO2015001547A1 (en) | Aligning gaze and pointing directions | |
CN109213363A (en) | Predictive indicator touch location determines the system and method being directed toward in 3d space | |
WO2021175749A1 (en) | Method and system for triggering picture-taking of the interior of a vehicle based on a detection of a free-space gesture | |
JP2018055614A (en) | Gesture operation system, and gesture operation method and program | |
KR101892390B1 (en) | User interface, transport means and method for recognizing a hand of a user | |
WO2023124384A1 (en) | Methods and apparatuses for determining seat of occupant inside vehicle and for vehicle control, electronic device, medium, and program product | |
CN113799791B (en) | Steering wheel key false touch prevention method and device based on camera and vehicle | |
KR20210057358A (en) | Gesture recognition method and gesture recognition device performing the same | |
CN104375647B (en) | Exchange method and electronic equipment for electronic equipment | |
KR20150067679A (en) | System and method for gesture recognition of vehicle | |
JP7325625B2 (en) | Gesture detection device and gesture detection method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |