JP2019087279A - Systems and methods of direct pointing detection for interaction with digital device - Google Patents

Abstract

To provide systems and methods of direct pointing detection for interaction with a digital device.SOLUTION: Systems, methods and non-transitory computer-readable media are disclosed. For example, a touch-free gesture recognition system is disclosed that includes at least one processor. The processor may be configured to enable presentation of first display information to a user to prompt a first touch-free gesture at at least a first location on a display. The processor may also be configured to receive first gesture information from at least one image sensor corresponding to a first gesturing location on the display correlated with a first touch-free gesture by the user.SELECTED DRAWING: Figure 1

Description

(Related application)
This application claims priority to US Provisional Application No. 61 / 840,447, filed Jun. 27, 2013, which is incorporated herein by reference in its entirety. Be done.

  The present disclosure relates to the field of touch free gesture detection, and more particularly to devices and computer readable media for using sensors that detect touch free gestures.

  Allowing the user to interact with the device or an application running on the device is useful in many different situations. For example, keyboards, mice, joysticks are often included in electronic systems to allow the user to enter data, manipulate data, and cause the system's processor to perform various other actions. Be However, touch-based input devices such as keyboards, mice, joysticks etc. are increasingly being replaced or supplemented by devices that allow touch-free user interaction. For example, the system may include an image sensor and may capture an image of the user, including, for example, the user's hands and / or fingers. The processor may be configured to receive such an image and initiate an action based on a touch free gesture performed by the user. It is desirable to improve the techniques for detecting and inputting touch free gestures.

  In one disclosed embodiment, a touch free gesture recognition system is disclosed. The touch free gesture recognition system may include at least one processor. At least one processor enables presentation of first display information prompting the first touch free gesture at least at a first position on the display to the user and correlates to the first touch free gesture by the user Receiving first gesture information from at least one image sensor corresponding to a first gestured position on the attached display, the first gestured position being one of the user's eyes being the user The position of the first display information and at least partially different from the position of the first display information as a result of being superior to the other eye of the Determining a first offset to be set and at least a second position on the display Enabling presentation of a second piece of information prompting the user to perform a subsequent touch free gesture; receiving subsequent gesture information from at least one image sensor corresponding to the subsequent touch free gesture by the user And using a first offset to determine a position on the display that is affected by the subsequent touch free gesture.

  Additional aspects related to the embodiments are described in part in the following description, and some may be understood from the description or may be learned by the practice of the disclosed embodiments.

It is to be understood that both the foregoing general description and the following detailed description are for the purpose of illustration and description only and are not intended to limit the scope of the claims.
For example, the present application provides the following items.
(Item 1)
Enabling presentation to a user of first display information prompting a first touch free gesture at at least a first position on the display;
Receiving first gesture information from at least one image sensor corresponding to a first gesturing position on the display, correlated to a first touch free gesture by the user; The gesturing position of 1 being at least partially different from the position of the first display information as a result of the user's one eye predominating over the other eye of the user;
Determining a first offset associated with the position of the first display information and the first gesturing position;
Enabling presentation of second information prompting the user to perform a subsequent touch free gesture at at least a second position on the display;
Receiving subsequent gesture information from the at least one image sensor, corresponding to a subsequent touch free gesture by the user;
A touch-free gesture recognition system comprising at least one processor configured to: use the first offset to determine a position on the display affected by the subsequent touch-free gesture. .
(Item 2)
The first touch free gesture and the subsequent touch free gesture include a pointing, and the at least one processor is further operable to determine the pointing position from the at least one image sensor to determine a pointing position on the display. The system of claim 1, wherein the system is configured to use one gesture information and the subsequent gesture information.
(Item 3)
The system of claim 1, wherein the at least one processor is further configured to transmit an output to a destination reflecting a recognized gesture that is associated with the subsequent touch free gesture.
(Item 4)
Item 1 wherein the subsequent gesturing position is at least partially different from the position of the second information on the display as a result of one eye of the user taking precedence over the other eye of the user The system described in.
(Item 5)
The at least one processor is further configured to: generate a first candidate pointing area generated using at least one first line of sight associated with a first eye of the user; and a second associated with a second eye of the user The system of claim 1, wherein the system is configured to determine the first offset by identifying at least one of a second candidate pointing area generated using two eyes.
(Item 6)
The information associated with the first offset corresponds to a location on the face other than the pupil that acts as a starting point for the gaze calculation, and the at least one processor further determines the intended location of the subsequent touch free gesture The system according to claim 1, configured to use a position on the face excluding the pupil to determine.
(Item 7)
The system of claim 1, wherein the at least one processor is configured to use information associated with the first offset to determine an intended position of the subsequent touch free gesture on the display. .
(Item 8)
The at least one processor further comprises:
A first point in a first area on the display, associated with a first line of sight passing from a position associated with the first eye through the fingertip to the top of the display;
Selecting at least one of a second point in a second area on the display associated with a second line of sight passing from the second position associated with the second eye to the display through the fingertip to the display The system according to claim 1, wherein the system is configured to determine the first gesturing position by:
(Item 9)
The at least one processor is further configured to determine the first gesturing position by determining at least a third point substantially proximate to the first point and the second point; The system according to item 8.
(Item 10)
The system of claim 1, wherein the at least one processor is further configured to determine different offsets associated with ocular dominance for different areas of the display.
(Item 11)
The system of claim 1, wherein the at least one processor is further configured to determine a position of pointing by using multiple candidate pointing area regions at multiple different time intervals.
(Item 12)
The system of claim 1, wherein the at least one processor is further configured to store, for each of a plurality of users, unique information associated with the first offset.
(Item 13)
The at least one processor is further configured to store, as the stored unique information, information received from the at least one image sensor associated with an identification for each of the plurality of users The system according to 12.
(Item 14)
13. The system of claim 12, wherein the at least one processor is further configured to store, as the stored unique information, information regarding a position on the face excluding a pupil for each of the plurality of users.
(Item 15)
The system of claim 1, wherein the information associated with the first offset includes an indication that one eye of the user is superior to the other eye of the user.
(Item 16)
The system of claim 1, wherein the information associated with the first offset includes an indication of the degree to which the one eye of the user is superior to the other eye of the user.
(Item 17)
When executed by at least one processor,
Enabling presentation to a user of first display information prompting a first touch free gesture at at least a first position on the display;
Receiving first gesture information from at least one image sensor corresponding to a first gesture location on the display correlated to a first touch free gesture by the user, the first gesture information The gesturing position of the first display information may differ from the position of the first display information, at least in part as a result of the user's one eye predominating over the other eye of the user
Determining a position of the first display information and a first offset associated with the first gesturing position;
Enabling the presentation of second information prompting the user to make a subsequent touch free gesture at at least a second position on the display;
Receiving subsequent gesture information from at least one image sensor corresponding to a subsequent touch free gesture by the user;
Non-one, including an indication to cause the at least one processor to perform an action that is affected by the subsequent touch free gesture, including using the first offset to determine a position on the display Transient computer readable medium.
(Item 18)
The subsequent gesturing position may be at least partially with the position of the second information on the display as a result of the information indicating that one eye of the user is superior to the other eye of the user. 18. A non-transitory computer readable medium according to item 17 which is different.
(Item 19)
The instruction is further executed by the at least one processor to: a first candidate pointing area generated using at least one first line of sight associated with the first eye of the user; Said additional action comprising determining said first offset by identifying at least one of a second candidate pointing area generated using a second line of sight associated with a second eye 18. A non-transitory computer readable medium according to item 17 which causes at least one processor to do.
(Item 20)
The information associated with the first offset acts as a starting point for gaze calculation in response to a position on the face other than the pupil, the indication being executed by the at least one processor. 18. A non-transient according to item 17, causing the at least one processor to perform additional operations including using a position on the face excluding a pupil to determine an intended position of a touch free gesture. Computer readable medium.

  The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate several exemplary embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

FIG. 1 illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

FIG. 2A illustrates an exemplary processing unit of a touch free user interface according to some of the disclosed embodiments.

FIG. 2B illustrates an exemplary block diagram of a system in accordance with some of the disclosed embodiments.

FIG. 3A illustrates an exemplary offset determination step that may be used by methods, systems and computer readable media according to embodiments of the present invention.

FIG. 3B illustrates exemplary gesture detection and identification steps that may be used by methods, systems and computer readable media according to embodiments of the present invention.

FIG. 4 illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

FIG. 5 illustrates exemplary steps for identifying a pointing location that may be used by methods, systems and computer readable media according to embodiments of the present invention.

FIG. 6A illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

FIG. 6B illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

FIG. 7 illustrates an exemplary calibration step that may be used by methods, systems and computer readable media according to embodiments of the present invention.

FIG. 8 illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

FIG. 9 illustrates exemplary steps for identifying candidate faces or regions that may be used by methods, systems and computer readable media according to embodiments of the present invention.

FIG. 10 illustrates a Venn diagram of an exemplary candidate surface according to some of the disclosed embodiments.

FIG. 11 illustrates exemplary gesture tracking steps that may be used by methods, systems and computer readable media associated with embodiments of the present invention.

FIG. 12 illustrates exemplary steps for determining ocular dominance that may be used by methods, systems and computer readable media according to embodiments of the present invention.

FIG. 13 illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

FIG. 14 illustrates an exemplary pictorial representation of a user's face in accordance with some of the disclosed embodiments.

FIG. 15 illustrates an exemplary pictorial representation of a user's hand, according to some of the disclosed embodiments.

FIG. 16A illustrates an exemplary pictorial representation of pointing visual effects in accordance with some of the disclosed embodiments.

FIG. 16B illustrates an exemplary pictorial representation of pointing visual effects in accordance with some of the disclosed embodiments.

FIG. 17 illustrates an exemplary pictorial representation of a touch free user interface system according to some of the disclosed embodiments.

  Reference will now be made in detail to the exemplary embodiments illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

  FIG. 1 depicts an embodiment of a system 2 for touch free operation of a device 4 having a display 6. The device 4 is, for example, a mobile phone, smart glasses, personal computer (PC), entertainment device, set top box, television, portable game console, tablet computer, electronic reader, portable game console, laptop or ultra book, etc. Computers, home appliances such as kitchenware, communication devices, air conditioning thermostats, docking stations, game machines such as portable video game devices, digital cameras, watches, entertainment devices, speakers, smart home devices, media players or media systems, position based Devices, pico projectors or embedded projectors, medical devices such as medical display devices, vehicles, in-vehicle / air infotainment systems, navigation systems Wearable devices, devices compatible with augmented reality, wearable goggles, robots, interactive digital signage, digital kiosks, vending machines, automated teller machines (ATMs), or any other means of receiving data from or outputting data to a user Other devices or systems.

  Display 6 may include any plane, surface, or other means that can cause the display of an image or other visual information. Further, the display 6 may include any type of projector that projects image or visible information on a flat or surface. For example, the display 6 may be a television receiver, computer monitor, head mounted display, broadcast reference monitor, liquid crystal display (LCD) screen, light emitting diode (LED) based display, LED backlight LCD display, cathode ray tube (CRT) display , Electroluminescent (ELD) display, electronic paper / ink display, plasma display panel, organic light emitting diode (OLED) display, thin film transistor (TFT) display, high performance addressing (HPA) display, surface conduction electron emission element display, quantum dot display , Interferometric modulator display, swept volume display, carbon nanotube display, variable focus mirror One or more of a splay, light emitting volume display, laser display, holographic display, light field display, wall, three dimensional display, electronic ink display, and any other electronic device for outputting visual information May include. The display 6 may include or be part of a touch screen. FIG. 1 depicts a display 6 as part of a device 4. However, in alternative embodiments, the display 6 may be external to the device 4.

  The system 2 may also include an image sensor 8 (or from which information is placed adjacent to the device 4 and configured to acquire an image of a three-dimensional (3-D) view space enclosed by a dashed line 10 Can receive The image sensor 8 is, for example, a camera, an optical sensor, an infrared (IR) sensor, an ultrasonic sensor, a proximity sensor, a CMOS image sensor, a short wave infrared (SWIR) image sensor, or a reflectance sensor, a CCD image sensor, a reflectance sensor, A depth video system that includes a three-dimensional image sensor or two or more two-dimensional (2-D) stereoscopic image sensors, and one of any other devices capable of detecting visual characteristics of the environment It may include any image acquisition device, including more than that. As an example, FIG. 1 depicts image sensor 8 adjacent to device 4, but in alternative embodiments, image sensor 8 may be incorporated into device 4 or be located remotely from device 4.

  A user 16 located in the viewing space indicated by the dashed line 10 may appear in the image obtained by the image sensor 8. The image sensor 8 may output 2-D or 3-D monochrome, color or IR video to the processing unit 12 connected to the image sensor 8 by a communication channel integral with the image sensor 8 or by a wired or wireless communication channel.

  Embodiments of the invention may include at least one processor. As used herein, the term "processor" may include an electrical circuit that performs logical operations on one or more inputs. For example, such a processor may be one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC), or any other circuit suitable for performing instructions or performing logic operations. The at least one processor may, for example, correspond to or form part of a processing device, such as the processing unit 12 depicted in FIG. 2A. The processing unit 12 of FIG. 2A may include, inter alia, a processor 15 and a memory 13 that may be used to store the image obtained by the image sensor 8. Processing unit 12 and / or processor 15 may be configured to execute one or more instructions within processor 15 and / or memory 13.

  The memory 13 includes, for example, one or more of persistent memory, ROM, EEPROM, EAROM, flash memory device, magnetic disk, magneto-optical disk, CD-ROM, DVD-ROM, and Blu-ray media. And may include instructions (ie, software or firmware) and / or other data. Although FIG. 2A depicts memory 13 as part of processing unit 12, in other embodiments, memory 13 may be external to processing unit 12.

  The processing unit 12 may be configured to analyze the image obtained by the image sensor 8 and track one or more predetermined pointing elements that may be utilized by the user 16 to interact with the display 6. The pointing element may include, for example, the fingertip 14 of the user 16 disposed in the viewing space of the image sensor 8. In some embodiments, the pointing element includes, for example, one or more hands of the user 16, a portion of the hand, one or more fingers, a portion of one or more fingers, and One or more finger tips 14 may be included, or a hand-held stylus. Although FIG. 1 shows a fingertip 14 as a pointing element, other pointing elements may be used as well and serve the same purpose. Thus, when the fingertip 14 is referred to herein, it should be considered as an example only, and should be broadly interpreted as also including other pointing elements.

  The processing unit 12 may be configured to present display information, such as an icon 20 on the display 6 that the user 16 may point to the fingertip 14. The processing unit 12 may be configured to show on the display 6 an output 18 (indicator 18) corresponding to the position to which the user 16 points. For example, as shown in FIG. 17, it is assumed that the user 16 intends to indicate display information (icon 20) on the display 6. In this example, processing unit 12 may determine that user 16 is actually pointing to gesturing position 19. However, the processing unit 12 may determine an offset O associated with the difference between the display information and the gesturing position 19. Using, in part, the information regarding the offset O, the processing unit 12 may send an output (e.g., an indicator 18) to the display 6 destination that reflects the user's intent. In the embodiment shown in FIG. 17, the indicator 18 is displayed at the same position as the icon 20, even though the user 16 actually points to the gesturing position 19.

  The gesturing position 19 may be any mathematical representation associated with a position on the display 6 defined at some point by the system 2 as the position the user points to. The gesturing position 19 may include specific coordinates (x, y) or (x, y, z in the case of a 3-D display) on the display. The gesturing position 19 may include an area or position (eg, a candidate surface) on the display 6. In addition, the gesturing position 19 may be defined as a probability function (eg, a three-dimensional Gaussian function, etc.) associated with the position on the display. The gesturing position 19 may be associated with a set of additional numbers describing the quality of the detection, such as a probability index indicating how accurate the estimation of the position of the gesturing position 19 on the display 6 is.

  For example, in the case of smart glasses, such as wearable glasses that include the ability to present digital information to the user 16, the gesturing position is such that the user perceives the digital information presented by the smart glasses display to be viewed It can be defined as the position of a certain virtual plane.

  Display information may include still images, moving images, interactive objects (such as icons), videos, and / or visual representations of information. The display information may be displayed by any display method as described above, and may be directly on the flat display, curved display, transparent display as used in projectors, wearable glasses and / or the eye or pupil of the user or It may include a display that projects indirectly.

  Marking or feedback of the pointed icon may be provided, for example, by one or more of visual marking, audio marking, tactile marking, ultrasound marking, and tactile marking. Displaying a visual indicator may, for example, display an icon on the display 6, change the icon on the display 6, change the color of the icon on the display 6, display an indicator light, highlight , Displaying shadowing or other effects, moving an indicator on the display 6, providing a vibratory indication of direction, and / or providing a tactile indication of air. A visual indicator may appear above (or in front of) any other image or video appearing on the display 6. A visual indicator such as an icon on display 6 selected by user 16 may be co-linear with the user's eyes 22 and fingertips 14 on a common line of sight 24 (or gaze). As used herein, and for reasons that will be described in more detail below, the term "user's eye" is an abbreviation that defines the position or area of the user's face relative to the line of sight. Thus, as used herein, the term "user's eye" refers to the pupil or any other feature of the eye, the position of the face between the user's eyes, or at least one eye of the user And position on the user's face associated with the subject, or some other anatomical feature of the face that can be correlated to the line of sight. This concept is sometimes referred to as a "virtual eye."

  The icons are exemplary graphical elements that can be displayed on the display 6 and selected by the user 16. In addition to the icons, the graphical elements are also displayed, for example, the objects displayed in the displayed image and / or the animation, the text displayed on the display 6 or in the displayed file, and in the interactive game May contain objects. Throughout this description, the terms "icon" and "graphical element" are used broadly to include any displayed information.

  As depicted in step 313 of FIG. 3A, embodiments of the present invention provide the user 16 with a first display of information prompting a first touch free gesture at at least a first position on the display 6. May make presentation possible. Enabling the presentation may occur, for example, by transmitting data from the processor 15 to the display device such that the information or a representation thereof is presented to the user 16 via the display 6. The representation may be designed to evoke or prompt the user 16 to make a touch free gesture towards the display 6, as illustrated by way of example only in FIG. In particular, FIG. 1 depicts a user 16 performing a pointing hand free gesture. The first position on the display 6 generally corresponds to the position of the display information. The correspondence is not accurate as there may be a difference between the user's perception (i.e. the displayed information) and the actual position of the displayed information (i.e. the gesturing position) as described in more detail below. There is a possibility. The first information may include, for example, graphical elements, icons, or other information as detailed below.

  For example, processing unit 12 may also be configured to display one or more selectable graphical elements or icons on display 6. FIG. 1 illustrates four exemplary icons 20a, 20b, 20c, and 20d. However, in some embodiments, any number of icons may be displayed on the display 6 at any time. Any of the icons 20a-20d may be selected by the user 16 by pointing at the particular icon that the user 16 is to select.

  As depicted in step 315 of FIG. 3A, embodiments of the present invention correspond to a first gesturing position on the display 6 that is correlated and reflects a first touch free gesture by the user 16 The first gesture information may be received. The first gesture information may correspond to any movement or pose, as described in more detail below. The gesture information may include, for example, numerical value or graphic data corresponding to the detected hand gesture or pointing element. Numeric or graphic data may be received by the at least one processor 15, which may correlate it to a known display position.

  Processing unit 12 may be configured to identify the selection of the icon by user 16 based on predetermined criteria. For example, processing unit 12 may select the icon when user 16 points the icon for a predetermined amount of time (ie, a predetermined number of seconds). In addition, processing unit 12 may be configured to identify the selection of the icon by user 16 as the user 16 performs a predetermined action towards the selected icon. For example, the predetermined movement may include a tapping movement followed by movement of the fingertip 14 away from the selected icon after movement of the fingertip 14 towards the selected icon. The system 2 may be configured to generate visual and / or audio feedback when the user 16 selects by pointing to an icon. For example, if one icon is placed on the display 6 and a user gesture is detected, the system 2 may be configured to assume that the user 16 gestures only to the displayed icon. If multiple icons are displayed, system 2 may be configured to assume that user 16 gestures towards the icon closest to the gesturing position 19 associated with the user's gesture.

  As will be described in more detail below, the user 16 may have one dominant eye over the other, so the actual gesturing position on the display 6 (pointing position to be detected) is pointed to by the user 16 himself. It may differ from the actual display position of the icon considered to be on. Thus, to enhance the accuracy of the touch free gesture system, the system is depicted generally at step 317 of FIGS. 17 and 3A, and as described in more detail below, of the first display information (eg, icon 20). It may be enabled to determine the position and the offset O associated with the first gesturing position 19. The offset O can be determined and applied in many different ways. For example, the offset O may be achieved by selecting a "virtual eye" at an offset position from the center point between the user's eyes, and the virtual eye may be used as a starting point of gaze. Instead, the offset O is determined based on the detected difference between the icon (e.g. icon 20) the user 16 gestures and the detected position of the gesturing, i.e. the gesturing position 19 obtain. Examples and further details will be provided later in connection with FIGS. 12, 13 and 17.

  The description sometimes refers to the determination of a single offset O, but this is an omission, and the system has a large number of offsets (at consecutive or periodic intervals) to potentially achieve a high level of accuracy. It is possible to determine and relate to different positions on the display 6, relate to different sizes of icons, relate to different clusters of icons, or within the area or position on the display where the position of the gesture is detected It should be understood that it is also possible to determine a number of offsets related to the distance between the icons at. Furthermore, the offset may be in any direction represented by one coordinate, two coordinates, or three coordinates in the case of 3-D. Furthermore, the offset O may be any mathematical expression associated with the difference between the position on the display that the user perceives as pointing and the gesturing position 19 as directed by the system 2. The offset O is the distance between these two positions on the display 6 itself (e.g. 24 pixels to the right, 8 pixels on the top), a vector (2D or 3D vector) indicating the difference between the two positions, these two positions A set of numbers related to the distance between and / or an expression representing the distance between these two positions. The offset O may relate to the position of the "virtual eye" relative to other anatomical parts of the user's face (such as the user's eye 22). The offset O may be related to the position of the user's pointing element (e.g. the fingertip 14) and may be used to correct the detected position of the pointing element. The offset O may also relate to any feature of the calculated eye 24 that departs in front of the "virtual eye" and intersects the display 6 through the pointing element. The second information is then displayed as shown in step 319 of FIG. 3A, and when the user 16 makes a gesture to the second information, as shown in step 321 of FIG. 3A, the second gesture Graphical or numerical data associated with may be received via an image sensor (eg, image sensor 8 in FIG. 1) and transferred to a processor (eg, processor 15 of FIG. 2A). Due to the fact that the user has one eye that is superior to the other, the second or subsequent gesture position may not be exactly aligned with the second information, ie the position that the user intends to point to is there. However, as shown in step 323, processor 15 may be configured to correct for misalignment using the first offset (or offsets) previously determined in step 317. As a result of the correction, the user's intention may be determined and the corresponding action may be initiated. One such action sends output to the display 6 to provide a display representation of the user's intention (eg, as the system detects that the user's pointing position is 3 inches to the left of the icon) Also, the offset correction may include causing the cursor to be displayed on or near the icon). Exemplary operation of the offset is described in further detail in connection with FIGS. 12 and 13.

  In some embodiments, the finger tip 14 does not touch the display 6 when pointing or tapping. In the embodiment illustrated in FIG. 1, the fingertip 14 points towards the icon 20b. The processing unit 12 detects the user 16 pointing the fingertip 14 towards the icon 20b for a predetermined time, by analyzing the image of the image stream (e.g. video stream) obtained by the image sensor 8, or the icon 20b The user 16 who performs a tapping operation (or a clicking operation) toward the user can be detected. When the processing unit 12 determines that a particular icon has been selected, the messages associated with the corresponding command and / or the selected icon may be respectively executed or generated. The generation of a message or the execution of a command results in the corresponding result being any one or more operating systems, one or more local services, one or more applications, one or more devices, One or more remote applications, one or more remote services, or one or more remote devices may be addressed.

  In some embodiments, device 4 may communicate with an external device or website in response to selection of graphical elements that involve communication. Communication may be by one or more applications running on an external device or website, services running on an external device or website, an operating system running on an external device or website, an external device or website A process running on the site, one or more applications running on an external device or website processor, a software program running in the background on an external device or website, or on an external device or website Sending the message to one or more of the services running at. Furthermore, communication is performed on the application running on device 4, the service running on device 4, the operating system running on device 4, the process running on device 4, the processor 15 of device 4 It may include sending the message to one or more of the applications, a software program running in the background of the device 4, or one or more services running on the device 4. Communication with external devices or websites may occur over a communication network.

  In response to the selection of the graphical element, the message may be an application running on the external device or website, a service running on the external device or website, an operating system running on the external device or website, an external Process running on the device or website, one or more applications running on the external device or processor of the website, a software program running in the background of the external device or website, or an external device or website One or more services running on the site may be sent to an external device or website requesting data related to the graphical element identified in the image. The message may also be an application running on device 4, a service running on device 4, an operating system running on device 4, a process running on device 4, a running on processor 15 of device 4. Data associated with graphical elements identified in the image from one or more of the following applications, a software program running in the background of device 4, or one or more services running on device 4 It can be sent to the requesting device 4.

  The message sent to the external device or website may include commands. This command is, for example, a command to execute an application on an external device or website, a command to stop an application running on an external device or website, a command to start a service executed on an external device or website, It may include a command to stop the service running on the external device or website, or a command to send data related to the graphical element identified in the image. The message to device 4 may include a command. This command is, for example, a command to execute an application on device 4, a command to stop an application running on device 4, a command to start a service to be executed on device 4, a service running on device 4 Or a command to send data associated with the graphical element identified in the image.

  In response to the selection of the graphical element, device 4 may receive data associated with the graphical element identified in the image from the external device or website, and device 4 presents the received data to user 16 obtain.

  The command executed and / or the message generated by pointing with two pointing elements, such as two hands, for example select an area, enlarge the selected area by moving the fingertips towards or away from each other Or reducing and rotating the selected area by rotating the fingertip. The command executed and / or the message generated by pointing with two fingers may include forming an interaction between two objects, such as combining a music track with a video track. In addition, the user 16 may execute a command to form a game interaction by pointing the display 6 at one finger, pointing at another finger towards another location on the display 6, and moving it. And / or may generate a message. In addition, the user 16 slides the icon across the screen (this action can be used to unlock the device), scroll the page or folder, and command to increase or decrease the volume May be performed and / or messages may be generated. The command may be a "swipe command" on one or more icons.

  After identifying the location on display 6 that user 16 was pointing to, commands may be performed and / or messages may be generated in response to predetermined gestures performed by user 16. System 2 may be configured to detect gestures, execute associated commands, and / or generate associated messages. The gesture to be detected includes, for example, swipe action, pinch action of two fingers, pointing, gesture from left to right, gesture from right to left, upward gesture, downward gesture, pressing gesture, opening a fist Opening the fist, moving toward the image sensor 8, tapping gesture, waving gesture, circular gesture with finger or hand, clockwise and / or counterclockwise gesture, clap gesture, reverse clap gesture, Close the hand into a fist, pinch gesture, reverse pinch gesture, fan finger open the finger, close finger finger together, point to the graphical element, hold the active object for a predetermined time Clicking on graphical elements, graphica Double clicking on the element, clicking on the right side of the graphical element, clicking on the left side of the graphical element, clicking on the bottom of the graphical element, clicking on the top of the graphical element, grabbing the object, right Gesturing from graphical elements to graphical elements, gesturing from graphical elements to the left, passing through graphical elements from the left, pushing objects, hitting hands, waving hands to graphical elements, blasting gestures, graphical Clockwise or counterclockwise gesture on the element, grasping the graphical element with two fingers, click-drag-release action, sliding an icon, and / or any sensor detectable by a sensor It may include those over one or it of the other operation or pause.

  FIG. 2B depicts an exemplary schematic representation of System 2. In particular, the processing unit 12 may, for example, be an image data repository 202 configured to store the image captured by the image sensor 8 and graphical element generation configured to generate and display an icon on the display 6 (I.e., an icon generator) 204, a command execution module 206 configured to enable the execution of commands associated with the respective icons displayed on the display 6, and a calibration process. A calibration module 208 and a machine vision unit 210 may be provided. The machine vision unit 210 identifies, for example, from the captured image, a gesture recognition module 212 configured to identify a gesture performed by the user 16 and a point or position on the display 6 to which the user 16 points. And an eye identification module 214 configured to:

  FIG. 3B depicts a flowchart illustrating an exemplary method that may be performed by system 2. The method may include capturing 301 at least one image of viewing space in front of the display 6. For example, the image sensor 8 may capture a continuous image of the viewing space in front of the display 6. Images may be captured and stored in a data repository, such as image data repository 202. The captured images may be processed later, or may be processed immediately when they are captured. The captured image may, for example, be located in front of the display 6 and attempt to interact with one or more graphical elements displayed on the display 6 (ie an icon displayed on the display 6 or A user 16 may wish to point to text pointing a pointing element such as a finger tip 14).

  The captured image may also include one or more predetermined portions of the user's 16 face, such as one or more user's eyes 22, and one or more pointing elements, such as a finger tip. The method may include detecting 303 a pointing gesture in at least one image. In particular, the identification of the user 16 and the pointing element can be achieved with the aid of the machine vision unit 210, which comprises, for example, the gesture recognition module 212.

  The method may include identifying 305 a point or position on the display 6 to which one or more pointing elements point towards. This point may be identified, for example, with the help of the eye identification module 214. The processing unit 12 identifies the eye 24 as the user's eye 22, the fingertip 14 and the point on the display 6 at which the selected icon is located are collinear and located on the common eye 24. Can be configured as follows. Step 305 may include one or more processes specified to improve the identification of the point on display 6 to which user 16 points. Although FIG. 1 shows the user's eye 22 as part of the face of the user 16 that may be used to identify the line of sight 24, other parts of the face may be identified as well and used for that purpose. Thus, whenever one or more user eyes 22 are mentioned herein, they should be considered as examples only, and should be interpreted broadly as including other parts of the face as well. is there.

  The method may include identifying 307 the visual object displayed on the display 6 that meets predetermined criteria for the identified point on the display 6. For example, the visual object (or graphical element) may be the visual object closest to the identified point on display 6 selected by user 16.

  The method may include the step 309 of determining when the predetermined condition is met while pointing element pointing towards the visual object. Step 309 may be performed after the selected visual object (graphical element) has been determined. As mentioned above, the condition is, for example, a predetermined period of time in which the pointing element points towards the determined point and / or performs a predetermined gesture (e.g. tapping movement towards the display 6) or any other suitable condition May be included.

  The method may include the step 311 of performing a predetermined action command associated with the visual object. Step 311 may be performed after a predetermined condition is met. For example, the command execution module 206 may be configured to execute commands, such as interacting with other functional elements in the device 4 associated with the selected graphical element.

  In order to determine which graphical element was selected by the interacting user 16, it is desirable to first identify the point on the display 6 to which the user 16 points.

  In some embodiments, the image sensor 8 may include a 3-D image sensor or a pair of two-dimensional stereoscopic image sensors. In these embodiments, processing unit 12 may be configured to locate the x, y, and z coordinates of the user's eye 22 and / or finger 14 in 3-D viewing space. The processing unit 12 may then determine the line of sight 24 having the apex on the eye 22 and passing through the fingertip 14. As illustrated in FIG. 1, dots or markers 18 may be placed at locations on the display 6 where the eyes 24 intersect the display 6. In some embodiments, system 2 may include a combination of 2-D and 3-D image sensors, and from the image captured by the 2-D image sensor, the x and y coordinates of the user's eyes 22 and fingertips 14 , And may be configured to obtain z-coordinates of the user's eye 22 and finger 14 from the image captured by the 3-D image sensor. In some embodiments, the 3-D image sensor may be configured to acquire low resolution images. Composite information from the 2-D and 3-D image sensors may be used to determine the 3-D line of sight 24.

  In another embodiment, the image sensor 8 may include a 2-D position sensor. The user 16 standing in front of the display 6 may point anywhere on the display 6 and the particular line of sight 24 associated with the user's pointing is the eye 22 of the user captured by the image sensor 8 (or other anatomy Position (ie, x and y coordinates) and pointing element (ie, fingertip 14), and the distance between the anatomical position and the display 6 and the pointing element and display It may be determined based on the distance between the user 6 or the user 16 or the eye 22 of the user.

  In some embodiments, processing unit 12 may be configured to determine the point on display 6 to which user 16 points. For example, the x and y coordinates of the user's eye 22 may be determined from the image captured by the image sensor 8. For example, as illustrated in FIG. 6, the distance R1 between the user 16 and the display 6 may be estimated. Based on the estimated distance R1, the z-coordinate of the user's eye 22 in the viewing space may be obtained.

  The distance R1 may be determined based on, for example, the distance between the user's eyes 22 identified in the image obtained by the image sensor 8. Similar distances exist between the eyes (eg, pupils) of different individuals. The degree of similarity is higher for individuals belonging to the same ethnic group. Thus, the distance between the user 16's eyes may indicate the distance R1 between the user's eye 22 and the display 6.

  The x and y coordinates of the pointing element may be determined from the image captured by the image sensor 8. Using the estimated distance R1 between the pointing element and the user 16, the z-coordinate in the visual space of the pointing element may be obtained.

  Once the x, y, and z coordinates of the user's eye 22 (or a virtual eye or other anatomical feature) and the x, y, and z coordinates of the pointing element are obtained, these two points A straight line (eyeline 24) extending through may be determined, and a point at which the line intersects the display 6 may be identified. The identified intersection of this line with the display 6 may be determined as a point on the display 6 to which the user 16 points, for example a gesture position.

  In some embodiments, the display 6 may be a 3-D display and the displayed graphical elements are recognized by the user 16 as being located in the viewing space in front of the display 6 or behind the device 4 It may be a 3-D graphical element. Thus, determining the position of the point to which the user 16 points may be performed on the 3-D display, where the identified line of sight 24 is from the user's eye 22 through the pointing element in front of the display 6 It extends to visual elements representing the respective graphical elements that are perceived by the user 16 as being in space.

  In some embodiments, display 6 is a 2-D display, and the graphical elements displayed may be 2-D graphical elements. Thus, determining the position of the point to which the user 16 points is performed on the 2-D display 6 and the identified line of sight 24 intersects the line of sight 24 with the display 6 through the pointing element from the user's eye 22 Extend up to

  FIG. 4 depicts an exemplary pictorial representation of the user 16 pointing towards the display 6. To better determine the point on the display 6 to which the user 16 points, it may be advantageous to locate the pointing element within the viewing space while pointing. Therefore, it is advantageous to identify a point P 'that is the position of the pointing element during the pointing gesture when the pointing element is closest to the display 6. The movement of the hand of the user 16 during the pointing gesture is illustrated as a motion vector characterized by the point P 'located closest to the display 6.

  System 2 may be configured to identify point P 'during pointing gestures. In some embodiments where the image sensor 8 includes a 3-D image sensor or a stereoscopic image sensor, the processing unit 12 may point when the z-coordinate of the point P 'is located closest to the display 6 It may be configured to extract a point P 'from the captured image so as to indicate the position of the element.

  In some embodiments where the image sensor 8 comprises a 2-D image sensor, the point P 'is identified based on analysis of one or more features of the pointing element at the time of the pointing gesture, for example Motion of the pointing element, motion vector of the pointing element, change of motion vector of the pointing element, pose of the pointing element, size of the pointing element, acceleration and / or deceleration of the pointing element. In some embodiments, the identification of the point P 'may be based on a change in position of the pointing element in the field of view. For example, after the user 16 stretches the arm and reaches the point P ', the user 16 may retract the arm. A change in the position (i.e., coordinates) of the pointing element within the viewing space may indicate that the arm has been retracted and may be used to identify point P '.

  In some embodiments, the point P 'may be identified based on a change in size of the pointing element during a pointing gesture. For example, as the user 16 stretches the arm towards the display 6, the size of the pointing element may increase (e.g., the size of the fingertip 14 in the captured image may increase). The processing unit 12 may be configured to track the size of the pointing element and to determine the position of the pointing element at which the size of the pointing element is at a maximum.

  FIG. 5 depicts a flowchart illustrating an exemplary method for identifying point P '. The operations described with reference to FIG. 5 may be performed by the processing unit 12 including, for example, a machine vision module 210.

  The method may include capturing 501 at least one image of the user 16 interacting with the display 6. In particular, the image sensor 8 may capture an image of the user 16 attempting to interact with the icons displayed on the display 6. The method may include detecting 503 the pointing element in at least one image. The method may include identifying 505 a change in size of the tip of the pointing element. During step 505, captured images may be processed to identify changes in the size of the pointing element in different images. For example, the ratio between the size of the fingertip 14 at the initial position at the start of the pointing gesture and the size of the fingertip 14 at different locations along the movement of the arm towards the display 6 may be calculated, increasing the size of the fingertip 14 Rates may be identified. The maximum size of pointing may occur once the pointing element is placed at the shortest distance from the display 6, and the largest change in size relative to the initial position may be recorded. When the arm is subsequently retracted, the size change decreases as the size of the pointing element decreases.

  The method may include locating 507 the position of the pointing element at which the largest change between the initial size of the pointing element tip and the current size is determined. The method may include the step 508 of specifying the location identified as point P '.

  FIG. 6A is a top view depicting a user 16 pointing towards the display 6. As shown in FIG. 6A, the image sensor 8 may be disposed adjacent to the display 6 and can capture an image of 3-D viewing space indicated by the dashed line 10. The distance R1 of the user 16 from the display 6 is indicated by a line R1 extending from the display 6 to a point on an imaginary line between the eyes of the user 16 (eg, between the pupils). As mentioned above, the distance R1 may be determined, for example, based on the distance between the user's eyes identified in the image obtained by the image sensor.

  Furthermore, based on typical human behavior, when the user 16 points towards the display 6, it can be assumed that his arm is not fully extended from the body, and the arm returns to the body Sometimes it can be assumed that the arms are not necessarily completely retracted. The extended length of the pointing element from the body of the user 16 towards the display 6 is shown in FIG. 6A and 6B as the range R2.

  The range R2 may be estimated based on, for example, the height of the user as well as the estimated ratio of the body, and in addition, the range R2 may be estimated based on information indicating the distance R1 of the user 16 from the display 6. In addition, information regarding the age and gender of user 16 may also be used in estimating range R2. In addition, the range R2 can be estimated from statistical recording information obtained from a plurality of users 16 (which may take into account the height and proportion of each user 16) showing the extension of the arms of the user 16 when pointing. Further, this information may be classified based on the geographic and / or racial origin of the user 16 to distinguish different pointing gestures that may be related to different geographic and / or racial origins. Thus, for example, the average length of the range R2 can be calculated from the information recorded for the arm extension length of many users 16.

  The range R2 may allow estimation of the distance of the pointing element from the user's body, while some deviation may be present during the extension of the arm in a pointing gesture performed by different users 16. FIG. 6A also illustrates a deviation amount Δr that represents the potential deviation of arm extension between different pointing gestures. As described above, assuming that the range R2 is an average value of measured arm extension in different pointing gestures of the user 16, the deviation amount Δr is the above average value (for example, after normalization of the height factor) It may be determined as a value equal to one or more standard deviations of.

  FIG. 6B is a top view depicting different exemplary extensions of the pointing element. As shown in FIG. 6B, an image sensor 8 may be disposed adjacent to the display 6 and may be capable of capturing an image of 3-D viewing space. An image of the user 16 may be captured by the image sensor 8. Unlike the images captured by the 3-D image sensor, the images captured by the 2-D image sensor may provide an image projection of the user 16 on the display 6 without a sense of depth. As shown in FIG. 6B, the user 16 may extend his or her arm and point towards the display 6. Two stretches with different lengths are shown. A line extending from extension a (long extension) intersects the display 6 at point P1, and a line extending from extension b (short extension) intersects the display 6 at point P2.

  Although the extensions a and b may point to different locations on the display 6, both of these extensions can be seen by the image sensor 8 to converge to the same or the same line R3.

  As described above, in order to identify the gesturing position 19 (the position on the display 6 to which the user 16 points), the eye 24 connecting each intersection point of the eye 16 with the eye of the user 16 and the display 6 of the eye 24. Can be determined. However, as a result of the various uncertainties that may be present during the identification of the line of sight 24, there may also be uncertainty as to where exactly the user 16 is pointing to the display 6. These uncertainties may include, for example, the uncertainty regarding the identification of the tip of the pointing object and the uncertainty regarding the identification of a point located between the two eyes of the user 16 who may best represent the eye line 24. Furthermore, as mentioned above, the 2-D image sensor may include additional uncertainty regarding the actual extension of the pointing element from the body of the user 16 relative to the distance of the user 16 from the display 6.

  As a result of these potential uncertainties, when the user 16 points towards the display 6, instead of identifying a point on the display 6 that the user 16 would be pointing to, a larger area (or a candidate A face may be identified on the display 6. The candidate surface may represent the area where the plurality of possible lines extending from the user's eye 22 intersect the display 6 through the pointing element. The image obtained by the image sensor 8 may include the user's eye 22 and the fingertip 14 and may include the projection of the eye 24 onto the projection plane (the projection plane is the plane captured by the image sensor 8). The set of all the lines in the visual space that the projection on the projection plane is the same as the projection on the projection plane of the eye 24 forms a candidate plane. Each line extending from the user's eye 22 through the pointing element and the common candidate plane is a candidate eye. The candidate surface may be, for example, the number of pixels in the candidate surface (the candidate surface may include one or more pixels), its size (eg, represented by the number of pixels or the measured diameter), its shape, screen It may be characterized by different features, including its position above.

  Therefore, regardless of the above-mentioned uncertainty, it would be advantageous to more accurately identify the point or position of the display 6 to which the user 16 points. The operations described with reference to the following figures are applicable to systems equipped with any one of 2-D, 3-D, and stereoscopic image sensors.

  In some embodiments, user 16 may perform a calibration procedure. In the calibration procedure, the user 16 may be instructed to sequentially point to one or more graphical elements (eg, icons) displayed on the display 6. One of the graphical elements may include, for example, a graphical element such that the selection may switch the device 4 from standby mode to active mode, or the selection may unlock the display 6. When the user 16 points towards such a graphical element, the processing unit 12 determines the position of the pointing object and the user's eye 22 in the image obtained by the image sensor 8 while the user 16 is pointing to the graphical element. Can be determined. The fingertip 14 is, for example, disclosed in co-pending US patent application Ser. No. 10 / 593,628, published as US Patent Application Publication No. 2008/0042981, which is incorporated herein by reference in its entirety. It can be identified in the image obtained by the disclosed image sensor 8.

  Methods for identifying faces and eyes in images are known to those skilled in the art. When the user 16 subsequently points to an icon on the display 16, the processing unit 12 determines the position on the display 6 and thus the position of the icon to which the user 16 points using calibration data, with the user's eye 22 and the fingertip 14. And a constraint on the distance between the user 16 and the display 6 which can be determined from the distance between the user's eyes in the image obtained by the image sensor 8.

  FIG. 7 depicts a flow chart illustrating an exemplary method for performing a calibration process in processing unit 12 that includes calibration module 208. During the calibration process, the distance R1 between the user 16 and the display 6 and the range R2 between the pointing element and the user's body may be more accurately estimated. The method comprises the step 701 of displaying the calibration sequence on the display 6. During step 701, one or more graphical elements (eg, icons) may be displayed on the display 6 in a predetermined order or shape, to provide a calibration order. The calibration order may be characterized by the position of the icon on the display 6 as well as the time and / or order of interaction with another icon. The user 16 may be required to interact (eg, point or swipe) with the icons in a predetermined order. In some embodiments, the calibration sequence may also include icons that are swiped or moved according to a predetermined pattern or shape.

  In some cases, user 16 may not even notice the calibration process. For example, as described above, the calibration process may be presented to the user 16 as an interaction to unlock the display 6 (eg, swiping an icon displayed on the display 6 from left to right, etc.). Alternatively, calibration may occur during normal use where the user 16 points to the displayed icon.

  The method may include capturing 703 at least one image of the user 16 interacting with the display 6. During step 703, the user 16 may interact with the display 6 as required by the calibration sequence, and the image sensor 8 may capture an image as the user 16 does so. The method may include the step 705 of determining the position of the user 16 in the viewing plane. During step 705, the position (i.e., x-coordinate and y-coordinate) of the user 16 (e.g., the user's eye 22) in the image captured by the image sensor 8 may be determined. The method may include the step 707 of determining individual points P 'of the pointing element. During step 707, a pointing gesture may be identified and a point P 'indicating the position (i.e., x and y coordinates) of the pointing element at which the pointing element is located closest to the display 6 may be determined.

  The method may include determining 709 a line of sight 24 connecting each icon on display 6 to the 2-D position of point P 'and the 2-D position of user 16. Since the calibration process may be performed according to a predetermined calibration sequence, information indicating which graphical elements displayed on the display 6 the user 16 is currently required to point to is available to the processing unit 12. Thus, the processing unit 12 may more easily correlate between the position of the user 16, the point P 'and the line of sight 24 connecting the respective graphical elements on the display 6 to which the user 16 points. Thereby, each line of sight 24 can extend from the associated graphical element on the display 6 to the respective position of the user 16 through the pointing element located at the respective point P '.

FIG. 8 is a top view depicting an exemplary calibration process. As shown in FIG. 8, the image sensor 8 may be disposed adjacent to the display 6 and may be able to capture an image of 3-D viewing space indicated by the dashed line 10. User 16, may refer towards the icon G _1, which is part of the calibration sequence (not shown). The 2-D position of the user 16 standing in front of the display 6 may be located on the line L1. Although the distance R1 between the user 16 and the display 6 may be estimated as described above, the calibration process may allow for an improvement of this estimation. The 2-D position of point P 'may be located on line L2. Although the range R2 extending between line L1 and line L2 may be estimated as described above, the calibration process may allow for an improvement of this estimation.

As described above with reference to FIG. 6A, uncertainty may exist for the range R2, as indicated above by the deviation Δr. Thus, the distance between line L1 and line L2 can be estimated as either between R2 + Δr / 2 and R2-Δr / 2. During the calibration process, the first straight line L3 extends from the icon _{G 1} on the display 6, the lines L1 and L2 as the distance between equals R2-Δr / 2 line L1 and line L2 connected, the second straight line L4 extends from the icon _{G 1} on the display 6, the distance between the lines L1 and line L2 is connected to line L2 and the line L1 to be equal to R2 + [Delta] r / 2. As shown in FIG. 8, the distance between the point a on the line L2 and the point u1 on the line L1 is equal to R2-Δr / 2, and the distance between the point b on the line L2 and the point u2 on the line L1 is The distance R2 ′ ′ between them is equal to R2 + Δr / 2.

  Point EL on line L1 may represent the estimated position of user 16 which may be determined as any point located between point u1 and point u2. For example, the point EL may be a point located in the middle between the point u1 and the point u2. Each point on line L2 may be determined by extending a straight line from the selected point on line L1 to icon i through line L2. The calibration method may include a step 711 of providing feedback to the user 16 and repeating steps 703 to 709 as the user 16 follows the calibration sequence. Feedback may be provided to the user 16, for example, by changing one or more of the color, size, shape and position of each graphical element, or by audible marking. Steps 703 through 709 may be repeated as user 16 interacts with each graphical element on display 6 according to the calibration sequence.

For any icon G _1, represents the position of the user 16, each point ELi on the line L1, may be determined as described above. The method may include the step 713 of determining a range R1 and a range R2 based on multiple eye lines generated with different visual elements. During step 713, final estimates of the ranges R1 and R2 may be determined. The final estimates of the ranges R1 and R2 may be determined based on all points EL obtained for different icons in calibration order (e.g. points located at the center of all identified points).

  The method may include identifying 715 each eye line 24 based on the ranges R1 and R2 in response to a particular pointing gesture directed to the display 6. Step 715 may occur once the system has been calibrated and / or during normal operation.

  The processing unit 12 may be configured to determine a different range R2 for each icon on the display 6, and may create a map that associates the different icons and their respective position on the display 6 with the corresponding range R2. . Any other point of the display 6 (not occupied by the calibration icon) may be associated with the respective range R2 based on the map. This association may be based, for example, on the position of the icons and the linear combination of the respective ranges R2.

  During normal operation, the processing unit 12 is responsive to the pointing gesture of the user 16 towards the display 6 to identify the general direction of the pointing gesture, and the point on the display 6 closest to the specified general direction. , May be configured to select a particular range R2 associated with the calibration icon.

  In addition to, or instead of, the calibration process described above, other techniques may be possible to more accurately identify the point or position on the display 6 to which the user 16 points. As explained above, due to the different uncertainty regarding the parameters used in determining the point on display 6 to which user 16 points, candidate faces may be identified on display 6 from both eyes of user 16 A plurality of possible lines extending may represent a plane intersecting the display 6 through the pointing element.

  FIG. 9 illustrates an exemplary method that may use processing unit 12 including machine vision unit 210 to reduce the size of the candidate surface and to more accurately identify the point on display 6 that user 16 points to. Portray a flow chart.

  The method may include capturing 901 at least one image of the user 16 interacting with the display 6. During step 901, the image sensor 8 may capture an image of the user 16 attempting to interact with the graphical elements displayed on the display 6. The method may include the step 903 of identifying the user's eye 22 and the pointing element within the at least one image. In particular, the captured image may be processed and identified within the captured image of the user's eye 22 and pointing element. The method may include the step 905 of obtaining a series of position data. During step 905, the movement path of the pointing element may be tracked while it is extended towards the display 6, and a series of positions including different positions of the pointing element (eg, the fingertip 14) in the movement path The value of can be obtained.

  The position values include parameters that may describe the position of the pointing element in a given image. For example, the value of the position may be the x and y coordinates of the tip of the pointing element in the field of view captured by the image sensor 8. The user 16 can stretch the arm towards the display 6 and the 2-D position (shown in x and y coordinates) of the finger 14 relative to the viewing plane can be obtained and stored.

  While the arm is extended towards the display 6 during the pointing gesture, starting at a distance from the display 6 and moving from the convergence point forward, the pointing element may point to the same position of the display 6 ( For example, the pointing element may be aligned with the same line of sight 24). During this movement path, the image sensor 8 aligns the different points of the pointing element on the same eye 24 passing through the user's eye 22, the pointing element (ie the fingertip 14) and the points on the display 6 A series of images can be captured as

  The method may include the step 907 of identifying a pointing element at point P '. During step 907, movement of the pointing element (i.e., the fingertip 14) may be tracked until it is identified that the pointing element is located at point P '. At this point, a predetermined number of position values may be selected from the order of the position values extracted from the movement path of the pointing element towards the display 6. The value of the selected position may be, for example, the value of the position correlated with N images (or frames) before the pointing element reaches the point P ′ which is the point closest to the display 6. Alternatively, the value of the selected position may be a sequential position located on the same line of sight.

  As described above, the value of each position in the sequence of position values (represented, for example, by the x coordinate and y coordinate on the field of view) passes through the pointing element placed at the position indicated by the value of the respective position It may be associated with each line of sight 24 connecting the user's eyes 22 intersecting the display 6. Due to the above-mentioned uncertainty, each line of sight may be associated with each candidate face on the display 6.

  The method may include the step 909 of selecting a predetermined number of position data elements and identifying each candidate surface on the display 6. During step 909, each candidate face may be associated with the value of the selected position identified on the display 6. The method may include the step 911 of identifying the overlap area between all the candidate faces as the selected area on the display 6. During step 911, overlapping areas between candidate planes may be identified and shown as selected view planes, thereby reducing the size of the candidate planes and displaying on display 6 as the point to which user 16 points. Identify a small area of

  FIG. 10 depicts an exemplary Venn diagram showing three candidate planes that partially overlap. Each of the three candidate planes is illustrated as a circle covering an area or position on display 6. Each face is associated with different position values of the last three position values (position values: n, n-1 and n-2) in the order of recording of position values.

  As mentioned above, at the convergence point and beyond, each of the three different viewing planes associated with different position values within the sequence of position values may cover the overlapping area. As illustrated in FIG. 9, the three candidate planes may point to the same area and may share an overlap (indicated by the cross hatching pattern) designated as the selected projection plane.

  FIG. 11 depicts a flowchart illustrating an exemplary method that may be performed at processing unit 12. System 2 may be configured to provide feedback to user 16 pointing towards display 6. This feedback may point the pointing element towards the desired point on the display 6 and assist the user 16 to more easily select the desired graphical element. The following operations associated with FIG. 11 are provided in the context of a 2-D display, although similar operations may be performed with a 3-D display.

  The method may include capturing 1101 at least one image of the user 16 interacting with the display 6. During step 1101, the image sensor 8 may capture an image of the user 16 attempting to interact with the icons displayed on the display 6. The method may include the step 1103 of identifying the user's eye 22 and the pointing element within the at least one image. The method may include displaying 1105 the pointing visual effect on the display 6. Step 1105 may occur in response to the detection of a pointing gesture. The pointing visual effect may be, for example, a stain of a color characterized by any shape or color, transparency or opacity, and may be any of the graphical elements displayed on the display 6, which may be on or behind the graphical elements It can be placed in the position of For example, the pointing visual effect may have one or more round symmetrical shapes of transparent color to which the graphical element transmitting it is recognizable, and may be displayed to the user 16 as a flashlight beam that illuminates a portion of the display 6 It can be done. The initial position on the display 6 of the pointing visual effect may be determined based on the direction in which the pointing element initially points.

  The method may include the step of tracking 1107 the movement path of the pointing element. While tracking the pointing elements, as shown in FIG. 9 described above with reference to step 905, an order of each of the position values may be obtained.

  The method may include the step 1109 of changing one or more characteristics of the pointing visual effect in correlation with the movement of the pointing element. For example, when the pointing element is stretched closer towards the display 6, one or more characteristics of the pointing visual effect may be changed in correlation with the movement of the pointing element. The characteristics may include, for example, one or more of the shape, size, and color of the pointing visual effect. For example, pointing visual effects may decrease in size as the distance between the pointing element and the display 6 decreases.

  As the user 16 stretches his arm towards the display 6, the user 16 can intuitively try to move the pointing visual effect towards the position of the desired graphical element displayed on the display 6. The corrective action of the pointing element, which may be performed by the user 16 attempting to move the pointing visual effect towards the selected graphical element, may be identified by the processing unit 12. Optionally, the position of the pointing visual effect on display 6 may change based on the corrective action of user 16. For example, the position of the pointing visual effect on the display 6 may change relative to the direction of the correction gesture made by the user 16.

  In another embodiment, the circular pointing visual effect may decrease as the pointing element approaches the display 6. Optionally, pointing visual effects can be reduced in the direction of the correction gesture. FIG. 16A illustrates the pointing visual effect as it is reduced in size and moved to the left in response to the correction gesture directed to the icon located at the left (shown as a square), and FIG. 16B illustrates the icon located at the right (shown as a square) The size is reduced in response to the correction gesture towards) and illustrates the pointing visual effect moved to the right.

  The method may include determining 1111 a graphical element based on the known position of the pointing visual effect on the display 6 and the movement of the pointing element. For example, an icon that conforms to predetermined criteria for pointing visual effects may be identified as the selected icon. Further, the icon closest to the stain or in the direction of the correction gesture with respect to the stain may be determined as the icon selected by the user 16.

  As mentioned above, the processing unit 12 may be configured to identify a line of sight 24 that passes from the user's eye 22 past the tip of the pointing element and then through the display 6. Thus, the processing unit 12 may be configured to identify the user's eye 22 in the captured image.

  FIG. 12 takes into account the line of sight 24 from the user's eye 22 through the tip of the pointing element to the desired position on the display 6 pointed to by the user 16, which eye of the user dominates when the user 16 points to the display 6 10 depicts a flowchart illustrating an exemplary method for calculating by When pointing to graphical elements on the display 6, the ocular dominance causes an offset between the actual position of the icon on the display 6 and the position pointed to by the user 16. In other words, when calculating the eye line 24 from the midpoint between the user's eyes through the tip of the pointing element, the resulting eye line may not intersect the position of the icon on the display 6. Thus, the first gesturing position is at least partially different from the position of the first display information as a result of one eye of the user 16 being superior to the other eye of the user 16. In order to correct this offset, the method and system according to the invention may calculate the line of sight 24 using a-priori information associated with the user's dominant eye. The information regarding this superiority can know the exact position of the icon displayed by the system, and thus offset when the user 16 prompted to point to the icon is detected pointing to a position offset from the icon Can be derived by the system during calibration or during normal use.

  A priori information may also be obtained from the user's actions, such as during a database or wink. The user 16 may be directly asked for such information, or the user 16 may be asked to wink. During winking, the dominant eye may turn out to be an open eye. In this way, system 2 may calculate eye line 24 with the user's dominant eye and establish an estimate of the position pointed to by user 16 during the different pointing gestures described above with higher accuracy.

  Further, the eye line 24 may be extracted from a point on the display 6 that points through the finger 14 of the user 16. However, it may not reach the user's pupil, but instead may reach any point between the two pupils associated with the more dominant "dominant eye". As described above and depicted in FIG. 14 below, due to the offset associated with the dominant eye, the eye 24 or line of sight is from the offset position (virtual eye Ve) to the icon through the pointing element to the position between the pupils of the user's face. Extend up to Thus, rather than fixing the starting point from one of the user's pupils, it may be advantageous to establish a more accurate starting point between the user's eyes for which the eyes 24 are calculated.

  However, in practice, the line of sight 24 lies on a virtual line extending between the two eyes of the user 16 in that it may overlap with either the left eye or the right eye or any other point therebetween. It can intersect with any point. A user 16 corresponding to the eye 24 associated with the intended display position to which the user intends to point in order to more accurately determine the gesturing position 19, ie the point on the display 6 at which a given eye 24 extends. Determining a more accurate position, which represents the 'virtual eye' of the, is advantageous rather than arbitrarily choosing one of the right or left eyes of any user or any point on the line connecting them. obtain.

  FIG. 12 depicts in more detail an exemplary method for determining at the processing unit 12 including the calibration module 208 a reference point between the user's eyes. The reference point may represent a more accurate intersection of a given eye line 24 and a line of sight extending between the user's 16 eyes (e.g., the pupil). The process described with reference to FIG. 12 may, for example, be performed as a one-time calibration process or may be updated periodically.

  The method may include the step 1201 of displaying an icon on the display 6. The method may include capturing 1203 an image of the user 16 interacting with the display 6. The image captured at step 1203 may be captured by image sensor 8.

  Optionally, the user 16 may be instructed or required to point towards an icon displayed on the display 6. The indication may be provided to the user 16 on the display 6 as an audible indication, as part of a calibration process, or just as a display of information requiring interaction. The method may include identifying 1205 a first candidate surface generated by a line of sight tied to the first eye of the user 16. During step 1205, a pointing gesture of the user 16 towards the display 6 may be identified, and a first line of sight leading to one eye of the user through the pointing element and a first gesturing position on the display 6 may be determined Each candidate surface may be identified on the display 6.

  The method may include identifying 1207 a second candidate surface generated by a line of sight leading to the second eye of the user 16. During step 1207, a pointing gesture of the user 16 towards the display 6 may be identified, and a second line of sight connecting the other eye of the user to the second gesturing position on the display 6 through the pointing object is determined Each candidate surface may be identified on the display 6. As mentioned above, the first and second candidate planes may be determined based on the method described above, but in particular the first and second candidate planes are the distance R1 between the user 16 and the display 6 and It may be determined in part based on an estimate of the extension range R2 between the pointing element and the position of the pointing element in the field of view.

  FIG. 13 is a schematic diagram showing the calibration icon Cl that may be displayed on the display 6. FIG. 13 shows two that one may be generated by the line of sight (indicated by the letter L) connecting the left eye of the user 16 and the other by the line of sight (indicated by the letter R) connecting the right eye of the user 16 The candidate plane is further illustrated. It should be noted that in the illustrated embodiment shown in FIG. 13, unlike the candidate plane L, a portion of the candidate plane R overlaps the calibration icon Cl.

  Based on the position of each of the two candidate planes with respect to the position of the icon, the processing unit 12 may be configured to determine which of the eyes is more dominant, and the offset O associated with the gesturing position 19 It can be decided. System 2 may use that eye or “virtual eye” in determining additional gesturing positions. The processing unit 12 may be configured to indicate which eye is more dominant through the information associated with the offset O. Furthermore, the processing unit may, for example, “relative to the candidate faces L and R and the calibration icon Cl and / or the relative position of the“ virtual eye ”with respect to the user's eye, ie the total distance between the user's left and right eyes It may be configured to indicate the degree of dominance between the left and right eyes, which may be based on the percentage of overlap of the ratio of the distance between the virtual eye "Ve" and one of the user's left and right eyes. The method disclosed with respect to FIG. 12 may also include extending 1209 a line connecting the first and second candidate planes. According to one embodiment illustrated with reference to FIG. 12, the line 130 (as shown in FIG. 13) may be extended between the center of the candidate plane L and the center of the candidate plane R it can. The method may include the step 1211 of determining a point closest to the icon Cl (close to the line 130, i.e., substantially close to the line 130). For example, a third point on line 130 that is closest to the center of calibration icon Cl (which may be the target that user 16 points to) is identified. The point closest to the calibration icon Cl may divide the line 130 into segments A and B. In the example shown in FIG. 13, segment A “starts” the left eye of the user 16 from the point closest to the icon Cl and has a line of sight intersecting the display 6 towards the pointing element (eg the fingertip 14) It may extend to a gesturing position 19 determined by calculation. The gesture position may also be defined as the center of plane L. Also, segment B is determined by “starting” the right eye of the user 16 from the point closest to the calibration icon Cl and calculating the line of sight that intersects the display 6 towards the pointing element (eg, the fingertip 14) Can extend to the center of the gesturing position. The gesture position may also be defined as the center of plane R. The method may include identifying 1213 a corresponding point Ve (or “virtual eye”) on a line extending between the eyes or pupils of the user 16 (as shown in FIG. 14).

  Point Ve may divide the line extending between the eyes of user 16 into segment C and segment D. For example, segment C may extend from point Ve to the user's left pupil, and segment D may extend from point Ve to the user's right pupil. The position of point Ve on a line extending between the eyes or pupils of user 16 is determined by interpolating the position of point Ve such that the ratio A / B is equal to the ratio C / D. obtain.

  The processing unit 12 may be configured to use the corresponding point Ve as an intersection with the face of the user 16 when identifying the eye line. The process fine-tunes the location of the "virtual eye" and each time calibration is performed to reach more collected values to fine-tune it according to the different locations of the user 16 relative to the display 6. Note that it can be done.

  Optionally, the operations described with reference to FIG. 12 may be repeated for a plurality of icons, each located at a different position on the display 6. Thus, each different point Ve between the eyes can be identified by processing pointing gestures directed to different areas or locations on the display 6.

  The “virtual eye” location and / or “virtual eye” mapping information is associated with all the specific users 16 for later use by system 2 or other devices using system 2 (device 4) or external to device 4). When system 2 identifies user 16 (e.g., by face recognition, user voice signature, or explicitness by user 16), system 2 determines "virtual eye" location and / or "virtual eye" mapping information Can be retrieved to indicate stored information. The “virtual eyes” and / or “virtual eye mapping information” stored for each user 6 may include unique identification information obtained by the image sensor 8 and information on the offset associated with each user 6 May be included. For example, the information stored for each user 6 may include face information including the location of facial features, such as the user's eyes, including the user's pupil. In addition, the offset information stored for each user may include the offset information of user 6 with respect to the user's eyes, including the user's pupil.

Embodiments of the present invention store one or more offsets associated with a vector representing the difference on the display 6 between the gesturing position 19 and the position on the display 8 that the user 16 recognizes as pointing to May include steps. For example,
The distance from the center of the candidate surface R and the center of the icon that the user 16 is prompted to point to. And, when the user 16 makes a subsequent touch free gesture, the system 2 may use the stored offset to correct the gesturing position 19 to accurately determine the display information (icon) pointed to by the user . The system 2 may store multiple offsets associated with different areas of the display, and may use a suitable one of the stored offsets in response to the gesturing position 19. Embodiments of the present invention may allow for the identification of the pointing element of the user 16 such as the fingertip 14. Hand features serve to identify the fingertip 14 or any other pointing element that is held or worn by the hand. FIG. 15 depicts the user's hand including the fingertip 14, the nail 26 on the fingertip 14, and the forefinger joint 28.

  The method for determining the pointing element may include the step of determining whether the pointing element is associated with or part of the user's right hand or left hand. From the image captured by the image sensor 8 it can be easily determined which hand to move when the user 16 makes a gesture. The method may include the step of determining the nail 26 on the index finger tip 14. Instead of or in addition to the step of determining the fingernails 26, the method is on the left side of the hand if the user 16 points with the right hand and the right side of the hand if the user 16 points with the left hand The joint 28 of the forefinger can be detected. Alternatively, the system may be configured to detect a series of joint signatures, such as the joints of the third, fourth and fifth fingers. The system may also detect the folded thumb illustrated in FIG. 15, which is a feature of the pointing hand. This may be useful as illustrated in FIG. 15 because the tip of the forefinger is less distinguishable in its recognizable features from other more distinct parts of the hand. However, once a more distinct part of the hand is found, the position of the fingertip can be estimated.

  More specifically, the estimation may occur through measurement or estimation of the width, height, and / or position of the hand relative to the user's face. This data can help to estimate the specific anatomy and dimensions of the user's hand, and thus predict the location of the fingertip 14 if it is not readily apparent from the image from which it was captured I can help. In addition, known relationships between finger spacings can help to improve the estimation. Gems such as one or more rings may be used as reference points to determine the position of the fingertip 14. Reference data or average anatomical data stored in memory 13 for a particular user 16 may help to improve the estimation. Any of the above reference data may be used in the algorithm associated with identifying the pointing position. The reference point may include, for example, the center position of one or more joints, such as the hand, a gem, any nail (including the nail 26 on the fingertip 14), and the joint 28 of the forefinger. Once the reference point is identified, the algorithm may calculate an offset distance in x, y coordinates or x, y, z coordinates as the estimated position of the finger 14 relative to the reference point. The method may identify and track one or more reference points, and may apply corresponding offsets associated with the reference points to estimate the position of the fingertip 14.

  Another embodiment of the invention includes a device for calibrating a touch free gesture recognition system, the device enabling the presentation of a prompt for the user to perform a touch free gesture at at least one prompt position on the display And receiving information from the camera indicating the corresponding gesturing position on the display, wherein the gesturing results from misalignment between the gesturing position on the display and the user's intended gesturing position. The position may include at least one processor configured to perform different than the position of the at least one prompt and to calibrate the gesture recognition system to account for misalignment.

  In addition, in a device for calibrating a touch-free gesture recognition system, the at least one processor further uses, in addition to the calibration, a touch-free gesture for at least one operation; Allowing the prompt to occur during use of the application without specifying for the user; the location of gestures detected on the display during use and on the display that the user is trying to select Periodically improve the accuracy of the gesture position while using the application by periodically recalibrating the gesture recognition system, taking into account the difference between the position of the icon of the and the calibration based on the prediction; To the specific icon of the gesture position on the display Predicting the user's intention to activate a specific icon based on touch; enabling periodic recalibration during normal use of the application, without the user needing a special calibration procedure Improve the accuracy of the gesture position by delaying the calibration until after the user has performed the next action following the touch-free gesture at the position of at least one prompt; recalibration during the appliance unlocking process Determining whether the user's intention is to activate a particular icon based on the layout of the icon on the display, when there is a difference between the position of the icon and the location of the gesture. And after detecting the user's predetermined movement relative to the display, lock the associated appliance Prompting the user to divide; storing information associated with the plurality of pointing positions and the relative position of the icon associated with each of the plurality of pointing positions, and calibrating the gesture recognition system using the stored information Storing the information related to the relative position of the gesture position and the icon associated with the gesture position, and / or calibrating the gesture recognition system based on the stored information; specific to the user's vision Can be configured to perform and calibrate during the unlocking process taking into account certain characteristics.

  In addition, in a device for calibrating a touch free gesture recognition system, the prompt may include at least one icon for signaling the user to unlock the associated appliance or at least one icon, the touch free gesture Is a pointing gesture and at least one processor is configured to signal the user to point at least one icon. The at least one position on the display may include a first known position and a second known position, and the calibration allows the user to gesture from the first known position to the second known position. Based on information related to prompting. The predetermined movement of the user may include at least a portion of the body of the user moving out of the field of view of the camera.

  The gesturing position may be a pointing position, and the at least one processor further calibrates the gesture recognition system initially based on the pointing position and the position of the at least first icon; after the first calibration, the user Enabling control of the application using a gesture; collecting information on the gesture for the at least second icon on the display when controlling the application; and / or the user fingering at least the second icon When doing so, it may be configured to perform recalibration during application control.

  Misalignment may be the result of the user's one eye predominating over the user's other eye, and the at least one processor further indicates that one eye of the user predominates over the other eye of the user Calibrating the gesture recognition system taking into account that: determining an offset between the gesture position and the at least one prompt position; and / or applying to the subsequent determination of the gesture position based on the offset It may be configured to make an adjustment decision.

  Another embodiment of the invention makes it possible to present a prompt to the user to make a touch free pointing gesture at the position of at least one prompt on the display; from an image sensor corresponding to a touch free pointing gesture The position of the fingertip is at least partially dependent on the user's anatomical features in addition to the user's finger tip, by receiving the image information of the; and determining the position of the fingertip associated with the touch free pointing gesture. At least one configured to: determine based on the selected position; and determining a pointing position on the display associated with the touch free gesture based at least in part on the determined position of the fingertip. Touch Gesture Recognition System with Two Processors It may include a beam.

  The touch free gesture recognition system may also include at least one memory device storing a plurality of algorithms for determining a pointing position based on differences in hand orientation, and the at least one processor is further operable to: The preferred algorithm may be selected from the plurality of algorithms based at least in part on the orientation of and the pointing position may be determined based at least in part on the preferred algorithm.

  The touch free gesture recognition system may also include at least one memory device for storing a plurality of algorithms for determining the pointing position based on whether a right hand or a left hand is detected in the image information, the at least one memory device. The one processor further selects a suitable algorithm from the plurality of algorithms based at least in part on whether the right hand or the left hand is detected, and determines the pointing position based at least in part on the suitable algorithm. And / or configured to determine whether the user's hand is the left hand or the right hand based at least in part on the offset between the anatomical features of the user and at least a portion of the user's hand Be done.

  In a touch-free gesture recognition system, the anatomical feature may include a fingernail, may be at least one of the user's hand width, length, size, or position, and / or used in a pointing gesture It may be a second fingertip that is not a fingertip, and the at least one processor may be further configured to determine the location of the fingertip based at least in part on the fingertip used for pointing and the second fingertip.

  In a touch-free gesture recognition system, the at least one processor is further configured to at least partially position the fingertip in a predetermined area other than the fingertip of the user's hand in the absence of the location of the fingertip in the image information. The location of the fingertip may be configured to be determined based at least in part on the determined length of the finger used to determine and / or point.

  In a touch-free gesture recognition system, the at least one processor further determines whether the hand in the image information is a left hand or a right hand, and if determined as a right hand, is determined as the upper left region of the hand or the left hand If the position of the fingertip is determined as a position relative to the upper right area of the hand; positioning the hand by applying a filter mask related to the size of the hand to the image information; 2 of the user's hand within the image information Detecting the fingers of the book and determining the position of the fingertip as a position between the two fingers; and / or detecting the fingers of the user's hand in the image information, at least partially on the fingers Determining the fingertip position based on, wherein each of the plurality of fingers may be configured to be associated with different weights for determining the fingertip position.

  In a touch-free gesture recognition system, the at least one processor may be further configured to continue tracking the fingertip with non-fingertip information after determining the position of the fingertip without first positioning the fingertip in the image information .

  In a touch-free gesture recognition system, at least one processor further positions the fingertip position at the first time, but fails to position the fingertip position at the second time, using the first fingertip position and non-fingertip information , Can be configured to track the fingertips.

  Other embodiments of the invention may include a device for providing visual feedback in a touch free gesture recognition system, the device enabling presentation of an image on a display; performed by a user Receiving information from the camera indicating a touch free gesture; and enabling presentation of a gesturing indicator on the display, the gesture ring indicator representing the position of the touch free gesture of the user relative to the display, Presenting the gesturing indicator in the first screen on the display when the touch free gesture of the user is at the first distance from the display; display the touch free gesture of the user being different from the first distance Second distance from When in, including at least one processor configured to perform the method comprising: presenting a gesturing indicator to a second screen on the display.

  In a touch free gesture recognition system, the gesture ring indicator may be a pointing indicator. The information received from the camera may reflect at least a portion of the user's hand. The image on the display includes at least one icon.

  In a touch-free gesture recognition system, at least one processor further reduces the size of the gesturing indicator as the distance from the display of the touch-free gesture by the user decreases; the distance from the display of the touch-free gesture by the user increases Increasing the size of the gesturing indicator as it detects; detecting the corrective action associated with the user's touch-free gesture; determining the user's intention to select the icon based on the corrective action; part of the gesture detecting Presenting the gesture indicator only after being detected and / or when the user's hand is detected in a predetermined area of the field of view of the image sensor; the detection position of the gesture movement relative to the display is predetermined Presenting the gesture indicator only after entering into the value; increasing the size of the gesturing indicator as the distance of the touch-free gesture of the user from the display decreases; and / or display of the touch-free gesture of the user As the distance from Z increases, the size of the gesturing indicator may be configured to decrease.

  In a touch free gesture recognition system, the display may be a virtual 3D display, and the at least one processor further changes the distance between the gesture indicator and the display as the distance between the user's hand and the display changes. Making the virtual 3D indicator appear as if coming out of at least a part of the user's hand; as the distance from the display of the user's touch-free gesture decreases, Reduce the size of the 3D indicator; and / or display the gesturing indicator in two forms, a first form prior to information selection on the display and a second form after information selection on the display Can be configured as follows.

  It will be appreciated that the system 2 according to the subject matter of the present disclosure may comprise a suitably programmed computer. Similarly, the subject matter of the present disclosure contemplates a computer program readable by a computer for performing the methods of the disclosed subject matter. The subject matter of the present disclosure further contemplates a machine readable memory embodying a program of instructions executable by the machine for performing the methods of the subject matter of the present disclosure.

  It is to be understood that the subject matter of the present disclosure is not limited in its application to the detailed description contained in the specification or specified in the description illustrated in the drawings. The subject matter of the present disclosure can be implemented in other embodiments and in various ways. Thus, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting. Thus, one of ordinary skill in the art can readily employ the concepts on which the present disclosure is based as a basis for designing other structures, methods, and systems for carrying out the several objectives presented by the subject matter of the present disclosure. Will understand.

  For clarity, certain features described herein in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, various features that are, for brevity, described in the context of a single embodiment, can also be provided separately as a number of embodiments or any suitable subcombination. Further, the features may be described above as acting in a particular combination, and may be initially claimed as such, but one or more features from the combination described in the claims may in some cases be May be eliminated from the combination, and the combination of claims may be directed to variants of subcombinations or subcombinations.

Specific embodiments have been described. Other embodiments are within the scope of the following claims.

Claims (1)

The invention described in the specification.