CN106125921B - Gaze detection in 3D map environment - Google Patents

Abstract

This disclosure relates to the gaze detection in 3D map environment.A kind of method includes: three-dimensional (3D) figure for receiving at least one physical feeling of user (22) of computer system；And two dimension (2D) image of user is received, which includes the eyes (34) of user.The 3D coordinate of user's head (32) is extracted from 3D figure and 2D image, and the image recognition of 3D coordinate and eyes based on head goes out the direction performed by the user stared.

Description

Gaze detection in 3D map environment

The application be international filing date be on 2 9th, 2012, application No. is 201280007484.1, entitled " 3D The divisional application of the international application into National Phase in China of gaze detection in map environment ".

Related application

This application claims the U.S. Provisional Patent Application submitted for 9th for 2 months on August 24th, 61/440877,2,011 2011 to mention The U.S. Provisional Patent Application 61/ that the U.S. Provisional Patent Application 61/526692 of friendship and September in 2011 are submitted on the 25th 538867 priority is incorporated them into herein by reference herein.

Technical field

The present invention relates generally to man-machine interfaces, and more particularly to the interface for combining multiple user's interactive forms.

Background technique

It can use many different types of user interface apparatus and method at present.Common tactile interface equipment includes meter Calculate switch disk, mouse and control stick.The touch carried out in touch screen detection display area by finger or other objects is deposited And position.Infrared remote control is widely used, and " wearable " hardware device also have been developed, for remote The purpose of process control.

The computer interface of three-dimensional (3D) induction of physical feeling based on user has also been suggested.For example, PCT is international Open WO 03/071410, the disclosure is incorporated herein by reference, describes a kind of using depth-detecting sensor hand Gesture identifying system.The 3D sensor close to user being generally arranged in room provides location information, and the location information is for knowing Gesture caused by other target body site.Shape and its position based on physical feeling and the direction at one section of interval are come Identify the gesture.The gesture is classified to be used to determine the input for arriving relevant electronic equipment.

The document being incorporated by reference into present patent application will be considered as the intact part of the application, unless The definition explicitly or implicitly made in any term and this specification defined in these incorporated documents mutually conflicts, (at this time) it should only consider the definition in this specification.

As another example, United States Patent (USP) 7,348,963, the disclosure is incorporated herein by reference, which retouches A kind of interactive video display system is stated, display screen shows visual image within the system, and camera capture is about position The 3D information of the object in interactive area before the display screen.Computer system indicates to show in response to the change of object Screen changes visual image.

Three-dimensional human-computer interface system can not only identify the hand of user, but also can identify other positions of body, packet Include head, trunk and limbs.For example, U.S. Patent application 2010/0034457, the disclosure is incorporated herein by reference, It describes a kind of for simulating humanoid method according to depth map.The profile to find body is split to the depth map. The profile is processed for identify the trunk of the object and one or more limbs.By in analysis depth figure at least The arrangement of one limbs identified inputs to generate to control the application program run on computers.

The gaze-direction of some user interface system tracking users.For example, United States Patent (USP) 7,762,665, the disclosure of which It is incorporated herein by reference, which depict a kind of methods for adjustment equipment operation, this method comprises: providing for being closed In the attention user interface of the information of the attention state of user；And the operation based on information adjustment equipment obtained, wherein The operation being conditioned is started by the equipment.Preferably, the information about the attention state of user is that user uses with by the attention The Eye contact that the equipment that family interface senses carries out.

Summary of the invention

According to an embodiment of the invention, providing a method, this method comprises: receiving the user of computer system extremely Three-dimensional (3D) figure of a few physical feeling；The image of the two dimension (2D) of the user is received, which includes the eyes of the user, The 3D coordinate on the head of the user is extracted from 3D figure and 2D image；The image recognition of 3D coordinate and eyes based on the head The direction performed by the user stared.

According to an embodiment of the invention, a kind of method is additionally provided, this method comprises: receiving the use including computer system The image of the eyes at family；The direction stared executed based on the image recognition of the eyes by the user；It is identified based on gaze-direction It is coupled to the region on the display of computer system, and operation is executed to the content presented in this region.

According to an embodiment of the invention, in addition provide a method, this method comprises: by computer system in coupling again It closes and multiple interactive projects is presented on the display of computer；Receiving from the sensing equipment for be coupled to the computer indicates user Gaze-direction input；Target point based on gaze-direction identification on the display；Show that the target point with this is appeared in Show that the first interaction project on device is associated；And in response to the target point, the one or more second opened on the display is handed over Mutually exclusive project.

According to an embodiment of the invention, a kind of method is additionally provided, this method comprises: receiving and separation calculation machine system The First ray of three-dimensional (3D) figure of at least one physical feeling of user at any time, to extract at first point and of the user 2 points of 3D coordinate, the 3D figure indicate movement of the second point relative to the display for being coupled to the computer system；Calculate with First point and second point intersection line segment；It identifies the target point of the line segment Yu the display intersection, and is bonded on display The interaction project close to the target point of upper presentation.

According to an embodiment of the invention, additionally providing a kind of device, which includes sensing equipment, which is matched It is set to the eye image for receiving three-dimensional (3D) figure and the user of at least one physical feeling of user, and receives the two of the user (2D) image is tieed up, which includes the eyes of user；And computer, the computer are coupled to the sensing equipment and are configured as The 3D coordinate that the head of the user is extracted from 3D figure and 2D image, 3D coordinate and eye image based on the head are identified by this The direction stared that user executes.

According to an embodiment of the invention, still further providing a kind of device, which includes sensing equipment, the sensing equipment It is configured as receiving the image including eyes of user；And computer, the computer be configured as based on the eye image identification by The direction stared that the user executes identifies the region being coupled on the display of the computer system based on gaze-direction, with And operation is executed to the content presented in this region.

According to an embodiment of the invention, additionally providing a kind of device, which includes display and computer, the computer It is coupled to the display and is configured as that multiple interactive projects are presented on the display；It is set from the sensing for being coupled to the computer It is standby to receive the input for indicating the gaze-direction of user；Target point based on gaze-direction identification on the display；Make the mesh Punctuate is associated with the first interaction project of appearance on the display；And in response to the target point, open on the display One or more the second interaction project.

According to an embodiment of the invention, additionally providing a kind of device, which includes display and computer, the computer Be coupled to the display and be configured as receiving and at least one physical feeling of the user of separation calculation machine system at any time The First ray of three-dimensional (3D) figure, to extract and the 3D coordinate of second point at first point of the user, which indicates second Movement of the point relative to the display for being coupled to the computer system；Calculate the line segment intersected with and second point at first point；Identification The target point of the line segment and the display intersection；And the interaction item close to the target point that engagement is presented over the display Mesh.

According to an embodiment of the invention, a kind of computer software product is still further provided, the computer software product packet The computer-readable medium for being wherein stored with the nonvolatile of program instruction is included, when the instruction is read by computer, makes the calculating Machine receives three-dimensional (3D) figure of at least one physical feeling of the user of the computer；Two dimension (2D) image of the user is received, The image includes the eyes of the user；The 3D coordinate of the user's head is extracted from 3D figure and 2D image；And it is based on the head 3D coordinate and the eye image identify the direction stared that is executed by the user.

According to an embodiment of the invention, additionally providing a kind of computer software product, which includes it In be stored with program instruction the computer-readable medium of nonvolatile connect the computer when the instruction is read by computer Packet receiving includes the image of the eyes of the user of the computer system；The side stared executed by the user is identified based on the eye image To；The region being coupled on the display of the computer system is identified based on the gaze-direction, and in this region to presentation Content execute operation.

According to an embodiment of the invention, additionally providing a kind of computer software product, which includes it In be stored with program instruction nonvolatile computer-readable medium, when the instruction is read by computer, so that the computer Multiple interactive projects are presented on the display for be coupled to the computer；Receiving from the sensing equipment for being coupled to the computer indicates The input of the gaze-direction of user；The target point on the display is identified based on the gaze-direction；Make the target point and appears in The first interaction project on the display is associated；And in response to the target point, one or more the on the display are opened Two interaction projects.

According to an embodiment of the invention, a kind of computer software product is still further provided, the computer software product packet The computer-readable medium for being wherein stored with the nonvolatile of program instruction is included, when the instruction is read by computer, makes the calculating Machine receives and divides the First ray of three-dimensional (3D) figure of at least one physical feeling of the user of the computer at any time, so as to Extract and the 3D coordinate of second point at first point of the user, which indicates second point relative to being coupled to the aobvious of the computer Show the movement of device；Calculate the line segment intersected with and second point at first point；Identify the target point of the line segment Yu the display intersection； And the interaction project close to the target point that engagement is presented over the display.

Detailed description of the invention

Only the disclosure is described by way of example referring to attached drawing herein.

Fig. 1 is the schematic figure of the computer system of the user interface of the realization mixed form of embodiment according to the present invention Show explanation；

Fig. 2 is the computer for schematically showing the user interface for realizing the mixed form of embodiment according to the present invention The block diagram of the functional unit of system；

Fig. 3 is the process for schematically showing the method for the gaze-direction of detection user of embodiment according to the present invention Figure；

Fig. 4 is the schematic diagram of the numeric keypad for being configurable for input password of embodiment according to the present invention；

Fig. 5 is the method with the user interface interaction for staring operation for schematically showing embodiment according to the present invention Flow chart；

Fig. 6 A to 6C be show embodiment according to the present invention use stare operation user interface execute a system Arrange the schematic diagram of operation；

Fig. 7 is the detection for schematically showing embodiment according to the present invention and the method for staring related instruction gesture Flow chart；

Fig. 8 is that the user of embodiment according to the present invention executes direction-selection gesture to select the first given interaction item Purpose schematic, pictorial illustration；

Fig. 9 is that the user of embodiment according to the present invention executes direction-touch gestures to operate the second given interaction item Purpose schematic, pictorial illustration；

Figure 10 be show embodiment according to the present invention optional direction-selection gesture (herein also referred to as triggering hand Gesture) schematic, pictorial illustration；

Figure 11 be the user of embodiment according to the present invention finger is presented on display given icon so as to school The schematic, pictorial illustration of semicomputer's system；And

Figure 12 A and 12B are that embodiment according to the present invention by the way that smaller or larger size icon is presented helps to use The schematic, pictorial illustration of the computer system of the given icon of family selection.

Specific embodiment

It summarizes

When using physical tactile input equipment, when such as button, idler wheel or touch screen, user usually pass through touch and/or The physical equipment is operated to participate in and be detached from the control to user interface.Present embodiments describe for be coupled to The pointer device that the interaction project presented on the display of the computer of the user interface of mixed form is interacted is run, this is mixed The user interface of conjunction form include by 3D sensor to the change in location of one or more physical feelings of user or move into Capable three-dimensional (3D) sensing, which is usually hand or finger.Instruction gesture described herein includes in detail below Direction-selection, direction-touch and direction-holding gesture of explanation.

Direction-selection gesture allows users to the interaction project that selection is presented over the display.For example, using direction-choosing Gesture is selected, user can start by executing direction-selection gesture towards icon (over the display) associated with film Watch film.Direction-touch gestures allow users to the interaction project that operation is presented over the display.For example, user passes through finger The interaction project (for example, film) presented over the display can be horizontally rolled to-touch gestures operation horizontal rolling frame List.Direction-holding gesture allows users to see the contextual information of the interaction project presented over the display.For example, ringing Direction-holding gesture should be executed on the icon for indicating film in user, pop-up window can be presented in computer, which includes all Such as the information of synopsis, film review and cast member.In some embodiments, when user executes above-mentioned instruction gesture, this is mixed The user interface of conjunction form can also transmit visual feedback.

When with traditional two dimension (2D) user interface interaction, it is anti-that above-mentioned physical equipment usually transmits tactile to user Feedback.However, the user is not engaging when the 3D user interface interaction of the user interface with mixed form as described herein Gesture can be executed in the case where any physical equipment, to not receive any touch feedback.The embodiment provides For non-tactile feedback to be interacted and received with the project presented over the display to compensate for the shortage of touch feedback Method and system.

System describe

Fig. 1 is the user interface 20 for the mixed form of embodiment according to the present invention operated by the user 22 of computer 26 Schematic, pictorial illustration.Although (for brevity, only single user and user interface is shown in figure, actually Interface 20 can be interacted with multiple users simultaneously.Different user interface and/or branch can be used in alternative embodiment of the invention Hold multiple user interfaces across distinct device).By way of example, user interface 20 in the illustrated embodiment is based on 3D sensing equipment 24, the sensing equipment capture 3D scene information, which includes body or at least region, example Such as finger 30, hand 31, head 32 or eyes 34.Equipment 24 or individual camera (not shown) can also be with capturing scenes Color video frequency image.The information that equipment 24 captures is handled by computer 26, which correspondingly drives display screen 28 To present and the interaction project 36 (also referred to as interaction project herein) in operation display.Optionally, the user interface can with it is any The computer equipment of type is used in combination with, such as laptop, tablet computer, television set etc..

Although fig 1 illustrate that the computer 26 of tower structure, but the computer of other structures is recognized as of the invention Within spirit and scope.For example, computer 26 can be configured to desktop computer, portable computer (for example, notebook is electric Brain) or all-in-one computer.

The data that the processing of computer 26 is generated by equipment 24, to rebuild the 3D figure of user 22.Term " 3D figure " (or it is corresponding " depth map ") refer to one group of 3D coordinate for indicating the surface of given object, in this case, given object refers to user Body.In one embodiment, speckle patterns are projected to the image on object and capturing projected pattern by equipment 24.So Afterwards, the lateral displacement based on the spot in imagewise pattern, computer 26 pass through on the body surface of trigonometric calculations user The 3D coordinate of point.By way of example, it is based on equipment 24, referring to the X-axis 40 of basic horizontal, substantially vertical Y-axis 42 and depth Z axis 44 is spent, 3D coordinate is measured.For example, the side for this 3D mapping based on triangulation using projection pattern Method and equipment are described in PCT International Publication WO 2007/043036, WO 2007/105205 and WO 2008/120217, institute Disclosure is incorporated herein by reference.Optionally, as known in the art, other sides of 3D mapping can be used in system 20 Method uses single or multiple cameras or other types of sensor.

In some embodiments, equipment 24 includes usually the light reflected from one or two eyes 34 by processing and analysis The image of (usually infrared light and/or be added the color that color model generates by red-green-blue) detects the eye of user 22 The position and direction of eyeball 34, in order to find the gaze-direction of the user.In an alternate embodiment of the invention, computer 26 (passes through it Itself is combined with equipment 24) detection user eyes 34 position and direction.The light reflected can be originated from equipment 24 Light projection source or any other natural source (for example, daylight) or artificial light sources (for example, lamp).Using known in the art Technology, such as detection pupil center and corneal reflection (PCCR), equipment 24 can handle and analyzes including the composition from eyes 34 The partially image of the light of (such as pupil 38, iris 39 or cornea 41) reflection, to find the gaze-direction of user.In addition, setting Standby 24 can transmit from the light of corneal reflection as flicker effect (to computer 26).

The position for the user's head (for example, the edge of eyes, nose or nostril) extracted from 3D figure by computer 26 and Feature can be used for finding the rough position of eyes of user, so that the accurate measurement of eye position and gaze-direction is simplified, and And it is more reliable and steady to make this stare measurement.In addition, computer 26 can be easily by the portion on the head 32 provided by 3D figure Divide the position 3D of (for example, eyes 34) and is combined to know via the angle information of staring of the image analysis acquisition of eye portion The other user object 36 all on the given screen of viewing at any given time.The some eyes being such as known in the art In tracking system, 3D, which maps and stares tracking and is used in conjunction with, allows user 22 to move freely through head 32, while by using head On sensor or transmitter mitigate initiatively track head needs.

By tracking eyes 34, the embodiment of the present invention is after user's moving-head 32, it is possible to reduce recalibrates user 22 needs.In some embodiments, the depth information on head 32, eyes 34 and pupil 38 can be used in computer 26, so as to The movement on head is tracked, so as to realize that reliable angle of gaze calculates based on the individual calibration of user 22.Using in this field Known technology, such as PCCR, pupil tracking and pupil shape, computer 26 can calculate eye according to the fixed point on head 32 The angle of gaze of eyeball 34, and using the location information on the head to recalculate the angle of gaze and to enhance the accurate of aforementioned techniques Property.In addition to reduce recalibrate, the further advantage for tracking head include: reduce the quantity of light projection source and reduce for The quantity of the camera of track eyes 34.

Computer 26 generally includes general-purpose computer processor, is programmed to implement function described below in software Energy.For example, the software can electronically be downloaded to by network on processor or it also can be provided in such as light It learns, on the tangible computer-readable medium of the nonvolatile of magnetic or electronic storage medium.Additionally or alternatively, computer disposal Some or all of device function can be in such as integrated circuit or programmable digital signal processor of customization or semi-custom (DSP) it is realized in specialized hardware.Although by way of example computer 26 is shown as dividing with sensing equipment 24 in Fig. 1 From unit, some or all of computer processing function can it is intracorporal by the sensing equipment shell or with the sensing equipment phase Associated suitable special circuit executes.

Alternatively, these processing functions can by with display 28 (for example, in a television set) It integrates, or is integrated in the computer equipment such as game console or media player of any other suitable type Suitable processor together is implemented.The sensing function of equipment 24 equally can be incorporated into computer or the output by sensor In other computer installations of control.

Various technologies can be used to rebuild the 3D of the body of user 22 figure.In one embodiment, computer 26 from by The 3D connection component for corresponding to physical feeling is extracted in the depth data that equipment 24 generates.For example, being submitted on August 11st, 2010 U.S. Patent application 12/854187 in describe technology for this purpose, the disclosure is incorporated herein by reference.Meter Calculation machine analyzes these extracted components, so as to " skeleton " of the body of reconstructing user, U.S. Patent application as mentioned above Disclose 2010/0034457, or described in the U.S. Patent application 12/854188 submitted for 11st in August in 2010, disclosure Content is also by being incorporated herein by reference.In an alternate embodiment of the invention, other technologies also can be used identify user's body certain A little positions, and do not need 24 whole of equipment or be even partially seen entire body or skeleton to be reconstructed.

Even if being asked due to such as smallest object size and reducing these apart from the biggish distance punishment resolution of equipment 24 Topic, so that physical feeling (for example, finger tip) may not be detected by depth map, but by using the skeleton of reconstruct, computer 26 can be assumed the position at the physical feeling such as tip of finger 30.In some embodiments, it is examined according to the early stage of physical feeling It surveys or tracks physical feeling according to along several (previous) depth maps received, computer 26 can be based on the institute of human body Expected shape is automatically completed physical feeling.

In some embodiments, as the reconstruct of this skeleton as a result, including user's by the information that computer 26 generates Head and arm, trunk there may also be the position and direction of two legs, hand and other features.In different frames (that is, depth map) The variation of these features or the variation of user's posture be capable of providing the instruction of gesture and other movements that user makes.User's appearance Gesture, gesture and other movements can provide the control input interacted for user with interface 20.These body actions can with by Other interactive forms that equipment 24 senses combine, these body actions include user eyeball movement as described above and language Sound instruction and other sound.Therefore, interface 20 makes user 22 be able to carry out various remote control functions and can be with appearance Application program, interface, video program, image, game and other multimedia content on the display 28 interacts.

Fig. 2 is the block diagram for schematically showing the functional unit of user interface 20 of embodiment according to the present invention. Sensing equipment 24 includes illumination sub-component 50, is projected a pattern on target scene.Such as appropriately configured video camera The image of pattern in 52 capturing scenes of Depth Imaging sub-component.In general, illumination sub-component 50 is with imaging sub-component 52 infrared Operation in range, although other spectral regions can also be used.Selectively, the color video camera (not shown) in equipment 24 2D color image in capturing scenes, and microphone 54 can also capture sound.

Processor 56 from sub-component 52 receive image and by each image pattern and the ginseng that is stored in memory 58 Pattern is examined to compare.By being projected in pattern away from the reference planes at 24 known distance of equipment, reference pattern is usually pre- First it is captured.Local offset of the 56 calculating section pattern of processor on the region of 3D figure, and depth is converted by these offsets Spend coordinate.For example, the details of the process is described in PCT International Publication WO 2010/004542, the disclosure of which is logical It crosses and is incorporated herein by reference.Optionally, as previously mentioned, by other modes known in the art, such as three-dimensional imaging, similar sonar Equipment (based on sound/acoustics), wearable tool, laser or propagation time measurement, equipment 24 can be configured to give birth to Scheme at 3D.

Processor 56 generally comprises embedded microprocessor, described below to implement by being programmed with software (or firmware) Processing function.For example, the software can electronically be supplied to processor by network；Optionally or additionally, which can To be stored in the visible computer readable medium of nonvolatile, such as it is stored in light, magnetic or in electronic storage medium.Place Reason device 56 also includes suitably outputting and inputting interface, and the processor may include functional for realizing its certain or institute Dedicated and/or programmable hardware logic electric circuit.Some processing functions and it can be used for implementing these processing functions The detailed description of circuit is shown in WO 2010/004542 disclosed above.

In some embodiments, Staring Sensors 60 detect user 22 by capturing and handling the two dimensional image of user 22 Eyes 34 gaze-direction.In other embodiments, the sequence 3D figure that computer 26 is transmitted by processing equipment 24 come Detect gaze-direction.Any method appropriate of eyes tracking as known in the art can be used in sensor 60, for example, upper It states United States Patent (USP) 7,762,665 or the method described in United States Patent (USP) 7,809,160, disclosures of which passes through reference It is incorporated herein, or optional method described in the bibliography quoted in these patents.For example, sensor 60 can be with Capture the image of the light (usually infrared light) from the eyeground or corneal reflection of the eyes of user.This light passes through irradiation sub-component It 50 or can be projected towards eyes by another projection element (not shown) relevant to sensor 60.Sensor 60 is in user The entire target area at interface 20 can capture its high-definition image and can then position in target area from eyes Reflection.Optionally, in addition to capturing the pattern image for 3D mapping, imaging sub-component 52 can be captured from eyes of user It reflects (environment light, the reflection from monitor).

Alternatively, processor 56 can drive scanning control 62 to guide the visual field court of Staring Sensors 60 The face of user or the position of eyes 34.As described above, based on depth map or based on the skeleton according to 3D figure reconstruct, or use Face recognition method known in the art based on image can determine the position by processor 60 or by computer 26. For example, scanning control 62 may include the ability of electromechanical frame or scanning optics or photoelectric cell or any other suitable type Scanner known to domain, such as be configured to scene reflectivity to Staring Sensors 60 based on the anti-of MEMS (MEMS) Penetrate mirror.

In some embodiments, scanning control 62 can also include optics or electronic zoom, be mentioned according to by 3D figure The equipment 24 of confession adjusts the magnifying power of sensor 60 to the distance of user's head.Made by the above-mentioned technology that scanning control 62 is implemented The image of eyes of user can be captured with high precision by obtaining the only Staring Sensors 60 with intermediate resolution, to provide accurate The information of gaze-direction.

In other embodiments, computer 26 can be used scanning control angle (that is, relative to Z axis 44) it is solidifying to calculate Visual angle.In a further embodiment, computer 26 can compare is known in the scenery and 3D depth map that Staring Sensors 60 capture Other scenery.In another embodiment, scenery and 2D the camera capture that computer 26 can capture Staring Sensors 60 Scenery compares, which has the wide visual field including entire target scene.Additionally or alternatively, scanning control 62 can wrap Include the sensor (usually optical or electric) for being configured as verification eyes move angle.

The gaze angle of user is extracted in the processing of processor 56 by 60 captured image of Staring Sensors.By by sensor 60 angle measurement done combine with the position 3D of user's head provided by Depth Imaging sub-component 52, and processor can Accurately obtain the Real line-of-sight of user in 3d space.3D mapping and the combination of gaze-direction sensing reduce or eliminate accurate school The needs of true gaze-direction are just extracted with more multiple reflection signals.The sight information extracted by processor 56 makes to count Calculation machine 26 can reliably identify the interaction project that user is just watching attentively.

As not needing detection point of scintillation (for example, used in PCCR method), the combination of two kinds of forms can be permitted Perhaps gaze detection is carried out without using active projection device (that is, irradiation sub-component 50).It can solve this field using the combination It is known other to stare known glass-reflected problem in method.Using reflected from natural light, 2D image (that is, detection pupil position Set) and the information that obtains of 3D depth map (that is, position that head is identified by the feature on detection head), computer 26 can be with It calculates angle of gaze and identifies the given interaction project that user is just watching attentively.

As described above, Staring Sensors 60 and processor 56 can track one or two eyes of user.If can Two eyes 34 are tracked with enough precision, processor can also provide individual gaze angle measurement for each eye.Work as eyes When watching the object of distant place attentively, the gaze angle of eyes will be parallel, but for object nearby, which usually converges Gather in any close to target object.In the 3D coordinate for extracting user's point that its angle of gaze is fixed at any given time When, which can be used together with depth information.

As described above, equipment 24 can produce while the 3D figure of multiple users in the equipment visual field.It is whole by providing The single high-definition picture of a visual field, or by being scanned scanning control 62 to the head position of each user, stare Sensor 60 can similarly find in these users everyone gaze-direction.

3D is schemed and to stare information defeated by the communication link 64 that processor 56 is connected via such as universal serial bus (USB) Out on the suitable interface 66 of computer 26.The computer include with memory 70 central processing unit (CPU) 68 with And user interface 72, it drives display 28 and also may include other components.As described above, 24 property of can choose of equipment It only exports original image, and 3D figure as described above and stares calculating and can be executed by CPU68 with software.For scheming from 3D It can be run in processor 56, CPU68 or in the two with the middleware for extracting higher information is stared in information. CPU68 runs one or more application program, these application programs are usually via application programming interfaces (API), based on by centre The information that part provides drives user interface 72.For example, these applications may include game, amusement, surfing on the net and/or office Using.

Although processor 56 and CPU68 are since the processing task between them is with specific divide and as independent function Can unit be shown in Fig. 2, the function of processor and CPU can optionally be implemented by single processing unit or these Function can be divided into the processing unit implementation by three or more.In addition, although equipment 24 is shown as comprising specific arrangements In element certain combination, other device configurations can be used for purpose described herein, and be considered as in the scope of the invention Within.

Gaze detection

Fig. 3 is the process for schematically showing the method for the gaze-direction of detection user 22 of embodiment according to the present invention Figure.In initial step 80, computer 26 receives at least one physical feeling including user 22 from Depth Imaging sub-component 52 3D figure, and in extraction step 82, computer divides the received 3D figure of institute to extract the 3D coordinate on head 32.? In receiving step 84, computer 26 receives the two dimensional image including eyes 34 of user from Staring Sensors 60.As described above, according to Penetrate sub-component 50 can towards 22 projection light of user, and received image may include from eyeground and/or from eyes 34 The light reflected on cornea.In identification step 86, computer 26 analyzes received depth map and image in order to identify The gaze-direction of user 22, and this method terminates.

In some embodiments, computer 26 is by identifying head from 3D figure along X-axis 40, Y-axis 42 and Z axis 44 The 3D coordinate on head 32 is extracted in position.In other embodiments, computer 26 by along X-axis 40 and Y-axis 42 from 2D image In identify the first position on head, and extract head by identifying the second position on head from 3D figure along Z axis 44 The 3D coordinate in portion 32.

The embodiment of the present invention can be used gaze detection by controlled in response to the direction stared computer system for example in terms of The function of calculation machine 26.In some embodiments, computer 26 can identify on the position of gaze-direction on the display 28 The given interaction project presented, and change in response to gaze-direction the state of given interaction project.

In the first embodiment, the state for changing given interaction project 36 may include executing and the given interaction item The associated operation of mesh.For example, interaction project 36 may include user 22 can choose present on the display 28 it is specific in Hold the menu option of (for example, movie or television program).In a second embodiment, computer 26 will be by will be received defeated from user Enter and be directed to given interaction project, thus it is possible to vary the state of given interaction project.For example, given interaction project can wrap Text box is included, and if user is staring text frame, computer 26 can provide received from keyboard to text frame Alphanumeric input.

In other embodiments, computer 26 can identify presented on the display 28 on the position of gaze-direction to Fixed interaction project, and the state in response to changing given interaction project from the received sound instruction of user.For example, given Interaction project may include icon relevant to software application, and the user can stare the icon and say " beginning " In order to execute the application.In a further embodiment, user interface 20 can be configured to identify in response to gaze-direction to Fixed interaction project, and given interaction is manipulated in response to the gesture performed by limbs (for example, finger 30 or hand 31) Project.For example, after having selected given interaction project, if the computer, which receives, indicates the user to brandish movement (that is, along plane including X-axis 40 and Y-axis 42) is schemed to move a sequence 3D of hand 30, and computer 26 can waved correspondingly Selected interactive project is relocated on dynamic direction (that is, from left to right).

In another embodiment, the embodiment of the present invention can be used for receiving image (2D image or 3D including eyes 34 Figure)；Based on the image recognition gaze-direction；Based on the region in gaze-direction on gaze-direction identification display, and Operation is executed on the region.For example, computer 26 may include the desktop computer in conjunction with digital camera, display 28 can be with Including the display for the desktop computer, and the camera may be configured to the lens focus of camera in institute In the project presented in the region identified.

Additionally or alternatively, computer 26 can identify equipment that is being coupled to the computer and being located in gaze-direction, And in response to the function of gaze-direction control equipment.For example, if user stares the top for being coupled to the loudspeaker of computer, The volume level of loudspeaker can be improved in computer, and if user stares the bottom of loudspeaker, computer can reduce volume Grade.

Fig. 4 is the number key for being configured for input password of embodiment according to the present invention presented on the display 28 The schematic diagram of disk 90.Scheme herein and subsequent figure in the presence and staring of user 22 indicate that the eyes have with eyes 34 Sight 92.Sensing equipment 24 in the visual field 94 of the equipment by finding the position 3D of user's head and the side of staring of eyes 34 To can determine sight.

In the present specification, interaction project 36 can be by using different descriptions (for example, using icon or scroll box generation For interaction project) and additional letter to identifying that label distinguishes.For example, interaction project 46 includes numeral input in Fig. 4 Key 36A and start button 36B.When enter key and start button is presented, computer 26 prompts user 22 to input password.Although number Keyboard is shown in Figure 4, and the graphic element on any kind of screen may be used as " key " for the purpose.When user wants to open When beginning to operate computer 26 or operate the specific application program executed on computers, user can be by his sight from one Given key 36A guides next key in appropriate sequence into, in order to input password.Alternatively, when staring for the user rests on often When on a given key 36A (that is, in a series of keys comprising password), which can make specific gesture or sound instruction (such as " input "), in order to notify equipment 24 to record the selection.

In some embodiments, computer 26 may be configured to made with or without user it is any additional In the case where inputting (that is, gesture or sound), select a series of interactive projects 36 (for example, when user by his sight from one Interaction project is when being moved to next) indicate the interaction project selected.Similarly, computer 26 can be configured as not Provide a user any video or audible feedback for indicating the selection.Therefore, it even if user inputs password in public, looks on Person will be unable to determine or replicate password.

Input password after, the user can guide his sight towards start button 36B in order to continue and computer into Row interaction.

In addition to Password Input, gaze detection can be used for otherwise enhancing safety.For example, computer 26 can be with Learn the other biological statistical natures of the distinctive eye movement mode of user and/or eyes 34 using as additional recognition methods.As Another example, equipment 24 can be configured to the gaze angle that can not only find user 22, but also can find and be located at view Other people gaze angle in 94 (for example, being reconstructed by skeleton from being identified in 3D figure).In this case, when another Individual is being when just watching display 28 attentively, and equipment 24, which can be configured as warning user, (or even can prompt computer 26 over the display Other people image that display equipment 24 is captured).Such function can help user 22 to protect oneself from eavesdropping Listener-in continues to look at display with prevention.

As power saving features, equipment 24 can detecte when user does not look at display 28, and when user's When sight leaves the time of display 28 more than during threshold time, computer 26 can activate power saving technique.For example, when using When family does not look at display, computer 26 can make display completely dimmed or dim (that is, reducing brightness).When the user returns When coming over to see to display, computer can deactivate power saving technique.For example, detecting sight return of the user 22 by him When on to display, computer 26 can increase brightness, make brightness back to full brightness.This screen control stared that relies on exists In terms of improving the battery life of portable device and power consumption is being reduced generally for cost and environment friendly is saved Aspect is also useful.

Stare the user interface of operation

Interactive user interface can be used to create by staring tracking, which can detecte out user and be just look at Which interaction project (for example, text box or application program such as word processor) on screen, to eliminate for mouse And/or the needs of keyboard.For example, text can be automatically directed to the text that the user is just being look at when user keys in text Frame.As another example, when first time button is carried out at a word or sentence as user, " mouse click " type event quilt It is sent in watched attentively text box, this is input into text in text frame.In this way, user is not necessarily to mobile mouse Field (" name ", " surname ", " password " etc.) switches over and just could fill out web form mark one by one.

In addition, staring the combination of tracking Yu other forms (such as 3D mapping/gestures detection and/or speech detection), use Family can sufficiently control the object on screen, without using mouse or touch screen.In this way, full model can be implemented in user The direction and selection function enclosed, including scanned for by a large amount of information project and selection.Combined interface form can also Be used to search in the context of some interaction project and execute control function, such as executes and look into open file It looks for, shear, replicating and paste functionality.

Fig. 5 is the method with the user interface interaction for staring operation for schematically showing embodiment according to the present invention Flow chart, and Fig. 6 A to 6C is to show the use of embodiment according to the present invention to stare the user interface of operation and execute The schematic diagram of sequence of operations, in initialization step 100, computer system such as computer 26 presents more on the display 28 A interactive project 36.Shown in Fig. 6 A-6C configuration in, computer 26 initially at the lower right corner of display 28 present start by Button 36B.Since user 22 makes its sight 92 towards start button 36B, computer 26 receives instruction user in receiving step 102 The input (for example, depth map) of 22 gaze-direction.Once it is determined that gaze-direction, computer 26 is based in identification step 104 Gaze-direction (that is, point on the display that the user is just watching attentively) identifies the target point 120 on display 28.

In comparison step 106, if the first interaction project 36 in associated steps 108, is calculated close to target point Machine 26 is associated with the first interaction project by target point.In the example shown in Fig. 6 A, the first interaction project includes start button 36B, and target point 120 is located at start button 36.Back in step 106, if target point is not close to any interaction Project, then this method proceeds to step 102.

In response of step 110, it is one or more close to the second of the first interaction project that computer 26 opens (that is, presentation) Interaction project 36, and this method terminates.Computer 26 can in response to user adjust gaze-direction make target point close to its In one second interactive project, come select one of them second interaction project.

Although the configuration of the interactive user interface in Fig. 6 A to Fig. 6 B is shown computer 26 and is presented with concentric pattern Positioned at the second interaction project 36 of the first interaction project radial outside, other configurations are recognized as in spirit and model of the invention Within enclosing.For example, the first and second interaction projects can be rendered as nested rectangle icon by computer 26.

In some embodiments, computer 26 can be in response to the hand as performed by limbs (for example, finger 30 or hand 31) Gesture opens one or more the second interaction project.For example, after target point is interacted item association with first, if computer Instruction user is received to brandish the sequence 3D for acting (for example, along the plane for including X-axis 40 and Y-axis 42) mobile hand 30 Figure, computer 26 can responsively open one or more the second interaction project.

In the configuration shown in Fig. 6 A, computer 26 presents the menu area 122 of the second interaction project, second interaction Project includes the tabs 36C (being arranged with concentric pattern) of available various applications on computer.In some embodiments, Computer 26 stares target point 120 in response to user and opens menu area 122, to It is not necessary to user be asked to carry out any shape The body gesture or phonetic order of formula.

In fig. 6b, for user 22 by its sight 92 towards voicemail option card 36C, this causes computer 26 to automatically open use Email (E-mail) inbox at family, contains the list 36D of the message item received.When user is arranging its sight When being scanned up and down on table 36D, and when sight 92 is rested on specific list items 36D, computer present automatically by The preview 124 of the content of message item represented by the list items.User, can by moving up or down its sight 92 Preview others list items 36D, or main menu can be returned by the way that its sight 92 is directed to return push-button 36E.It is all these dynamic Work can be executed only by eye motion.

In order to open selected message item, user can input instructions into computer 26 by another form.For example, User is it may be said that " opening ", or the gesture opened can be executed while staring list items 36D corresponding with message item.Sensing Equipment 24 detects the audio input or three-dimensional motion that user is done, and will be in instruction input computer 26 appropriate.As a result, Fig. 6 C Shown in screen open, the full content 126 of selected message item is presented.The user can be by staring movement for him To put to return inbox or usable return push-button 36E return main menu from this to " returning to inbox " region 36F, such as above It is previously mentioned.

When the given interaction project 36 of selection, computer 26 visual feedback can be passed to the instruction selection (that is, Execute such as present full content 126 movement before) user.The example of visual feedback includes the size for changing selected item And/or appearance, or selected project is highlighted by surrounding selected project with frame.Transmitting visual feedback to use Family 22 concentrates on his stare near target point, to optimize user experience.For example, when user selects start button 36B When, computer 26 can transmit visual feedback via start button, so that user be guided to be maintained close to calculate by his stare The position of the tabs 36C in menu area 122 is presented in machine.

Instruction gesture relevant to staring

As detailed below, when user 22 executes direction-selection, direction-touch and direction-context gesture, user 22 are directed toward given interaction project 36.In an embodiment of the present invention, computer 26 passes through first point and second of restriction user Straight line between point identifies user's positive sense where, and identifies straight line wherein (that is, target point 120) and 28 phase of display It hands over.

Fig. 7 is the detection for schematically showing embodiment according to the present invention and the method for staring related instruction gesture Flow chart.Fig. 8 is according to an embodiment of the invention, user 22 executes direction-selection gesture to select to be on the display 28 The schematic, pictorial illustration and Fig. 9 of the given interaction project 36 of existing first are according to an embodiment of the invention, user 22 holds Row direction-touch gestures are to operate the schematic, pictorial illustration of the second given interaction project 36 presented on the display 28. User 22 includes icon 36G using the example of the selectable first interaction project of direction-selection gesture, and user 22 uses direction- The example of the operable second interaction project of touch gestures includes icon 36G, vertical scrolling frame 36H and horizontal rolling frame 36I.

Direction-selection gesture includes the given interaction project of finger 30 (usually index finger) direction first that user 22 will give 36, finger 30 (generally along z-axis 44) is shifted into the first given interaction project, stops or slows down finger, then retract finger User.Direction-touch gestures include user 22 by the given interaction project 36 of finger direction second, by finger 30 (usually along z-axis 44) the second given interaction project is shifted to, suspends finger, then by hand along the planar movement including X-axis 40 and Y-axis 42.

In the first receiving step 130, the computer system of such as computer 26 receive include user 22 3D figure the One sequence, and in extraction step 132, computer segmentation received 3D figure to extract first point of user and second point 3D coordinate.Although Fig. 8 and Fig. 9 show the second point of the and finger tip including finger 30 (index finger) including eyes 34 at first point, Different first point and second point are considered within the scope and spirit of the invention.For example, first point may include between eyes 34 A point or user some other point on the face, and second point may include the arbitrary point on any limbs of user.For with Different calculating can be used to identify and second point at first point in the different gestures at family 22, computer 26.

In the first comparison step 134, if the First ray of 3D figure indicates that the user is just aobvious relative to (being generally directed toward) Show the mobile second point of device 28, then computer 26 receives and divide the 3D that instruction second point slows down in the second receiving step 136 Second sequence of figure.In calculating step 138, computer 26 calculates the line segment 160, Yi Ji intersected with and second point at first point In identification step 140, line segment 160 is extended to display 28 by computer, and identifies the target of line segment 160 Yu display intersection Point 120.

In some embodiments, user 22 may not look at target point.For example, computer 26 can determine (that is, using Above-mentioned gaze detection embodiment) he just stares the first given interaction item guided into the presentation of the left side of display 28 by user Mesh, but step 140 recognizes the second given interaction project 36 that user's positive sense is located at 28 right side of display.In this feelings Under condition, although he is stared direction the first interaction project by the user, computer may be configured to selection the second interaction project.

In the second comparison step 142, if given interaction project is being presented close at target point 120 for computer 26 36, then computer engages (engage) with the given project 36 that interacts close to target point, and connects in third receiving step 144 Receive and divide the third sequence of 3D figure.In third comparison step 146, if the third sequence instruction user of 3D figure is just from display Device (that is, towards user 22) removes second point, then selection step 148 in, computer 26 be in close at target point 120 Existing given interaction project engages (that is, selection), and then this method terminates.In the example shown in Fig. 8, user 22 can be with Using include by finger 30 be directed toward icon " A " and finger is retracted direction-selection gesture selection icon " A ".

In some embodiments, engaging given interaction project may include that execution is associated with given interaction project Movement.For example, given interaction project may include the given icon 36G of film, engaging the given icon 36G includes executing application Program watches film.

Optionally, when being directed toward icon " A ", user 22 can issue the phonetic order such as " selected ".Once from Mike Wind 54 receives phonetic order, and computer 26 can be executed and be presented on close to the given interaction project at target point 120 (for example, icon " A ") relevant operation.

In some embodiments, although above-described gaze detection embodiment instruction user is given currently without looking at Interaction project, but direction-selection gesture can be used in computer 26, selects given interaction project 36.In first example, if Computer 26 does not change his gaze-direction in detecting user at the appointed time, and works as received 2D image When not indicating gaze-direction (for example, hand 31 blocks eyes 34) with 3D figure, computer 26 identifies direction-selection gesture, then Computer 26 " can cover " gaze detection and in response to direction-selection gesture.

In second example, if finger movement posture is " significant " (that is, at least with the first specified distance), And if the interaction project that 120 distance of target point is presented in gaze-direction is at least the second distance to a declared goal, computer 26 can respond direction-selection gesture.For example, if user is just watching the first interaction project in the presentation of the left side of display 28 attentively 36, and direction-selection gesture is executed to the second interaction project 36 presented on the right side of display, then computer 26 can be configured For engagement the second interaction project.

If the third sequence of 3D figure does not indicate that user just removes second point from display 28, compare the 4th In step 150, computer 26 analyzes the third sequence of 3D figure to determine whether user 22 is positive along the X-Y for including X-axis 40 and Y-axis 42 The planar movement second point.If the third sequence of 3D figure indicates that the user just moves the second point along X-Y plane, in weight In positioning step 152, computer 26 is located adjacent to the interaction project at target point 120 in response to the movement of second point, Then this method terminates.In the example shown in Fig. 9, user 22 can be used including finger 30 is directed toward vertical scrolling frame and is incited somebody to action Direction-touch gestures that finger moves up and down, mobile vertical scrolling frame 36H.

If the third sequence of 3D figure does not indicate that the user just moves second point along X-Y plane, compare the 5th In step 154, computer 26 analyzes the third sequence of 3D figure to determine whether user 22 just stably keeps second point relatively and hold Continue at least specific period (for example, two seconds).If the third sequence of 3D figure indicates that the user is just opposite and is stably keeping the 2 points continue the specific period, then in rendering step 156, interaction of the computer 26 to being located adjacent at target point 120 The project implementation keeps operation.For example, keeping operation may include the context that the interaction project being located adjacent at target point 120 is presented Information.The example of contextual information includes the attribute or the available option of user of interactive project.Disappear for example, computer 26 can be presented Breath " retracts finger to select or horizontal or vertical ground moves finger and relocates project." keep another example of operation can The interaction project being located adjacent at target point 120 is deleted including computer 26.

Return step 154, if the third sequence of 3D figure does not indicate user, the opposite second point that stably keeps continues spy The fixed period, then this method proceeds to step 130.Similarly, if in step 142 computer 26 do not present it is close Given interaction project 36 at target point 120, or if the First ray of 3D figure does not indicate user 22 in step 134 It is just that second point is mobile towards display 28, then this method also continues to step 130.

In some embodiments, computer 26 can be coupled to the equipment of computer in response to gesture control described herein Function.For example, if user executes direction-touch gestures, computer when direction is coupled to the loudspeaker of computer upward The volume level of loudspeaker can be improved.Similarly, if user executes direction-touch gestures in directional loudspeaker downward, that Computer can reduce the volume level of loudspeaker.

Although gesture described in the flow chart of Fig. 7 includes direction-selection gesture and direction-touch gestures, those warps It should be considered by other instruction gestures of line segment 160 (that is, the line segment intersected with first point and second point) identification target point 120 In the spirit and scope of the present invention.Additionally or alternatively, direction-selection gesture and direction-touch gestures can with it is above-mentioned Gaze detection embodiment is used in combination.

For example, computer 26 can be configured as once staring based on user, to indicate that user 28 is just shifting to finger aobvious The second sequence and instruction user for showing the 3D figure that the First ray of the 3D figure of device 28, instruction user are just slowing down finger just will When the third recognition sequence for the 3D figure that finger 30 is removed from display goes out target position 120, select (that is, direction-selection) given Interaction project 36.Similarly, computer 26 can be configured as once staring, indicating user 28 just by finger based on user 30 shift to the second sequence and instruction of the First ray of the 3D figure of display 28, the 3D figure that instruction user is just slowing down finger When user just goes out target position 120 along the third recognition sequence of the 3D figure for the planar movement finger 30 for including X-axis 40 and Y-axis 42, The responsively given interaction project 36 of reorientation (i.e. direction-contact).

Figure 10 is to show the optional direction-selection hand for also referred to as triggering gesture herein of embodiment according to the present invention The schematic, pictorial illustration of gesture.Index finger 30 is being referred to that after given interaction project 36, user 22 can be by raising or closing Hold together the given interaction project of thumb 170 (as shown in arrow 172) selection, wherein close up thumb complete the gesture (that is, without It is that finger is shifted to interactive project).Optionally, once finger 30 is directed toward given interaction project and sending sound by user 22 Sound instructs to select given interaction project (for example, user says word " opening "), and computer 26 may be selected by given Icon.

Additionally or alternatively, although example shown in Fig. 8 to Fig. 9 shows at first point on head 32, Its first point is considered within the scope and spirit of the invention.In the example shown in Fig. 10, first point includes index finger 30 Joint, and line segment 160 passes through finger tip and joint.

Non-tactile user interface calibration

In operation, non-tactile user interface 20 is usually used by more than one user, and each user may be different It is directed toward the same given interaction project 36 presented on the display 28 in ground.In some embodiments, using non-tactile user circle The element (that is, icon 30) in face, computer 26 can calculate for each user of non-tactile user interface and store calibration factor.

Figure 11 is that the user 22 of embodiment according to the present invention says the schematic illustration that finger 30 is directed toward given icon 36G It is bright.As shown in the drawing, the position based on user's finger 30, line segment 160 are directed toward the just following of icon " A ".Therefore, one use is identified Family 22, computer 26 can apply calibration factor appropriate when detecting gesture performed by the user.

For example, computer 26 " can be locked if non-tactile user interface 20 is inactive within specific a period of time Firmly " non-tactile user interface, and presentation user 22 can be directed toward to unlock the single solution lock icon 36G of user interface.Work as user When being directed toward solution lock icon, computer 26 can identify target point 120 and calculate calibration factor (namely based on target point for the user With the degree of closeness of solution lock icon).

Additionally or alternatively, might have such situation: when user 22 executes gesture, target point 120 is located at two Between a or multiple icon 36G, and computer cannot identify which icon user is directed toward.Where direction cannot be identified in computer 26 In the case where a icon 36G, computer can identify the icon subset close to target point 120, and identification is presented with bigger size The icon arrived, and user is prompted to refer to again once.

Figure 12 A is that the user 22 of embodiment according to the present invention positions finger 30 to be directed toward showing for one of icon 36G Meaning property illustrates.As shown in the drawing, target point 120 is located between icon " A ", " B ", " E " and " F ".

Figure 12 B is according to an embodiment of the invention, user 22 positions finger 30 to be directed toward the schematic of given icon 36G It illustrates, wherein icon is presented with bigger size in computer 26.Continue example shown in Figure 12 A, computer 26 is with bigger Size present icon " A ", " B ", " E " and " F ", then user 22 positioning finger 30 to be directed toward icon " A ".

In a similar way, as described above, when user selects via the gaze detection obtained from 2D image and/or 3D figure When given icon 36G, user interface 20 can calibrate user 22.Firstly, computer 26 " can lock " non-tactile user interface, And presentation user 22 can look to unlock user interface solution lock icon 36G.When user stares the solution lock icon, Computer 26 can identify target point 120 and calculate calibration factor (namely based on target point and solution lock icon close to journey for user Degree).

In second example, if user continues the specific interaction project 36G agaze given for a period of time, The contextual information of given interaction project can be presented in computer 26 in interaction project 36 (for example, pop-up dialog box).If User stares close to the interaction project that contextual information is presented, then computer 26 can be based on target point and contextual information Degree of closeness calculates calibration factor.

Middleware

As described above, for scheme from 3D and stare in information extract the middleware of higher level information can be in processor 50 And/or it is run on CPU 68, and CPU 68 can drive user circle based on usually being executed by the information that API is provided by middleware The application program in face 72.

It is the example for the middleware primitive that computer 26 can be used to extract information from the 3D figure for be received in equipment 24 below Son:

InteractStart (): the beginning that identification is interacted with user interface 20.

InteractHover (Pos2D, Radius): the current target point 120 on identification display 28 is (that is, user 22 Finger 30 be directed toward display on point coordinate).Pos2D parameter refers to (two dimension) position on display 28.

InteractPointNew (Pos2D, Radius): target is identified when user 22 executes direction-selection gesture Point 120.

InteractPointUpdate (Pos2D, Radius): when executing direction-touch gestures with user 22 along X- The mobile finger 30 of Y plane updates target point 120.

InteractEnd (Pos2D, Radius): identification user 22 when by finger 30 remove visual field 94.

Use the function of above-mentioned middleware, when user 22 executes direction-selection and direction-touch gestures, computer 26 can identify following " stage ":

1. finger identifies.In finger cognitive phase, middleware identifies and starts to track finger 30.Middleware tracks finger, And identify any intention of the finger towards display motion.This enables user interface 20 from other arbitrarily hand exercises Distinguish direction gesture.

2. gesture identification.In order to reach the gesture identification stage, it usually needs meet two conditions: (a) user 22 is by finger 30 shift to display 28, and (b) target point 120 in the boundary of display.In some configurations, target point 120 can " position In " outside of display 28 is a bit.For example, due to be effectively directed toward-touch gestures may include from " outside " display 28 drag Given interaction project 36 (for example, backspace key, alt-tab key etc.).

Once middleware just conveys InteractStart () event into second stage.Once conveying InteractStart () event, middleware, which is just constantly tracked finger 30 and finds, is limiting target point (the 3rd rank seen below Section) direction variation.When finger is mobile, middleware is just conveying InteractHover (Pos2D, Radius) event, makes User interface 20 is able to detect finger 30 just where is being directed toward at present by user 22.

3. interaction is directed toward.When user 20 stops finger 30, or finger 30 is removed from display 28, interaction is just reached The direction stage.According to the position of second point, middleware is calculated by connection (for example) eyes 30 and finger tip with creating line segment 160 Target point 120, and the line segment is extended into arrival display.Then middleware conveys the event of identification target point 120 InteractPointNew (Pos2D, Radius), so that user interface 20 be enable to select expected interaction project 36.

4. interaction terminates.User 22 finger 30 is removed and is removed from display 28 visual field 94 indicate gesture terminate (for example, Above-mentioned direction-selection and direction-touch gestures), so that user be made to be detached from user interface 20.It is re-engaged user interface 20, user 22 can relocate finger 30 in visual field 94, hence into finger cognitive phase described above.

In higher abstraction level, it can be defined similarly as in this field in conjunction with middleware primitive described above The following primitive for the existing touch screen primitive known:

Activation: direction-selection gesture of the given interaction project 36 of engagement (that is, " click ").Activation primitive can be used In activation application program, push button or follow hyperlink.

Translation: direction-touch gestures of given interaction project 36 are moved in any direction on an x-y plane.It is grasping In work, some application programs can be configured to make a response to the movement in X-axis or Y-axis.

Context: as described above, direction-holding gesture includes that finger is shifted to given interaction project 36 by user 22, With by finger it is opposite stably keep specific period (that is, in Microsoft's WinDowsTM environment by mouse picking in item On mesh and to press right mouse button similar).In response to direction-holding gesture, computer 26 can convey instruction to user in next step How this does the feedback of (for example, icon 36G that dragging and/or stopping give), or the information (example about given interaction project Such as, film plot described above is summarized).

In operation, in order to allow to develop the reflection user experience that executes on computer 26 in higher abstraction level (UX) application program of language primitive, middleware can convey following event:

Start (Pos2D, Radius): being started in a manner of being similar to InteractStart () event and user interface 20 interaction, but in higher abstraction level.

Activate (Pos2D, Radius): one in gesture primitive described below is activated.

Pan (Pos2D, radius): user 22 initiates direction-touch gestures.

PanEnd (pos2D, Radius): user 22 completes (that is, being detached from certainly) direction-touch translation gesture (for example, logical It crosses and removes finger 30 from display).

Context (Point2D, Radius): user 22 initiates direction-holding gesture.In operation, user 22 can be straight It connects from direction-holding gesture and is transformed into direction-touch gestures.

ContextEnd (): once direction-holding gesture (that is, being not transitioning to direction-touch gestures) is completed, user 22 are just detached from from user interface 20.

Interactive 3D display and game

The combined 3D mapping and gaze-direction information provided by sensing equipment 24, can use in various ways to improve 3D figure obtains and the quality and user experience of 3D rendering demonstration.For example, user moves through scene in interactive entertainment, Landscape on display 28 can become the focus in the direction of sight 92.Other regions are (such as the viewpoint nearby item as user The landscape of distant place when on mesh, or vice versa) landscape may deliberately be obscured, with simulate actual deep space and/ Or save bandwidth.

As another example, the game application run on computer 26 can be programmed to staring according to user " plot " of direction change game.For example, computer may at the time of given in user not in the display looked at Project (such as " enemy " role) is presented in region suddenly, keeps user surprised.It stares-guides above by reference to described in Fig. 6 A to Fig. 6 C Direction and selection method can equally be applied in game and other " virtual worlds " select interaction project 36 and " aiming " it On (weapon is such as directed toward target).

It the depth collected by equipment 24 and stares information and can be used for enhancing 3D display (especially autostereoscopic display) Ability and user experience.Operate the display by the way that different images is presented to the images of left and right eyes of user, but generally may only be from having The position of limit range is watched.By track user head position and gaze-direction, equipment 24 can guide the display with Change any image that it is presented, allows to watch them in bigger position range and show presenting over the display The different angle view of project.Applied to the near field project 36 presented on automatic stereoscopic display device (or other 3D display devices) Parallax can modify with a distance from display according to user's head, to enhance the sense of reality and to reduce some users at this The visual discomfort that may be felt in the environment of sample.

This kind of 3D display device can also be driven to interact with the 3D gesture with user.For example, the known bits based on user It sets and the gaze-direction of user, computer 26 can drive the 3D display device " can touch " position of phantom item in user The spatial display set they.Then user can item location in the 3 d space pass through he mobile hand (or other body Position) it operates phantom item and is interacted with phantom item.Equipment 24 senses the gesture of user and provides to computer 26 suitable When input so that computer can move in response to user's interaction or in addition modify the project.Such interactive mould For formula but also user can reach and interact with the object on given screen, which is located at the void created by display " back " of another object in quasi- space.

In another example, computer 26 can be presented including multiple interactive projects 36 on the display 28 (for example, game In role) content.In some embodiments, computer 26 can identify on display in the gaze-direction of user Region (that is, region around target point 12), and the content in the region identified is rendered as " focus alignment " (that is, clear Present clearly), and the content other than the region identified is rendered as (that is, faintly presenting) " out of focus ". In another embodiment, each of multiple interactive projects have a relevant depth value, give when user's direction When interaction project is stared, by the way that its relevant depth value is presented with the consistent project that interacts of the depth value of given interaction project For " focus alignment " (that is, clearly present), and its relevant depth value and the depth value of given interaction project is inconsistent Interaction project be rendered as (that is, faintly present) " out of focus ", computer can simulate 3D environment.

In further example, when playing game, user 22 is rendered as interactive icon by being directed toward over the display Weapon can choose weapon.If selected weapon is rifle, as described above, user " can be taken aim at direction-touch gestures It is quasi- " rifle, and use direction-selection gesture or triggering gesture "ON" rifle.Optionally, it if selected weapon is sword, uses It family can be by using direction-selection gesture and the combination of direction-touch gestures in three dimensions (that is, along X-Y plane and Z axis) Manipulate sword.

Above-mentioned application is only how the user interface of mixed form be used to improve system capability and user experience Several embodiments, but other similar applications are considered within the scope of the present invention.As another example, according to user Stared fixed to the object on screen, and when some program (such as commercial advertisement) is presented user whether just looking at it is aobvious Show device, the ability of sensing equipment 24 can be used for measuring user to the interest of the content of such as website or video program.These abilities It can be similarly used for extracting user interest overview, for example, the U.S. Patent Application No. 13/ submitted on November 14th, 2011 Described in 295106, it is incorporated herein by reference.

It should be understood that above-described embodiment is to be illustrated by way of example, and the present invention is not limited to above The content having had been particularly shown and described.More precisely, the scope of the present invention include various features as described above combination and Sub-portfolio, and for the ordinary skill in the art, made in the prior art after reading description above-mentioned There is no disclosed variations and modifications.

Claims (17)

1. a kind of method for controlling the function of computer system in response to the direction stared, comprising:

Receive the three-dimensional 3D figure of at least one physical feeling of the user of the computer system；

The two-dimentional 2D image of the user is received, described image includes the eyes of the user；

The 3D coordinate on the head of the user is extracted from 3D figure and the 2D image；

The image recognition of 3D coordinate and the eyes based on the head as the user execute described in the side that stares To；

Receive and divide the computer system the user at least one physical feeling three-dimensional 3D figure at any time One sequence, to extract and the 3D coordinate of second point at first point of the user, the 3D figure indicate the second point relative to It is coupled to the movement of the display of the computer system；

The line segment intersected with and the second point is calculated at described first point, wherein the line segment intersects with the display；

Identify the target point of the line segment Yu the display intersection；And

When detecting specific event, the second interaction item close to the target point presented on the display is engaged Mesh, the second interaction project are different from the first interaction item on the direction stared presented on the display Mesh,

Wherein the specific event include the second point movement have at least the first distance to a declared goal and the target point away from From the first interaction project at least second distance to a declared goal.

2. according to the method described in claim 1, the direction wherein stared described in identification includes the composition analyzed from the eyes The light of part reflection.

3. according to the method described in claim 1, the 3D coordinate for wherein extracting the head includes dividing the 3D figure to mention Take the head along trunnion axis, the position of vertical axes and depth axis.

4. according to the method described in claim 1, the 3D coordinate for wherein extracting the head includes identifying from the 2D image The head is along the first position of trunnion axis and vertical axes, and the segmentation 3D figure is so as to described in the identification from the 3D figure Head along depth axis the second position.

5. method according to claim 1 to 4, wherein the function of controlling the computer system includes changing The state of the second interaction project.

6. according to the method described in claim 5, further including receiving to indicate that the user moves limbs in a particular direction The sequence of three-dimensional figure, and the second interaction project is responsively relocated on the specific direction.

7. according to the method described in claim 5, further include based on the target point with described second interact project close to journey Degree is to calculate calibration factor.

8. method according to claim 1 to 4 connects wherein the function of controlling the computer system is included in It receives the First ray for the three-dimensional 3D figure for indicating limbs towards the display motion, receive the instruction limbs described in Second sequence of the 3D figure of display retarded motion and receive the instruction 3D figure of the limbs far from the display motion When third sequence, movement associated with the second interaction project is executed.

9. method according to claim 1 to 4, wherein the function of controlling the computer system is included in institute It states when staring described in user's guidance far from the display, power saving technique is activated by the computer system.

10. according to the method described in claim 9, wherein activation power saving technique includes reducing to be coupled to the department of computer science The brightness of the display of system.

11. method according to claim 1 to 4, wherein the function of controlling the computer system is included in When stared described in user guidance towards the display, power saving technique is deactivated by the computer system.

12. according to the method for claim 11, wherein deactivating power saving technique includes increasing to be coupled to the computer The brightness of the display of system.

13. method according to claim 1 to 4, wherein the function of controlling the computer system includes knowing It is not coupled to the equipment computer system and on the direction stared, and changes the state of the equipment.

14. method according to claim 1 to 4, wherein the function of controlling the computer system is included in It is coupled to presentation content on the display of the computer system；It identifies and is being coupled to the described aobvious of the computer system Show the region in the gaze-direction on device；And the content of the region exterior is made to fog.

15. method according to claim 1 to 4, wherein the function of controlling the computer system include: It is coupled on the display of the computer system and the content including multiple interactive projects is presented, the multiple interactive project Each of all have relevant depth value；Identify one on the direction stared in the interactive project；Know Not to one relevant depth value in the interactive project；And makes in the interactive project its depth value and identified The inconsistent any interactive project of depth value out fogs.

16. method as claimed in any of claims 1 to 4 further includes controlling in response to the direction stared It is coupled to the equipment of the computer system.

17. a kind of for controlling the device of the function of computer system in response to the direction stared, comprising:

Sensing equipment, the sensing equipment are configured as receiving the three-dimensional 3D figure of at least one physical feeling of user and the use The image of the eyes at family, and the two-dimentional 2D image of the user is received, the 2D image includes the eyes of the user；And

Computer, the computer are coupled to the sensing equipment, and are configured as scheming from the 3D and mentioning in the 2D image The 3D coordinate on the head of the user is taken, and is identified based on the image of the 3D coordinate on the head and the eyes by described The direction stared that user executes,

Wherein the computer is configured as: receiving and divide the three-dimensional 3D of at least one physical feeling of the user at any time The First ray of figure, to extract and the 3D coordinate of second point at first point of the user, the 3D figure indicates the second point Movement relative to the display for being coupled to described device；Calculate the line segment intersected with and the second point at described first point, Described in line segment intersect with the display；Identify the target point of the line segment Yu the display intersection；And it is detecting When to specific event, engage present on the display interact project close to the second of the target point, described second Interaction project is different from the first interaction project on the direction stared presented on the display,

Wherein the specific event include the second point movement have at least the first distance to a declared goal and the target point away from From the first interaction project at least second distance to a declared goal.